Nov 03 22:21:36.367699 ip-10-0-143-132 systemd[1]: Starting Kubernetes Kubelet... Nov 03 22:21:36.749369 ip-10-0-143-132 kubenswrapper[2541]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:21:36.749369 ip-10-0-143-132 kubenswrapper[2541]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 22:21:36.749369 ip-10-0-143-132 kubenswrapper[2541]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:21:36.749369 ip-10-0-143-132 kubenswrapper[2541]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:21:36.755174 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755047 2541 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:21:36.755174 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755170 2541 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:21:36.755174 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755175 2541 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:21:36.755174 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755179 2541 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:21:36.755464 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755184 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:21:36.755464 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755189 2541 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:21:36.755464 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755194 2541 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:21:36.755464 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755198 2541 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:21:36.755464 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755204 2541 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:21:36.755464 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755211 2541 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:21:36.755464 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755216 2541 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:21:36.755464 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755221 2541 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:21:36.755464 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755225 2541 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:21:36.755464 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755229 2541 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:21:36.755464 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755234 2541 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:21:36.755464 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755239 2541 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:21:36.755464 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755245 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:21:36.755464 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755250 2541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:21:36.755464 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755255 2541 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:21:36.755464 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755260 2541 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:21:36.755464 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755279 2541 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:21:36.755464 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755284 2541 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:21:36.755464 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755289 2541 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:21:36.756232 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755294 2541 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:21:36.756232 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755298 2541 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:21:36.756232 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755303 2541 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:21:36.756232 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755308 2541 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:21:36.756232 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755313 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:21:36.756232 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755327 2541 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:21:36.756232 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755332 2541 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:21:36.756232 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755337 2541 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:21:36.756232 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755341 2541 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:21:36.756232 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755346 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:21:36.756232 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755351 2541 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:21:36.756232 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755377 2541 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:21:36.756232 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755382 2541 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:21:36.756232 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755389 2541 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:21:36.756232 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755395 2541 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:21:36.756232 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755400 2541 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:21:36.756232 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755404 2541 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:21:36.756232 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755409 2541 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:21:36.756232 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755414 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:21:36.756232 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755418 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:21:36.757091 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755423 2541 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:21:36.757091 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755428 2541 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:21:36.757091 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755432 2541 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:21:36.757091 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755437 2541 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:21:36.757091 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755442 2541 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:21:36.757091 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755447 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:21:36.757091 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755452 2541 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:21:36.757091 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755458 2541 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:21:36.757091 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755463 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:21:36.757091 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755467 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:21:36.757091 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755473 2541 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:21:36.757091 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755477 2541 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:21:36.757091 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755482 2541 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:21:36.757091 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755486 2541 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:21:36.757091 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755491 2541 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:21:36.757091 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755496 2541 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:21:36.757091 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755500 2541 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:21:36.757091 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755505 2541 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:21:36.757091 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755510 2541 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:21:36.757091 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755515 2541 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:21:36.757970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755520 2541 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:21:36.757970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755525 2541 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:21:36.757970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755529 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:21:36.757970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755535 2541 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:21:36.757970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755540 2541 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:21:36.757970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755544 2541 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:21:36.757970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755549 2541 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:21:36.757970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755554 2541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:21:36.757970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755558 2541 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:21:36.757970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755563 2541 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:21:36.757970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755567 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:21:36.757970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755572 2541 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:21:36.757970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755577 2541 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:21:36.757970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755581 2541 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:21:36.757970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755586 2541 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:21:36.757970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755590 2541 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:21:36.757970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755595 2541 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:21:36.757970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755599 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:21:36.757970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755604 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:21:36.758825 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755609 2541 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:21:36.758825 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755614 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:21:36.758825 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755619 2541 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:21:36.758825 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755625 2541 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:21:36.758825 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755629 2541 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:21:36.758825 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755634 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:21:36.758825 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755639 2541 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:21:36.758825 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755644 2541 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:21:36.758825 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755648 2541 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:21:36.758825 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755653 2541 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:21:36.758825 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755658 2541 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:21:36.758825 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755663 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:21:36.758825 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755668 2541 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:21:36.758825 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755672 2541 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:21:36.758825 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755677 2541 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:21:36.758825 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755682 2541 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:21:36.758825 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755687 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:21:36.758825 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755691 2541 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:21:36.758825 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755696 2541 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:21:36.759438 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755701 2541 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:21:36.759438 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755705 2541 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:21:36.759438 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755710 2541 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:21:36.759438 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755715 2541 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:21:36.759438 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.755722 2541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:21:36.759438 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757031 2541 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:21:36.759438 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757041 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:21:36.759438 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757046 2541 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:21:36.759438 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757051 2541 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:21:36.759438 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757056 2541 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:21:36.759438 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757062 2541 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:21:36.759438 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757066 2541 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:21:36.759438 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757071 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:21:36.759438 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757075 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:21:36.759438 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757080 2541 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:21:36.759438 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757087 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:21:36.759438 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757093 2541 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:21:36.759438 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757097 2541 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:21:36.759438 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757102 2541 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:21:36.759438 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757106 2541 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:21:36.760158 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757111 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:21:36.760158 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757115 2541 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:21:36.760158 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757121 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:21:36.760158 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757125 2541 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:21:36.760158 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757130 2541 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:21:36.760158 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757135 2541 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:21:36.760158 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757140 2541 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:21:36.760158 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757145 2541 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:21:36.760158 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757149 2541 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:21:36.760158 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757154 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:21:36.760158 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757160 2541 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:21:36.760158 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757164 2541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:21:36.760158 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757169 2541 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:21:36.760158 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757173 2541 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:21:36.760158 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757178 2541 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:21:36.760158 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757183 2541 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:21:36.760158 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757188 2541 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:21:36.760158 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757192 2541 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:21:36.760158 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757197 2541 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:21:36.761062 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757202 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:21:36.761062 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757206 2541 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:21:36.761062 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757211 2541 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:21:36.761062 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757215 2541 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:21:36.761062 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757220 2541 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:21:36.761062 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757224 2541 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:21:36.761062 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757229 2541 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:21:36.761062 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757234 2541 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:21:36.761062 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757238 2541 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:21:36.761062 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757244 2541 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:21:36.761062 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757249 2541 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:21:36.761062 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757253 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:21:36.761062 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757258 2541 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:21:36.761062 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757262 2541 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:21:36.761062 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757267 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:21:36.761062 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757272 2541 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:21:36.761062 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757276 2541 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:21:36.761062 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757281 2541 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:21:36.761062 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757285 2541 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:21:36.761062 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757291 2541 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:21:36.762082 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757297 2541 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:21:36.762082 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757302 2541 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:21:36.762082 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757308 2541 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:21:36.762082 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757312 2541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:21:36.762082 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757317 2541 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:21:36.762082 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757323 2541 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:21:36.762082 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757327 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:21:36.762082 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757332 2541 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:21:36.762082 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757338 2541 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:21:36.762082 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757343 2541 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:21:36.762082 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757348 2541 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:21:36.762082 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757375 2541 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:21:36.762082 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757383 2541 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:21:36.762082 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757389 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:21:36.762082 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757394 2541 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:21:36.762082 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757399 2541 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:21:36.762082 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757403 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:21:36.762082 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757408 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:21:36.762082 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757413 2541 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:21:36.762711 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757417 2541 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:21:36.762711 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757422 2541 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:21:36.762711 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757426 2541 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:21:36.762711 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757432 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:21:36.762711 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757437 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:21:36.762711 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757442 2541 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:21:36.762711 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757446 2541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:21:36.762711 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757451 2541 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:21:36.762711 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757456 2541 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:21:36.762711 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757461 2541 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:21:36.762711 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757465 2541 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:21:36.762711 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757470 2541 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:21:36.762711 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757474 2541 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:21:36.762711 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757479 2541 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:21:36.762711 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757484 2541 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:21:36.762711 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757488 2541 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:21:36.762711 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757493 2541 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:21:36.762711 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757497 2541 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:21:36.762711 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757502 2541 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:21:36.763495 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757507 2541 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:21:36.763495 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757512 2541 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:21:36.763495 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757517 2541 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:21:36.763495 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757521 2541 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:21:36.763495 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757526 2541 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:21:36.763495 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757532 2541 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:21:36.763495 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757536 2541 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:21:36.763495 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757541 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:21:36.763495 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757546 2541 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:21:36.763495 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757551 2541 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:21:36.763495 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757555 2541 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:21:36.763495 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757560 2541 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:21:36.763495 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757564 2541 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:21:36.763495 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.757569 2541 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:21:36.763495 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758409 2541 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:21:36.763495 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758419 2541 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:21:36.763495 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758425 2541 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:21:36.763495 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758430 2541 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:21:36.763495 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758435 2541 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:21:36.763495 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758440 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:21:36.763970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758444 2541 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:21:36.763970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758448 2541 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:21:36.763970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758453 2541 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:21:36.763970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758457 2541 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:21:36.763970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758462 2541 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:21:36.763970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758467 2541 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:21:36.763970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758473 2541 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:21:36.763970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758478 2541 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:21:36.763970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758483 2541 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:21:36.763970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758487 2541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:21:36.763970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758492 2541 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:21:36.763970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758496 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:21:36.763970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758501 2541 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:21:36.763970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758506 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:21:36.763970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758511 2541 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:21:36.763970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758515 2541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:21:36.763970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758520 2541 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:21:36.763970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758525 2541 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:21:36.763970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758529 2541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:21:36.763970 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758534 2541 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:21:36.764527 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758538 2541 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:21:36.764527 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758543 2541 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:21:36.764527 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758548 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:21:36.764527 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758552 2541 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:21:36.764527 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758558 2541 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:21:36.764527 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758565 2541 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:21:36.764527 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758572 2541 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:21:36.764527 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758578 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:21:36.764527 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758584 2541 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:21:36.764527 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758589 2541 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:21:36.764527 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758594 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:21:36.764527 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758599 2541 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:21:36.764527 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758604 2541 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:21:36.764527 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758608 2541 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:21:36.764527 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758613 2541 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:21:36.764527 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758617 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:21:36.764527 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758624 2541 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:21:36.764527 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758630 2541 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:21:36.764527 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758636 2541 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:21:36.765002 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758641 2541 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:21:36.765002 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758645 2541 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:21:36.765002 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758650 2541 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:21:36.765002 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758654 2541 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:21:36.765002 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758659 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:21:36.765002 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758663 2541 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:21:36.765002 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758668 2541 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:21:36.765002 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758673 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:21:36.765002 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758677 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:21:36.765002 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758682 2541 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:21:36.765002 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758687 2541 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:21:36.765002 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758691 2541 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:21:36.765002 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758696 2541 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:21:36.765002 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758700 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:21:36.765002 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758704 2541 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:21:36.765002 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758709 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:21:36.765002 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758715 2541 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:21:36.765002 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758720 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:21:36.765002 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758725 2541 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:21:36.765002 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758730 2541 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:21:36.765550 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758735 2541 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:21:36.765550 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758739 2541 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:21:36.765550 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758744 2541 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:21:36.765550 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758750 2541 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:21:36.765550 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758754 2541 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:21:36.765550 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758759 2541 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:21:36.765550 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758763 2541 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:21:36.765550 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758769 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:21:36.765550 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758774 2541 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:21:36.765550 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758778 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:21:36.765550 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758783 2541 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:21:36.765550 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758788 2541 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:21:36.765550 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758793 2541 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:21:36.765550 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758798 2541 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:21:36.765550 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758803 2541 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:21:36.765550 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758807 2541 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:21:36.765550 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758812 2541 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:21:36.765550 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758816 2541 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:21:36.765550 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758821 2541 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:21:36.766024 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758825 2541 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:21:36.766024 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758830 2541 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:21:36.766024 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758834 2541 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:21:36.766024 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758838 2541 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:21:36.766024 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758844 2541 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:21:36.766024 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758848 2541 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:21:36.766024 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758853 2541 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:21:36.766024 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758857 2541 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:21:36.766024 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758862 2541 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:21:36.766024 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758867 2541 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:21:36.766024 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758871 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:21:36.766024 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758876 2541 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:21:36.766024 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758881 2541 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:21:36.766024 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758886 2541 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:21:36.766024 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758891 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:21:36.766024 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758895 2541 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:21:36.766024 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758899 2541 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:21:36.766024 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758905 2541 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:21:36.766024 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758910 2541 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:21:36.766024 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758915 2541 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:21:36.766516 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758921 2541 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:21:36.766516 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.758926 2541 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:21:36.766516 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759693 2541 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 22:21:36.766516 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759719 2541 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 22:21:36.766516 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759733 2541 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 22:21:36.766516 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759741 2541 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 22:21:36.766516 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759748 2541 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 22:21:36.766516 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759754 2541 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 22:21:36.766516 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759761 2541 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 22:21:36.766516 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759768 2541 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 22:21:36.766516 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759774 2541 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 22:21:36.766516 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759780 2541 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 22:21:36.766516 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759785 2541 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 22:21:36.766516 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759791 2541 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 22:21:36.766516 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759797 2541 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 22:21:36.766516 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759802 2541 flags.go:64] FLAG: --cgroup-root="" Nov 03 22:21:36.766516 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759807 2541 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 22:21:36.766516 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759812 2541 flags.go:64] FLAG: --client-ca-file="" Nov 03 22:21:36.766516 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759817 2541 flags.go:64] FLAG: --cloud-provider="external" Nov 03 22:21:36.766516 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759822 2541 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 22:21:36.766516 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759829 2541 flags.go:64] FLAG: --cluster-domain="" Nov 03 22:21:36.766516 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759834 2541 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 22:21:36.766516 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759839 2541 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 22:21:36.767049 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759845 2541 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 22:21:36.767049 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759851 2541 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 22:21:36.767049 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759859 2541 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 22:21:36.767049 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759864 2541 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 22:21:36.767049 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759869 2541 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 22:21:36.767049 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759875 2541 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 22:21:36.767049 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759880 2541 flags.go:64] FLAG: --contention-profiling="false" Nov 03 22:21:36.767049 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759887 2541 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 22:21:36.767049 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759892 2541 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 22:21:36.767049 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759900 2541 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 22:21:36.767049 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759905 2541 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 22:21:36.767049 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759912 2541 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 22:21:36.767049 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759923 2541 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 22:21:36.767049 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759928 2541 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 22:21:36.767049 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759933 2541 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 22:21:36.767049 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759938 2541 flags.go:64] FLAG: --enable-server="true" Nov 03 22:21:36.767049 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759944 2541 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 22:21:36.767049 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759960 2541 flags.go:64] FLAG: --event-burst="100" Nov 03 22:21:36.767049 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759966 2541 flags.go:64] FLAG: --event-qps="50" Nov 03 22:21:36.767049 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759972 2541 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 22:21:36.767049 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759977 2541 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 22:21:36.767049 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759982 2541 flags.go:64] FLAG: --eviction-hard="" Nov 03 22:21:36.767049 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.759996 2541 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 22:21:36.767049 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760001 2541 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 22:21:36.767750 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760007 2541 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 22:21:36.767750 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760012 2541 flags.go:64] FLAG: --eviction-soft="" Nov 03 22:21:36.767750 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760017 2541 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 22:21:36.767750 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760023 2541 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 22:21:36.767750 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760028 2541 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 22:21:36.767750 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760033 2541 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 22:21:36.767750 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760037 2541 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 22:21:36.767750 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760043 2541 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 22:21:36.767750 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760048 2541 flags.go:64] FLAG: --feature-gates="" Nov 03 22:21:36.767750 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760055 2541 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 22:21:36.767750 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760060 2541 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 22:21:36.767750 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760065 2541 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 22:21:36.767750 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760071 2541 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 22:21:36.767750 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760076 2541 flags.go:64] FLAG: --healthz-port="10248" Nov 03 22:21:36.767750 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760081 2541 flags.go:64] FLAG: --help="false" Nov 03 22:21:36.767750 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760086 2541 flags.go:64] FLAG: --hostname-override="ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:21:36.767750 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760093 2541 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 22:21:36.767750 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760101 2541 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 22:21:36.767750 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760107 2541 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 22:21:36.767750 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760113 2541 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 22:21:36.767750 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760120 2541 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 22:21:36.767750 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760125 2541 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 22:21:36.767750 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760131 2541 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 22:21:36.768291 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760136 2541 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 22:21:36.768291 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760141 2541 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 22:21:36.768291 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760146 2541 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 22:21:36.768291 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760152 2541 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 22:21:36.768291 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760156 2541 flags.go:64] FLAG: --kube-reserved="" Nov 03 22:21:36.768291 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760162 2541 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 22:21:36.768291 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760166 2541 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 22:21:36.768291 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760171 2541 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 22:21:36.768291 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760176 2541 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 22:21:36.768291 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760181 2541 flags.go:64] FLAG: --lock-file="" Nov 03 22:21:36.768291 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760186 2541 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 22:21:36.768291 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760191 2541 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 22:21:36.768291 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760196 2541 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 22:21:36.768291 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760206 2541 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 22:21:36.768291 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760211 2541 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 22:21:36.768291 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760216 2541 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 22:21:36.768291 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760221 2541 flags.go:64] FLAG: --logging-format="text" Nov 03 22:21:36.768291 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760226 2541 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 22:21:36.768291 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760231 2541 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 22:21:36.768291 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760237 2541 flags.go:64] FLAG: --manifest-url="" Nov 03 22:21:36.768291 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760242 2541 flags.go:64] FLAG: --manifest-url-header="" Nov 03 22:21:36.768291 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760250 2541 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 22:21:36.768291 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760255 2541 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 22:21:36.768291 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760262 2541 flags.go:64] FLAG: --max-pods="110" Nov 03 22:21:36.768291 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760267 2541 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 22:21:36.768899 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760272 2541 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 22:21:36.768899 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760288 2541 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 22:21:36.768899 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760294 2541 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 22:21:36.768899 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760300 2541 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 22:21:36.768899 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760306 2541 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 22:21:36.768899 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760311 2541 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 22:21:36.768899 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760325 2541 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 22:21:36.768899 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760330 2541 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 22:21:36.768899 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760335 2541 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 22:21:36.768899 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760341 2541 flags.go:64] FLAG: --pod-cidr="" Nov 03 22:21:36.768899 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760346 2541 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 22:21:36.768899 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760350 2541 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 22:21:36.768899 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760373 2541 flags.go:64] FLAG: --pods-per-core="0" Nov 03 22:21:36.768899 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760378 2541 flags.go:64] FLAG: --port="10250" Nov 03 22:21:36.768899 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760383 2541 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 22:21:36.768899 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760388 2541 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0a97294caf4c6a653" Nov 03 22:21:36.768899 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760394 2541 flags.go:64] FLAG: --qos-reserved="" Nov 03 22:21:36.768899 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760399 2541 flags.go:64] FLAG: --read-only-port="10255" Nov 03 22:21:36.768899 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760405 2541 flags.go:64] FLAG: --register-node="true" Nov 03 22:21:36.768899 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760410 2541 flags.go:64] FLAG: --register-with-taints="" Nov 03 22:21:36.768899 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760416 2541 flags.go:64] FLAG: --registry-burst="10" Nov 03 22:21:36.768899 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760421 2541 flags.go:64] FLAG: --registry-qps="5" Nov 03 22:21:36.768899 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760426 2541 flags.go:64] FLAG: --reserved-cpus="" Nov 03 22:21:36.768899 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760431 2541 flags.go:64] FLAG: --reserved-memory="" Nov 03 22:21:36.768899 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760438 2541 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 22:21:36.768899 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760443 2541 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 22:21:36.769573 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760448 2541 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 22:21:36.769573 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760454 2541 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 22:21:36.769573 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760459 2541 flags.go:64] FLAG: --runonce="false" Nov 03 22:21:36.769573 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760464 2541 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 22:21:36.769573 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760469 2541 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 22:21:36.769573 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760475 2541 flags.go:64] FLAG: --seccomp-default="false" Nov 03 22:21:36.769573 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760480 2541 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 22:21:36.769573 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760486 2541 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 22:21:36.769573 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760500 2541 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 22:21:36.769573 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760506 2541 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 22:21:36.769573 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760516 2541 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 22:21:36.769573 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760523 2541 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 22:21:36.769573 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760529 2541 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 22:21:36.769573 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760535 2541 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 22:21:36.769573 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760540 2541 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 22:21:36.769573 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760545 2541 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 22:21:36.769573 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760550 2541 flags.go:64] FLAG: --system-cgroups="" Nov 03 22:21:36.769573 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760555 2541 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 22:21:36.769573 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760565 2541 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 22:21:36.769573 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760571 2541 flags.go:64] FLAG: --tls-cert-file="" Nov 03 22:21:36.769573 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760576 2541 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 22:21:36.769573 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760583 2541 flags.go:64] FLAG: --tls-min-version="" Nov 03 22:21:36.769573 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760588 2541 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 22:21:36.769573 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760592 2541 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 22:21:36.769573 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760597 2541 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 22:21:36.770160 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760602 2541 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 22:21:36.770160 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760608 2541 flags.go:64] FLAG: --v="2" Nov 03 22:21:36.770160 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760621 2541 flags.go:64] FLAG: --version="false" Nov 03 22:21:36.770160 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760628 2541 flags.go:64] FLAG: --vmodule="" Nov 03 22:21:36.770160 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760635 2541 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 22:21:36.770160 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.760640 2541 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 22:21:36.770160 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.760859 2541 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:21:36.770160 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.760868 2541 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:21:36.770160 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.760876 2541 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:21:36.770160 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.760883 2541 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:21:36.770160 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.760889 2541 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:21:36.770160 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.760895 2541 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:21:36.770160 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.760900 2541 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:21:36.770160 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.760907 2541 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:21:36.770160 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.760914 2541 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:21:36.770160 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.760919 2541 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:21:36.770160 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.760925 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:21:36.770160 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.760932 2541 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:21:36.770160 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.760937 2541 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:21:36.770160 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.760941 2541 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:21:36.770160 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.760946 2541 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:21:36.770679 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.760951 2541 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:21:36.770679 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.760955 2541 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:21:36.770679 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.760960 2541 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:21:36.770679 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.760965 2541 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:21:36.770679 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.760969 2541 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:21:36.770679 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.760974 2541 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:21:36.770679 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.760978 2541 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:21:36.770679 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.760983 2541 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:21:36.770679 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.760988 2541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:21:36.770679 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.760993 2541 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:21:36.770679 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.760997 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:21:36.770679 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761002 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:21:36.770679 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761006 2541 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:21:36.770679 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761011 2541 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:21:36.770679 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761016 2541 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:21:36.770679 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761020 2541 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:21:36.770679 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761025 2541 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:21:36.770679 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761029 2541 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:21:36.770679 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761034 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:21:36.770679 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761039 2541 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:21:36.771146 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761044 2541 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:21:36.771146 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761048 2541 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:21:36.771146 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761053 2541 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:21:36.771146 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761058 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:21:36.771146 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761062 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:21:36.771146 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761068 2541 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:21:36.771146 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761073 2541 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:21:36.771146 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761078 2541 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:21:36.771146 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761085 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:21:36.771146 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761089 2541 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:21:36.771146 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761094 2541 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:21:36.771146 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761099 2541 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:21:36.771146 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761104 2541 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:21:36.771146 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761109 2541 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:21:36.771146 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761114 2541 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:21:36.771146 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761119 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:21:36.771146 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761124 2541 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:21:36.771146 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761128 2541 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:21:36.771146 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761132 2541 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:21:36.771653 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761137 2541 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:21:36.771653 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761142 2541 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:21:36.771653 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761147 2541 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:21:36.771653 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761151 2541 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:21:36.771653 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761156 2541 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:21:36.771653 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761161 2541 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:21:36.771653 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761166 2541 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:21:36.771653 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761170 2541 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:21:36.771653 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761174 2541 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:21:36.771653 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761179 2541 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:21:36.771653 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761183 2541 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:21:36.771653 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761188 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:21:36.771653 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761193 2541 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:21:36.771653 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761197 2541 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:21:36.771653 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761202 2541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:21:36.771653 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761207 2541 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:21:36.771653 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761211 2541 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:21:36.771653 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761216 2541 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:21:36.771653 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761224 2541 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:21:36.771653 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761229 2541 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:21:36.772145 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761234 2541 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:21:36.772145 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761240 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:21:36.772145 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761246 2541 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:21:36.772145 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761251 2541 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:21:36.772145 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761255 2541 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:21:36.772145 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761260 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:21:36.772145 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761265 2541 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:21:36.772145 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761270 2541 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:21:36.772145 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761274 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:21:36.772145 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761279 2541 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:21:36.772145 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761284 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:21:36.772145 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761288 2541 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:21:36.772145 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761293 2541 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:21:36.772145 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761298 2541 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:21:36.772145 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761302 2541 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:21:36.772145 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761307 2541 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:21:36.772145 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761313 2541 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:21:36.772145 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761317 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:21:36.772145 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761322 2541 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:21:36.772145 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761327 2541 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:21:36.772683 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761331 2541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:21:36.772683 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761336 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:21:36.772683 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761340 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:21:36.772683 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761345 2541 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:21:36.772683 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761350 2541 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:21:36.772683 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761374 2541 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:21:36.772683 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761379 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:21:36.772683 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761384 2541 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:21:36.772683 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761389 2541 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:21:36.772683 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761393 2541 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:21:36.772683 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761400 2541 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:21:36.772683 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761404 2541 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:21:36.772683 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.761450 2541 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:21:36.772683 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761507 2541 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:21:36.772683 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761518 2541 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:21:36.772683 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761524 2541 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:21:36.773085 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761529 2541 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:21:36.773085 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761534 2541 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:21:36.773085 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761539 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:21:36.773085 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761543 2541 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:21:36.773085 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761548 2541 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:21:36.773085 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761553 2541 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:21:36.773085 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761557 2541 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:21:36.773085 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761561 2541 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:21:36.773085 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761565 2541 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:21:36.773085 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761570 2541 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:21:36.773085 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761574 2541 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:21:36.773085 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761578 2541 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:21:36.773085 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761583 2541 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:21:36.773085 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761587 2541 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:21:36.773085 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761592 2541 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:21:36.773085 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761596 2541 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:21:36.773085 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761600 2541 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:21:36.773085 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761605 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:21:36.773085 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761609 2541 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:21:36.773561 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761613 2541 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:21:36.773561 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761619 2541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:21:36.773561 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761624 2541 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:21:36.773561 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761629 2541 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:21:36.773561 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761633 2541 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:21:36.773561 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761638 2541 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:21:36.773561 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761642 2541 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:21:36.773561 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761648 2541 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:21:36.773561 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761653 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:21:36.773561 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761657 2541 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:21:36.773561 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761662 2541 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:21:36.773561 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761668 2541 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:21:36.773561 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761673 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:21:36.773561 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761678 2541 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:21:36.773561 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761683 2541 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:21:36.773561 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761688 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:21:36.773561 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761692 2541 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:21:36.773561 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761697 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:21:36.773561 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761701 2541 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:21:36.774005 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761706 2541 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:21:36.774005 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761711 2541 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:21:36.774005 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761715 2541 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:21:36.774005 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761720 2541 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:21:36.774005 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761726 2541 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:21:36.774005 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761730 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:21:36.774005 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761735 2541 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:21:36.774005 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761741 2541 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:21:36.774005 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761745 2541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:21:36.774005 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761750 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:21:36.774005 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761755 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:21:36.774005 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761760 2541 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:21:36.774005 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761764 2541 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:21:36.774005 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761769 2541 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:21:36.774005 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761774 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:21:36.774005 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761778 2541 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:21:36.774005 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761783 2541 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:21:36.774005 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761787 2541 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:21:36.774005 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761792 2541 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:21:36.774528 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761796 2541 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:21:36.774528 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761803 2541 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:21:36.774528 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761810 2541 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:21:36.774528 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761817 2541 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:21:36.774528 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761824 2541 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:21:36.774528 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761831 2541 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:21:36.774528 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761837 2541 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:21:36.774528 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761842 2541 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:21:36.774528 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761857 2541 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:21:36.774528 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761862 2541 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:21:36.774528 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761866 2541 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:21:36.774528 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761871 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:21:36.774528 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761876 2541 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:21:36.774528 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761880 2541 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:21:36.774528 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761885 2541 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:21:36.774528 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761889 2541 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:21:36.774528 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761894 2541 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:21:36.774528 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761899 2541 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:21:36.774528 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761904 2541 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:21:36.774972 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761908 2541 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:21:36.774972 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761913 2541 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:21:36.774972 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761917 2541 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:21:36.774972 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761922 2541 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:21:36.774972 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761926 2541 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:21:36.774972 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761931 2541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:21:36.774972 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761936 2541 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:21:36.774972 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761940 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:21:36.774972 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761945 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:21:36.774972 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761950 2541 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:21:36.774972 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761955 2541 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:21:36.774972 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761959 2541 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:21:36.774972 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761963 2541 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:21:36.774972 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761968 2541 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:21:36.774972 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761974 2541 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:21:36.774972 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761979 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:21:36.774972 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761984 2541 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:21:36.774972 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761988 2541 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:21:36.774972 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761994 2541 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:21:36.774972 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.761999 2541 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:21:36.775472 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.762004 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:21:36.775472 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.762008 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:21:36.775472 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.762021 2541 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:21:36.775472 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.762028 2541 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:21:36.775472 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.762033 2541 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:21:36.775472 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.762038 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:21:36.775472 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.762042 2541 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:21:36.775472 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.769210 2541 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 22:21:36.775472 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.769227 2541 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 22:21:36.775472 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.769240 2541 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 22:21:36.775472 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.769245 2541 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 22:21:36.775472 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769303 2541 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:21:36.775472 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769308 2541 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:21:36.775472 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769311 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:21:36.775472 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769314 2541 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:21:36.775472 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769317 2541 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:21:36.775472 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769320 2541 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:21:36.775472 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769323 2541 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:21:36.775918 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769327 2541 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:21:36.775918 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769331 2541 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:21:36.775918 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769334 2541 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:21:36.775918 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769337 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:21:36.775918 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769339 2541 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:21:36.775918 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769342 2541 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:21:36.775918 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769345 2541 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:21:36.775918 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769348 2541 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:21:36.775918 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769350 2541 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:21:36.775918 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769353 2541 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:21:36.775918 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769371 2541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:21:36.775918 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769374 2541 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:21:36.775918 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769377 2541 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:21:36.775918 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769379 2541 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:21:36.775918 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769383 2541 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:21:36.775918 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769386 2541 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:21:36.775918 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769389 2541 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:21:36.775918 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769391 2541 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:21:36.775918 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769394 2541 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:21:36.776372 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769396 2541 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:21:36.776372 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769399 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:21:36.776372 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769401 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:21:36.776372 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769405 2541 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:21:36.776372 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769408 2541 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:21:36.776372 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769411 2541 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:21:36.776372 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769414 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:21:36.776372 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769416 2541 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:21:36.776372 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769418 2541 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:21:36.776372 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769421 2541 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:21:36.776372 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769424 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:21:36.776372 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769426 2541 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:21:36.776372 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769428 2541 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:21:36.776372 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769431 2541 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:21:36.776372 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769433 2541 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:21:36.776372 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769436 2541 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:21:36.776372 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769438 2541 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:21:36.776372 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769441 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:21:36.776372 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769445 2541 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:21:36.776822 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769449 2541 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:21:36.776822 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769451 2541 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:21:36.776822 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769455 2541 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:21:36.776822 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769457 2541 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:21:36.776822 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769460 2541 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:21:36.776822 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769462 2541 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:21:36.776822 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769465 2541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:21:36.776822 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769467 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:21:36.776822 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769470 2541 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:21:36.776822 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769473 2541 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:21:36.776822 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769475 2541 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:21:36.776822 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769478 2541 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:21:36.776822 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769480 2541 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:21:36.776822 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769483 2541 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:21:36.776822 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769485 2541 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:21:36.776822 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769488 2541 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:21:36.776822 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769491 2541 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:21:36.776822 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769500 2541 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:21:36.776822 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769503 2541 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:21:36.776822 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769505 2541 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:21:36.777284 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769508 2541 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:21:36.777284 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769510 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:21:36.777284 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769513 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:21:36.777284 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769515 2541 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:21:36.777284 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769518 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:21:36.777284 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769520 2541 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:21:36.777284 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769522 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:21:36.777284 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769525 2541 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:21:36.777284 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769527 2541 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:21:36.777284 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769530 2541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:21:36.777284 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769532 2541 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:21:36.777284 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769535 2541 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:21:36.777284 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769537 2541 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:21:36.777284 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769540 2541 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:21:36.777284 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769543 2541 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:21:36.777284 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769545 2541 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:21:36.777284 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769548 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:21:36.777284 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769550 2541 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:21:36.777284 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769553 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:21:36.777284 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769555 2541 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:21:36.777871 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769558 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:21:36.777871 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769561 2541 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:21:36.777871 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769563 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:21:36.777871 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769566 2541 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:21:36.777871 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769568 2541 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:21:36.777871 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769571 2541 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:21:36.777871 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769573 2541 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:21:36.777871 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769576 2541 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:21:36.777871 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769579 2541 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:21:36.777871 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769582 2541 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:21:36.777871 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769590 2541 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:21:36.777871 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769593 2541 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:21:36.777871 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769595 2541 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:21:36.777871 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769598 2541 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:21:36.777871 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769600 2541 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:21:36.777871 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769603 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:21:36.777871 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769605 2541 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:21:36.777871 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769607 2541 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:21:36.777871 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769610 2541 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:21:36.778326 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769612 2541 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:21:36.778326 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769615 2541 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:21:36.778326 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.769642 2541 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:21:36.778326 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769825 2541 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:21:36.778326 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769829 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:21:36.778326 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769832 2541 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:21:36.778326 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769835 2541 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:21:36.778326 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769838 2541 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:21:36.778326 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769841 2541 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:21:36.778326 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769843 2541 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:21:36.778326 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769846 2541 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:21:36.778326 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769848 2541 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:21:36.778326 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769851 2541 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:21:36.778326 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769853 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:21:36.778326 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769856 2541 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:21:36.778326 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769858 2541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:21:36.778944 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769861 2541 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:21:36.778944 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769863 2541 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:21:36.778944 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769866 2541 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:21:36.778944 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769868 2541 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:21:36.778944 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769871 2541 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:21:36.778944 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769873 2541 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:21:36.778944 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769876 2541 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:21:36.778944 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769879 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:21:36.778944 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769887 2541 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:21:36.778944 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769890 2541 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:21:36.778944 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769892 2541 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:21:36.778944 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769895 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:21:36.778944 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769898 2541 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:21:36.778944 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769900 2541 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:21:36.778944 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769903 2541 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:21:36.778944 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769905 2541 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:21:36.778944 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769907 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:21:36.778944 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769910 2541 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:21:36.778944 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769912 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:21:36.778944 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769915 2541 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:21:36.779755 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769917 2541 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:21:36.779755 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769920 2541 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:21:36.779755 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769922 2541 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:21:36.779755 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769925 2541 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:21:36.779755 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769927 2541 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:21:36.779755 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769930 2541 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:21:36.779755 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769932 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:21:36.779755 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769934 2541 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:21:36.779755 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769937 2541 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:21:36.779755 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769939 2541 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:21:36.779755 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769942 2541 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:21:36.779755 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769944 2541 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:21:36.779755 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769947 2541 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:21:36.779755 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769949 2541 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:21:36.779755 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769951 2541 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:21:36.779755 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769954 2541 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:21:36.779755 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769956 2541 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:21:36.779755 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769959 2541 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:21:36.779755 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769961 2541 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:21:36.779755 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769964 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:21:36.780273 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769966 2541 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:21:36.780273 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769974 2541 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:21:36.780273 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769977 2541 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:21:36.780273 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769979 2541 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:21:36.780273 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769982 2541 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:21:36.780273 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769984 2541 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:21:36.780273 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769987 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:21:36.780273 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769989 2541 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:21:36.780273 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769991 2541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:21:36.780273 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769994 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:21:36.780273 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769996 2541 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:21:36.780273 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.769999 2541 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:21:36.780273 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770001 2541 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:21:36.780273 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770004 2541 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:21:36.780273 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770006 2541 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:21:36.780273 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770009 2541 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:21:36.780273 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770011 2541 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:21:36.780273 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770014 2541 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:21:36.780273 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770017 2541 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:21:36.780763 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770020 2541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:21:36.780763 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770023 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:21:36.780763 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770026 2541 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:21:36.780763 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770034 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:21:36.780763 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770036 2541 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:21:36.780763 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770039 2541 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:21:36.780763 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770041 2541 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:21:36.780763 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770044 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:21:36.780763 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770046 2541 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:21:36.780763 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770048 2541 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:21:36.780763 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770051 2541 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:21:36.780763 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770054 2541 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:21:36.780763 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770057 2541 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:21:36.780763 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770059 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:21:36.780763 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770061 2541 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:21:36.780763 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770069 2541 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:21:36.780763 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770072 2541 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:21:36.780763 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770075 2541 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:21:36.780763 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770077 2541 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:21:36.781199 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770079 2541 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:21:36.781199 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770082 2541 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:21:36.781199 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770086 2541 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:21:36.781199 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770090 2541 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:21:36.781199 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770092 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:21:36.781199 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770095 2541 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:21:36.781199 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770097 2541 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:21:36.781199 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770100 2541 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:21:36.781199 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770102 2541 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:21:36.781199 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770105 2541 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:21:36.781199 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770107 2541 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:21:36.781199 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770110 2541 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:21:36.781199 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770112 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:21:36.781199 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770114 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:21:36.781199 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770117 2541 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:21:36.781199 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.770139 2541 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:21:36.781681 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770175 2541 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:21:36.781681 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770179 2541 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:21:36.781681 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770181 2541 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:21:36.781681 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770184 2541 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:21:36.781681 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770186 2541 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:21:36.781681 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770188 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:21:36.781681 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770193 2541 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:21:36.781681 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770196 2541 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:21:36.781681 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770198 2541 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:21:36.781681 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770201 2541 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:21:36.781681 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770203 2541 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:21:36.781681 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770206 2541 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:21:36.781681 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770208 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:21:36.781681 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770216 2541 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:21:36.781681 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770219 2541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:21:36.781681 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770222 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:21:36.781681 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770224 2541 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:21:36.781681 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770227 2541 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:21:36.781681 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770229 2541 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:21:36.782133 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770232 2541 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:21:36.782133 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770234 2541 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:21:36.782133 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770237 2541 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:21:36.782133 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770239 2541 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:21:36.782133 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770242 2541 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:21:36.782133 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770246 2541 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:21:36.782133 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770249 2541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:21:36.782133 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770252 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:21:36.782133 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770254 2541 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:21:36.782133 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770257 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:21:36.782133 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770259 2541 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:21:36.782133 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770261 2541 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:21:36.782133 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770264 2541 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:21:36.782133 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770266 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:21:36.782133 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770269 2541 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:21:36.782133 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770271 2541 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:21:36.782133 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770274 2541 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:21:36.782133 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770276 2541 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:21:36.782133 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770279 2541 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:21:36.782596 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770285 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:21:36.782596 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770288 2541 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:21:36.782596 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770290 2541 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:21:36.782596 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770293 2541 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:21:36.782596 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770295 2541 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:21:36.782596 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770298 2541 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:21:36.782596 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770300 2541 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:21:36.782596 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770303 2541 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:21:36.782596 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770314 2541 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:21:36.782596 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770318 2541 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:21:36.782596 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770321 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:21:36.782596 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770324 2541 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:21:36.782596 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770327 2541 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:21:36.782596 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770330 2541 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:21:36.782596 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770333 2541 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:21:36.782596 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770336 2541 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:21:36.782596 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770339 2541 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:21:36.782596 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770341 2541 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:21:36.782596 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770344 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:21:36.783073 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770346 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:21:36.783073 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770348 2541 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:21:36.783073 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770351 2541 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:21:36.783073 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770353 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:21:36.783073 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770369 2541 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:21:36.783073 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770371 2541 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:21:36.783073 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770374 2541 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:21:36.783073 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770376 2541 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:21:36.783073 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770379 2541 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:21:36.783073 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770381 2541 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:21:36.783073 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770383 2541 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:21:36.783073 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770386 2541 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:21:36.783073 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770388 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:21:36.783073 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770393 2541 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:21:36.783073 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770396 2541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:21:36.783073 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770398 2541 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:21:36.783073 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770401 2541 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:21:36.783073 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770403 2541 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:21:36.783073 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770406 2541 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:21:36.783073 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770409 2541 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:21:36.784735 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770411 2541 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:21:36.784735 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770414 2541 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:21:36.784735 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770422 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:21:36.784735 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770425 2541 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:21:36.784735 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770428 2541 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:21:36.784735 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770430 2541 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:21:36.784735 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770433 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:21:36.784735 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770435 2541 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:21:36.784735 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770438 2541 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:21:36.784735 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770440 2541 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:21:36.784735 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770443 2541 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:21:36.784735 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770445 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:21:36.784735 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770448 2541 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:21:36.784735 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770451 2541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:21:36.784735 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770453 2541 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:21:36.784735 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770456 2541 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:21:36.784735 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770458 2541 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:21:36.784735 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770460 2541 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:21:36.784735 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770463 2541 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:21:36.784735 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770466 2541 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:21:36.785285 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770469 2541 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:21:36.785285 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770471 2541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:21:36.785285 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770473 2541 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:21:36.785285 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770476 2541 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:21:36.785285 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770478 2541 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:21:36.785285 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770483 2541 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:21:36.785285 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770486 2541 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:21:36.785285 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770488 2541 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:21:36.785285 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:36.770490 2541 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:21:36.785285 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.770551 2541 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 22:21:36.785285 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.772537 2541 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 22:21:36.785285 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.773522 2541 server.go:1014] "Starting client certificate rotation" Nov 03 22:21:36.785285 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.773616 2541 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 22:21:36.785285 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.774258 2541 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 22:21:36.785285 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.775741 2541 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:21:36.785285 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.779793 2541 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:21:36.790470 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.790450 2541 log.go:25] "Validated CRI v1 runtime API" Nov 03 22:21:36.793666 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.793652 2541 log.go:25] "Validated CRI v1 image API" Nov 03 22:21:36.794470 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.794444 2541 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 22:21:36.797131 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.797103 2541 fs.go:135] Filesystem UUIDs: map[26b5d738-b058-4f16-92ca-cc05b3dcad21:/dev/nvme0n1p3 27b847a8-9336-408c-88e1-9c4c017547c8:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 03 22:21:36.797197 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.797130 2541 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 22:21:36.802278 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.802169 2541 manager.go:217] Machine: {Timestamp:2026-11-03 22:21:36.800858666 +0000 UTC m=+0.321553429 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103598 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2feb0695c7105c636a56e85bfc4670 SystemUUID:ec2feb06-95c7-105c-636a-56e85bfc4670 BootID:b8b014e0-7ed1-4ab1-a7b7-0ffee440ad2d Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:79:df:2e:8e:41 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:79:df:2e:8e:41 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2e:67:7f:b0:07:f6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 22:21:36.802380 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.802278 2541 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 22:21:36.802380 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.802376 2541 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 22:21:36.805329 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.805304 2541 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 22:21:36.805480 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.805330 2541 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-132.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 22:21:36.805523 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.805482 2541 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 22:21:36.805523 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.805491 2541 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 22:21:36.805579 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.805535 2541 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 22:21:36.805579 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.805554 2541 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 22:21:36.806318 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.806309 2541 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 22:21:36.809031 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.809019 2541 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 22:21:36.809148 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.809139 2541 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 22:21:36.809845 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.809835 2541 kubelet.go:493] "Attempting to sync node with API server" Nov 03 22:21:36.809910 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.809848 2541 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 22:21:36.809910 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.809860 2541 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 22:21:36.809910 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.809869 2541 kubelet.go:405] "Adding apiserver pod source" Nov 03 22:21:36.809910 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.809878 2541 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 22:21:36.810239 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.810227 2541 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 22:21:36.812435 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.812411 2541 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 22:21:36.817177 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.817163 2541 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 22:21:36.817925 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.817912 2541 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 22:21:36.817958 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.817944 2541 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 22:21:36.818135 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.818119 2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 22:21:36.818165 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.818152 2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 22:21:36.818201 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.818166 2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 22:21:36.818201 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.818180 2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 22:21:36.818201 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.818194 2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 22:21:36.818286 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.818208 2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 22:21:36.818583 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.818575 2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 22:21:36.818613 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.818585 2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 22:21:36.818613 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.818591 2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 22:21:36.818613 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.818597 2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 22:21:36.818613 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.818607 2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 22:21:36.818817 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.818615 2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 22:21:36.820141 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.820131 2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 22:21:36.820175 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.820146 2541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 22:21:36.823761 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.823749 2541 server.go:1263] "Started kubelet" Nov 03 22:21:36.823942 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.823902 2541 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 22:21:36.824046 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.823901 2541 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 22:21:36.824046 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.824030 2541 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 22:21:36.824547 ip-10-0-143-132 systemd[1]: Started Kubernetes Kubelet. Nov 03 22:21:36.827579 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.827556 2541 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 22:21:36.829389 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.829340 2541 server.go:317] "Adding debug handlers to kubelet server" Nov 03 22:21:36.832698 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:36.832677 2541 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 22:21:36.833493 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.833477 2541 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 22:21:36.833493 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.833482 2541 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 22:21:36.834010 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:36.833986 2541 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-132.us-west-1.compute.internal\" not found" Nov 03 22:21:36.834128 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.834116 2541 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 22:21:36.834293 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.834275 2541 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 22:21:36.834405 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.834308 2541 factory.go:55] Registering systemd factory Nov 03 22:21:36.834405 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.834116 2541 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 22:21:36.834405 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.834324 2541 factory.go:223] Registration of the systemd container factory successfully Nov 03 22:21:36.834715 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.834415 2541 reconstruct.go:97] "Volume reconstruction finished" Nov 03 22:21:36.834715 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.834424 2541 reconciler.go:29] "Reconciler: start to sync state" Nov 03 22:21:36.834715 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.834617 2541 factory.go:153] Registering CRI-O factory Nov 03 22:21:36.834715 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.834638 2541 factory.go:223] Registration of the crio container factory successfully Nov 03 22:21:36.834715 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.834685 2541 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 22:21:36.834715 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.834701 2541 factory.go:103] Registering Raw factory Nov 03 22:21:36.834715 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.834713 2541 manager.go:1199] Started watching for new ooms in manager Nov 03 22:21:36.835589 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.835572 2541 manager.go:319] Starting recovery of all containers Nov 03 22:21:36.845657 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.845636 2541 manager.go:324] Recovery completed Nov 03 22:21:36.849727 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.849715 2541 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:21:36.851147 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.851114 2541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:21:36.851213 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.851161 2541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:21:36.851213 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.851173 2541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 22:21:36.851780 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.851765 2541 cpu_manager.go:226] "Starting" policy="none" Nov 03 22:21:36.851780 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.851777 2541 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 22:21:36.851902 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.851793 2541 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 22:21:36.851902 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.851798 2541 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:21:36.852520 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.852505 2541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:21:36.852582 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.852532 2541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:21:36.852582 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.852543 2541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 22:21:36.852929 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.852917 2541 policy_none.go:50] "Start" Nov 03 22:21:36.852972 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.852934 2541 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 22:21:36.852972 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.852944 2541 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 22:21:36.853864 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.853853 2541 policy_none.go:44] "Start" Nov 03 22:21:36.900008 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.899884 2541 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 22:21:36.903922 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.903909 2541 manager.go:341] "Starting Device Plugin manager" Nov 03 22:21:36.903990 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:36.903940 2541 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 22:21:36.903990 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.903949 2541 server.go:87] "Starting device plugin registration server" Nov 03 22:21:36.904151 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.904139 2541 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 22:21:36.904228 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.904152 2541 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 22:21:36.904279 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.904242 2541 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 22:21:36.904324 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.904316 2541 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 22:21:36.904324 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.904324 2541 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 22:21:36.904882 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:36.904859 2541 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 22:21:36.904976 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:36.904906 2541 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-132.us-west-1.compute.internal\" not found" Nov 03 22:21:36.972624 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.972573 2541 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 22:21:36.973155 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.973137 2541 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-132.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 22:21:36.973241 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:36.973205 2541 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-132.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 22:21:36.973954 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.973938 2541 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 22:21:36.973999 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.973960 2541 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 22:21:36.973999 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:36.973977 2541 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 22:21:36.974075 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:36.974006 2541 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 22:21:36.974204 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:36.973128 2541 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-132.us-west-1.compute.internal.18e4a91dbefd4648 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-132.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-132.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 22:21:36.8237154 +0000 UTC m=+0.344410163,LastTimestamp:2026-11-03 22:21:36.8237154 +0000 UTC m=+0.344410163,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-132.us-west-1.compute.internal,}" Nov 03 22:21:37.004507 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.004457 2541 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:21:37.005219 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.005193 2541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:21:37.005309 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.005225 2541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:21:37.005309 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.005235 2541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 22:21:37.005309 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.005258 2541 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:21:37.033226 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:37.033180 2541 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 22:21:37.033325 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:37.033217 2541 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-132.us-west-1.compute.internal.18e4a91dc09fd745 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-132.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-132.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-132.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 22:21:36.851146565 +0000 UTC m=+0.371841334,LastTimestamp:2026-11-03 22:21:36.851146565 +0000 UTC m=+0.371841334,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-132.us-west-1.compute.internal,}" Nov 03 22:21:37.053215 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.053191 2541 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 22:21:37.053299 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.053260 2541 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kpf2w" Nov 03 22:21:37.074278 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.074256 2541 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-132.us-west-1.compute.internal"] Nov 03 22:21:37.074345 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.074311 2541 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:21:37.075149 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.075133 2541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:21:37.075246 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.075161 2541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:21:37.075246 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.075171 2541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 22:21:37.076405 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.076392 2541 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:21:37.076512 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.076496 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:21:37.076567 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.076537 2541 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:21:37.077092 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.077072 2541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:21:37.077092 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.077084 2541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:21:37.077209 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.077106 2541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:21:37.077209 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.077108 2541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:21:37.077209 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.077117 2541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 22:21:37.077209 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.077119 2541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 22:21:37.078187 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.078174 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:21:37.078238 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.078200 2541 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:21:37.078852 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.078838 2541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:21:37.078925 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.078863 2541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:21:37.078925 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.078875 2541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 22:21:37.107405 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:37.107383 2541 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-132.us-west-1.compute.internal\" not found" node="ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:21:37.110950 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:37.110933 2541 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-132.us-west-1.compute.internal\" not found" node="ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:21:37.134417 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.134397 2541 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:21:37.134884 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.134869 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9bb86a6441b6931e82229a352ca0e493-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal\" (UID: \"9bb86a6441b6931e82229a352ca0e493\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:21:37.134927 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.134909 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bb86a6441b6931e82229a352ca0e493-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal\" (UID: \"9bb86a6441b6931e82229a352ca0e493\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:21:37.134968 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.134936 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd09a9d6123011e89fe0e21d9dfdc2d8-config\") pod \"kube-apiserver-proxy-ip-10-0-143-132.us-west-1.compute.internal\" (UID: \"bd09a9d6123011e89fe0e21d9dfdc2d8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:21:37.182902 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.182877 2541 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:21:37.233282 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.233253 2541 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:21:37.233438 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.233427 2541 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:21:37.233479 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:37.233460 2541 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-132.us-west-1.compute.internal\": node \"ip-10-0-143-132.us-west-1.compute.internal\" not found" Nov 03 22:21:37.233661 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.233649 2541 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:21:37.235028 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.235015 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9bb86a6441b6931e82229a352ca0e493-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal\" (UID: \"9bb86a6441b6931e82229a352ca0e493\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:21:37.235075 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.235038 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bb86a6441b6931e82229a352ca0e493-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal\" (UID: \"9bb86a6441b6931e82229a352ca0e493\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:21:37.235075 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.235056 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd09a9d6123011e89fe0e21d9dfdc2d8-config\") pod \"kube-apiserver-proxy-ip-10-0-143-132.us-west-1.compute.internal\" (UID: \"bd09a9d6123011e89fe0e21d9dfdc2d8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:21:37.235137 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.235110 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9bb86a6441b6931e82229a352ca0e493-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal\" (UID: \"9bb86a6441b6931e82229a352ca0e493\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:21:37.235137 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.235127 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bb86a6441b6931e82229a352ca0e493-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal\" (UID: \"9bb86a6441b6931e82229a352ca0e493\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:21:37.235197 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.235132 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd09a9d6123011e89fe0e21d9dfdc2d8-config\") pod \"kube-apiserver-proxy-ip-10-0-143-132.us-west-1.compute.internal\" (UID: \"bd09a9d6123011e89fe0e21d9dfdc2d8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:21:37.311584 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.311514 2541 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:21:37.311707 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.311615 2541 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:21:37.367891 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.367864 2541 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:21:37.411108 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.411087 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:21:37.413916 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.413901 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:21:37.773301 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.773268 2541 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 22:21:37.773815 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.773444 2541 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 22:21:37.773815 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.773452 2541 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 22:21:37.773815 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:37.773504 2541 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a6c0d62e96a584816a3ee7bc88cce2ee-6000754553a9ad3e.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-143-132.us-west-1.compute.internal.18e4a91dc0a04e2a\": read tcp 10.0.143.132:52190->3.15.56.150:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-143-132.us-west-1.compute.internal.18e4a91dc0a04e2a default 5146 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-132.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-132.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-132.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 22:21:36 +0000 UTC,LastTimestamp:2026-11-03 22:21:37.005239414 +0000 UTC m=+0.525934178,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-132.us-west-1.compute.internal,}" Nov 03 22:21:37.810518 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.810491 2541 apiserver.go:52] "Watching apiserver" Nov 03 22:21:37.834265 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:37.834215 2541 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 22:21:38.056135 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.056052 2541 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 22:16:37 +0000 UTC" deadline="2028-06-09 01:33:47.715729398 +0000 UTC" Nov 03 22:21:38.056135 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.056086 2541 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13995h12m9.659647011s" Nov 03 22:21:38.087087 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.087060 2541 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 22:21:38.087411 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.087394 2541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-dwf9b","openshift-image-registry/node-ca-t5l5t","openshift-multus/multus-4gfds","openshift-network-operator/iptables-alerter-p55z4","kube-system/global-pull-secret-syncer-h8jxb","kube-system/konnectivity-agent-g6qjz","openshift-network-diagnostics/network-check-target-jkjnj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng","openshift-cluster-node-tuning-operator/tuned-j9x58","openshift-dns/node-resolver-k4h5g","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-sq9mk","kube-system/kube-apiserver-proxy-ip-10-0-143-132.us-west-1.compute.internal","openshift-multus/network-metrics-daemon-vqppk"] Nov 03 22:21:38.088892 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.088874 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.089942 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.089923 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t5l5t" Nov 03 22:21:38.090995 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.090974 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.091998 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.091985 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-p55z4" Nov 03 22:21:38.093016 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.092999 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:21:38.093086 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:38.093067 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8jxb" podUID="b6154c5a-4925-4c74-8db4-0acdb525b743" Nov 03 22:21:38.093901 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.093890 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g6qjz" Nov 03 22:21:38.094930 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.094917 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:21:38.094978 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:38.094964 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjnj" podUID="29b82faa-1aa1-4760-a9f6-1cbeb9427094" Nov 03 22:21:38.096077 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.096057 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:21:38.097277 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.097259 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.098324 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.098311 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k4h5g" Nov 03 22:21:38.099523 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.099509 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:21:38.100813 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.100532 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:21:38.100813 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:38.100584 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqppk" podUID="c027d694-64cc-46a2-b4e7-49b117f696d3" Nov 03 22:21:38.135569 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.135550 2541 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 22:21:38.142338 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.142317 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-m2wh6\"" Nov 03 22:21:38.142607 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.142583 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 22:21:38.142689 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.142674 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 22:21:38.142938 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.142923 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 22:21:38.143011 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.142972 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 22:21:38.143075 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.143062 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 22:21:38.143129 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.143088 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 22:21:38.143308 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.143287 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 22:21:38.143593 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.143576 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 22:21:38.143662 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.143603 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 22:21:38.143662 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.143621 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 22:21:38.143761 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.143733 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 22:21:38.143871 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.143855 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 22:21:38.143950 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.143934 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:21:38.144001 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.143863 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-b2scz\"" Nov 03 22:21:38.144055 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.144006 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-dx5sz\"" Nov 03 22:21:38.144193 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.144175 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-host\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.144248 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.144216 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1b2df441-74d4-435f-a56a-5eff3ec4575d-env-overrides\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.144306 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.144247 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b2df441-74d4-435f-a56a-5eff3ec4575d-ovn-node-metrics-cert\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.144306 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.144261 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 22:21:38.144306 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.144278 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4n2l\" (UniqueName: \"kubernetes.io/projected/1b2df441-74d4-435f-a56a-5eff3ec4575d-kube-api-access-s4n2l\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.144469 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.144311 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e68337a-9274-466d-a238-082fdf86f4cb-cni-binary-copy\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.144469 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.144375 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0ba09d3-25f2-422b-b5e2-357aabcaf81c-konnectivity-ca\") pod \"konnectivity-agent-g6qjz\" (UID: \"b0ba09d3-25f2-422b-b5e2-357aabcaf81c\") " pod="kube-system/konnectivity-agent-g6qjz" Nov 03 22:21:38.144469 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.144413 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-sys\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.144469 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.144454 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3f63f886-379d-4636-8fb3-21b932a67831-hosts-file\") pod \"node-resolver-k4h5g\" (UID: \"3f63f886-379d-4636-8fb3-21b932a67831\") " pod="openshift-dns/node-resolver-k4h5g" Nov 03 22:21:38.144644 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.144522 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-cnibin\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.144644 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.144568 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxz2n\" (UniqueName: \"kubernetes.io/projected/c6b58b28-fb4b-4635-a93a-0bd23222670b-kube-api-access-wxz2n\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:21:38.144644 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.144599 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/514a860a-f1e3-4714-8493-367b6b8b32a2-host\") pod \"node-ca-t5l5t\" (UID: \"514a860a-f1e3-4714-8493-367b6b8b32a2\") " pod="openshift-image-registry/node-ca-t5l5t" Nov 03 22:21:38.144775 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.144643 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d4425733-4528-4caa-99cb-e94072a00f78-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sq9mk\" (UID: \"d4425733-4528-4caa-99cb-e94072a00f78\") " pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:21:38.144775 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.144675 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlg2q\" (UniqueName: \"kubernetes.io/projected/d4425733-4528-4caa-99cb-e94072a00f78-kube-api-access-zlg2q\") pod \"multus-additional-cni-plugins-sq9mk\" (UID: \"d4425733-4528-4caa-99cb-e94072a00f78\") " pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:21:38.144775 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.144708 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-cni-bin\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.144775 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.144735 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-cni-netd\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.144945 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.144783 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-host-run-netns\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.144945 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.144807 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b6154c5a-4925-4c74-8db4-0acdb525b743-kubelet-config\") pod \"global-pull-secret-syncer-h8jxb\" (UID: \"b6154c5a-4925-4c74-8db4-0acdb525b743\") " pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:21:38.144945 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.144841 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6b58b28-fb4b-4635-a93a-0bd23222670b-socket-dir\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:21:38.144945 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.144869 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6b58b28-fb4b-4635-a93a-0bd23222670b-registration-dir\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:21:38.144945 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.144900 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-systemd-units\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.144945 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.144929 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-etc-openvswitch\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.145207 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.145036 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-system-cni-dir\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.145207 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.145090 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-var-lib-openvswitch\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.145207 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.145134 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-log-socket\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.145207 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.145168 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1b2df441-74d4-435f-a56a-5eff3ec4575d-ovnkube-config\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.145426 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.145386 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-host-run-multus-certs\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.145499 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.145457 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-etc-kubernetes\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.145571 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.145531 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-modprobe-d\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.145679 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.145620 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d4425733-4528-4caa-99cb-e94072a00f78-system-cni-dir\") pod \"multus-additional-cni-plugins-sq9mk\" (UID: \"d4425733-4528-4caa-99cb-e94072a00f78\") " pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:21:38.145972 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.145949 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-run-systemd\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.146046 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.145983 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-run-openvswitch\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.146046 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.146000 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-multus-conf-dir\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.146046 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.146019 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-kubernetes\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.146173 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.146069 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-slash\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.146173 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.146093 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6b58b28-fb4b-4635-a93a-0bd23222670b-sys-fs\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:21:38.146173 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.146144 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jqsx\" (UniqueName: \"kubernetes.io/projected/a90468d5-e140-4ae4-a1cb-c8f9dec579ff-kube-api-access-9jqsx\") pod \"iptables-alerter-p55z4\" (UID: \"a90468d5-e140-4ae4-a1cb-c8f9dec579ff\") " pod="openshift-network-operator/iptables-alerter-p55z4" Nov 03 22:21:38.146291 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.146189 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-var-lib-kubelet\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.146291 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.146254 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d4425733-4528-4caa-99cb-e94072a00f78-cnibin\") pod \"multus-additional-cni-plugins-sq9mk\" (UID: \"d4425733-4528-4caa-99cb-e94072a00f78\") " pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:21:38.146406 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.146297 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-node-log\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.146406 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.146341 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d4425733-4528-4caa-99cb-e94072a00f78-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sq9mk\" (UID: \"d4425733-4528-4caa-99cb-e94072a00f78\") " pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:21:38.146406 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.146401 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1b2df441-74d4-435f-a56a-5eff3ec4575d-ovnkube-script-lib\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.146533 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.146419 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-host-var-lib-kubelet\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.146533 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.146443 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-hostroot\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.146533 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.146483 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4e68337a-9274-466d-a238-082fdf86f4cb-multus-daemon-config\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.146663 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.146533 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6b58b28-fb4b-4635-a93a-0bd23222670b-etc-selinux\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:21:38.146663 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.146561 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d4425733-4528-4caa-99cb-e94072a00f78-cni-binary-copy\") pod \"multus-additional-cni-plugins-sq9mk\" (UID: \"d4425733-4528-4caa-99cb-e94072a00f78\") " pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:21:38.146663 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.146593 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d4425733-4528-4caa-99cb-e94072a00f78-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sq9mk\" (UID: \"d4425733-4528-4caa-99cb-e94072a00f78\") " pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:21:38.146663 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.146630 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9swx\" (UniqueName: \"kubernetes.io/projected/4e68337a-9274-466d-a238-082fdf86f4cb-kube-api-access-p9swx\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.146663 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.146652 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-run-netns\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.146839 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.146667 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-run-ovn-kubernetes\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.146839 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.146697 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqdbp\" (UniqueName: \"kubernetes.io/projected/c027d694-64cc-46a2-b4e7-49b117f696d3-kube-api-access-sqdbp\") pod \"network-metrics-daemon-vqppk\" (UID: \"c027d694-64cc-46a2-b4e7-49b117f696d3\") " pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:21:38.146839 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.146731 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-run\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.146839 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.146785 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-kubelet\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.146839 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.146814 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-host-var-lib-cni-bin\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.147034 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.146908 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5zc6\" (UniqueName: \"kubernetes.io/projected/29b82faa-1aa1-4760-a9f6-1cbeb9427094-kube-api-access-g5zc6\") pod \"network-check-target-jkjnj\" (UID: \"29b82faa-1aa1-4760-a9f6-1cbeb9427094\") " pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:21:38.147034 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.146933 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-systemd\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.147034 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.146958 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-lib-modules\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.147034 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.146989 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1618fd8a-9a1b-4437-8559-86690d19cc9d-tmp\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.147034 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.147010 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-run-ovn\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.147034 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.147031 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-host-var-lib-cni-multus\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.147265 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.147060 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b6154c5a-4925-4c74-8db4-0acdb525b743-dbus\") pod \"global-pull-secret-syncer-h8jxb\" (UID: \"b6154c5a-4925-4c74-8db4-0acdb525b743\") " pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:21:38.147265 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.147083 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b6154c5a-4925-4c74-8db4-0acdb525b743-original-pull-secret\") pod \"global-pull-secret-syncer-h8jxb\" (UID: \"b6154c5a-4925-4c74-8db4-0acdb525b743\") " pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:21:38.147265 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.147147 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6b58b28-fb4b-4635-a93a-0bd23222670b-device-dir\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:21:38.147265 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.147178 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c027d694-64cc-46a2-b4e7-49b117f696d3-metrics-certs\") pod \"network-metrics-daemon-vqppk\" (UID: \"c027d694-64cc-46a2-b4e7-49b117f696d3\") " pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:21:38.147265 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.147207 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-sysconfig\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.147265 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.147241 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-tuned\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.147265 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.147259 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-multus-socket-dir-parent\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.147585 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.147277 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.147585 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.147322 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-multus-cni-dir\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.147585 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.147348 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6b58b28-fb4b-4635-a93a-0bd23222670b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:21:38.147585 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.147397 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0ba09d3-25f2-422b-b5e2-357aabcaf81c-agent-certs\") pod \"konnectivity-agent-g6qjz\" (UID: \"b0ba09d3-25f2-422b-b5e2-357aabcaf81c\") " pod="kube-system/konnectivity-agent-g6qjz" Nov 03 22:21:38.147585 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.147428 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9988v\" (UniqueName: \"kubernetes.io/projected/1618fd8a-9a1b-4437-8559-86690d19cc9d-kube-api-access-9988v\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.147585 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.147459 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f63f886-379d-4636-8fb3-21b932a67831-tmp-dir\") pod \"node-resolver-k4h5g\" (UID: \"3f63f886-379d-4636-8fb3-21b932a67831\") " pod="openshift-dns/node-resolver-k4h5g" Nov 03 22:21:38.147585 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.147492 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-host-run-k8s-cni-cncf-io\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.147585 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.147536 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/514a860a-f1e3-4714-8493-367b6b8b32a2-serviceca\") pod \"node-ca-t5l5t\" (UID: \"514a860a-f1e3-4714-8493-367b6b8b32a2\") " pod="openshift-image-registry/node-ca-t5l5t" Nov 03 22:21:38.147585 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.147565 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmjhw\" (UniqueName: \"kubernetes.io/projected/514a860a-f1e3-4714-8493-367b6b8b32a2-kube-api-access-mmjhw\") pod \"node-ca-t5l5t\" (UID: \"514a860a-f1e3-4714-8493-367b6b8b32a2\") " pod="openshift-image-registry/node-ca-t5l5t" Nov 03 22:21:38.147919 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.147589 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a90468d5-e140-4ae4-a1cb-c8f9dec579ff-iptables-alerter-script\") pod \"iptables-alerter-p55z4\" (UID: \"a90468d5-e140-4ae4-a1cb-c8f9dec579ff\") " pod="openshift-network-operator/iptables-alerter-p55z4" Nov 03 22:21:38.147919 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.147612 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a90468d5-e140-4ae4-a1cb-c8f9dec579ff-host-slash\") pod \"iptables-alerter-p55z4\" (UID: \"a90468d5-e140-4ae4-a1cb-c8f9dec579ff\") " pod="openshift-network-operator/iptables-alerter-p55z4" Nov 03 22:21:38.147919 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.147635 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69c5d\" (UniqueName: \"kubernetes.io/projected/3f63f886-379d-4636-8fb3-21b932a67831-kube-api-access-69c5d\") pod \"node-resolver-k4h5g\" (UID: \"3f63f886-379d-4636-8fb3-21b932a67831\") " pod="openshift-dns/node-resolver-k4h5g" Nov 03 22:21:38.147919 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.147666 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-sysctl-d\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.147919 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.147695 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-sysctl-conf\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.189544 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.189522 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 22:21:38.189544 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.189523 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mx976\"" Nov 03 22:21:38.189544 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.189541 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 22:21:38.189831 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.189793 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 22:21:38.189831 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.189819 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-q82l2\"" Nov 03 22:21:38.189961 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.189839 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 22:21:38.192204 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.192190 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xglhz\"" Nov 03 22:21:38.192411 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.192394 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-px8pz\"" Nov 03 22:21:38.192452 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.192443 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dbbtb\"" Nov 03 22:21:38.192530 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.192513 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 22:21:38.192630 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.192549 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 22:21:38.192630 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.192577 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 22:21:38.192630 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.192589 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 22:21:38.192630 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.192591 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 22:21:38.192630 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.192609 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 22:21:38.192630 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.192611 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 22:21:38.192630 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.192620 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:21:38.192630 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.192629 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zhswt\"" Nov 03 22:21:38.192918 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.192651 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 22:21:38.193058 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.193042 2541 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 22:21:38.193124 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.193102 2541 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jk5zm" Nov 03 22:21:38.248266 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.248246 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-host-var-lib-kubelet\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.248350 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.248273 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-hostroot\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.248350 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.248288 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4e68337a-9274-466d-a238-082fdf86f4cb-multus-daemon-config\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.248350 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.248302 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6b58b28-fb4b-4635-a93a-0bd23222670b-etc-selinux\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:21:38.248350 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.248319 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d4425733-4528-4caa-99cb-e94072a00f78-cni-binary-copy\") pod \"multus-additional-cni-plugins-sq9mk\" (UID: \"d4425733-4528-4caa-99cb-e94072a00f78\") " pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:21:38.248505 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.248372 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-host-var-lib-kubelet\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.248505 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.248372 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-hostroot\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.248505 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.248419 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d4425733-4528-4caa-99cb-e94072a00f78-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sq9mk\" (UID: \"d4425733-4528-4caa-99cb-e94072a00f78\") " pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:21:38.248505 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.248444 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9swx\" (UniqueName: \"kubernetes.io/projected/4e68337a-9274-466d-a238-082fdf86f4cb-kube-api-access-p9swx\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.248505 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.248452 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6b58b28-fb4b-4635-a93a-0bd23222670b-etc-selinux\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:21:38.248505 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.248467 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-run-netns\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.248748 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.248531 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-run-ovn-kubernetes\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.248748 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.248556 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqdbp\" (UniqueName: \"kubernetes.io/projected/c027d694-64cc-46a2-b4e7-49b117f696d3-kube-api-access-sqdbp\") pod \"network-metrics-daemon-vqppk\" (UID: \"c027d694-64cc-46a2-b4e7-49b117f696d3\") " pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:21:38.248748 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.248576 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-run\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.248748 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.248599 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-kubelet\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.248748 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.248608 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-run-netns\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.248748 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.248618 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-run-ovn-kubernetes\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.248748 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.248623 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-host-var-lib-cni-bin\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.248748 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.248659 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-host-var-lib-cni-bin\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.248748 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.248676 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5zc6\" (UniqueName: \"kubernetes.io/projected/29b82faa-1aa1-4760-a9f6-1cbeb9427094-kube-api-access-g5zc6\") pod \"network-check-target-jkjnj\" (UID: \"29b82faa-1aa1-4760-a9f6-1cbeb9427094\") " pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:21:38.248748 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.248701 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-kubelet\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.248748 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.248705 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-systemd\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.248748 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.248730 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-lib-modules\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.249272 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.248751 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-run\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.249272 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.248779 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1618fd8a-9a1b-4437-8559-86690d19cc9d-tmp\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.249272 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.248808 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-systemd\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.249272 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.248823 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-run-ovn\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.249272 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.248850 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-host-var-lib-cni-multus\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.249272 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.248876 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b6154c5a-4925-4c74-8db4-0acdb525b743-dbus\") pod \"global-pull-secret-syncer-h8jxb\" (UID: \"b6154c5a-4925-4c74-8db4-0acdb525b743\") " pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:21:38.249272 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.248894 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-lib-modules\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.249272 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.248900 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b6154c5a-4925-4c74-8db4-0acdb525b743-original-pull-secret\") pod \"global-pull-secret-syncer-h8jxb\" (UID: \"b6154c5a-4925-4c74-8db4-0acdb525b743\") " pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:21:38.249272 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.248926 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6b58b28-fb4b-4635-a93a-0bd23222670b-device-dir\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:21:38.249272 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.248934 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4e68337a-9274-466d-a238-082fdf86f4cb-multus-daemon-config\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.249272 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.248945 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-run-ovn\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.249272 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.248954 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c027d694-64cc-46a2-b4e7-49b117f696d3-metrics-certs\") pod \"network-metrics-daemon-vqppk\" (UID: \"c027d694-64cc-46a2-b4e7-49b117f696d3\") " pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:21:38.249272 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.248983 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-sysconfig\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.249272 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.248986 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-host-var-lib-cni-multus\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.249272 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249000 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d4425733-4528-4caa-99cb-e94072a00f78-cni-binary-copy\") pod \"multus-additional-cni-plugins-sq9mk\" (UID: \"d4425733-4528-4caa-99cb-e94072a00f78\") " pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:21:38.249272 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249008 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-tuned\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.249272 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249031 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-multus-socket-dir-parent\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.250156 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249038 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d4425733-4528-4caa-99cb-e94072a00f78-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sq9mk\" (UID: \"d4425733-4528-4caa-99cb-e94072a00f78\") " pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:21:38.250156 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249042 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6b58b28-fb4b-4635-a93a-0bd23222670b-device-dir\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:21:38.250156 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249056 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.250156 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:38.249044 2541 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:21:38.250156 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249071 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-sysconfig\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.250156 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249081 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-multus-cni-dir\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.250156 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249083 2541 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 22:21:38.250156 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249106 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6b58b28-fb4b-4635-a93a-0bd23222670b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:21:38.250156 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249119 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b6154c5a-4925-4c74-8db4-0acdb525b743-dbus\") pod \"global-pull-secret-syncer-h8jxb\" (UID: \"b6154c5a-4925-4c74-8db4-0acdb525b743\") " pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:21:38.250156 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249129 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-multus-socket-dir-parent\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.250156 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:38.249090 2541 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:21:38.250156 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249159 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.250156 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249182 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-multus-cni-dir\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.250156 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249156 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6b58b28-fb4b-4635-a93a-0bd23222670b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:21:38.250156 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:38.249125 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c027d694-64cc-46a2-b4e7-49b117f696d3-metrics-certs podName:c027d694-64cc-46a2-b4e7-49b117f696d3 nodeName:}" failed. No retries permitted until 2026-11-03 22:21:38.749107648 +0000 UTC m=+2.269802416 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c027d694-64cc-46a2-b4e7-49b117f696d3-metrics-certs") pod "network-metrics-daemon-vqppk" (UID: "c027d694-64cc-46a2-b4e7-49b117f696d3") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:21:38.250156 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:38.249234 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6154c5a-4925-4c74-8db4-0acdb525b743-original-pull-secret podName:b6154c5a-4925-4c74-8db4-0acdb525b743 nodeName:}" failed. No retries permitted until 2026-11-03 22:21:38.749221824 +0000 UTC m=+2.269916574 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b6154c5a-4925-4c74-8db4-0acdb525b743-original-pull-secret") pod "global-pull-secret-syncer-h8jxb" (UID: "b6154c5a-4925-4c74-8db4-0acdb525b743") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:21:38.250156 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249249 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0ba09d3-25f2-422b-b5e2-357aabcaf81c-agent-certs\") pod \"konnectivity-agent-g6qjz\" (UID: \"b0ba09d3-25f2-422b-b5e2-357aabcaf81c\") " pod="kube-system/konnectivity-agent-g6qjz" Nov 03 22:21:38.250946 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249268 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9988v\" (UniqueName: \"kubernetes.io/projected/1618fd8a-9a1b-4437-8559-86690d19cc9d-kube-api-access-9988v\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.250946 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249295 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f63f886-379d-4636-8fb3-21b932a67831-tmp-dir\") pod \"node-resolver-k4h5g\" (UID: \"3f63f886-379d-4636-8fb3-21b932a67831\") " pod="openshift-dns/node-resolver-k4h5g" Nov 03 22:21:38.250946 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249336 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-host-run-k8s-cni-cncf-io\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.250946 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249379 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/514a860a-f1e3-4714-8493-367b6b8b32a2-serviceca\") pod \"node-ca-t5l5t\" (UID: \"514a860a-f1e3-4714-8493-367b6b8b32a2\") " pod="openshift-image-registry/node-ca-t5l5t" Nov 03 22:21:38.250946 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249404 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmjhw\" (UniqueName: \"kubernetes.io/projected/514a860a-f1e3-4714-8493-367b6b8b32a2-kube-api-access-mmjhw\") pod \"node-ca-t5l5t\" (UID: \"514a860a-f1e3-4714-8493-367b6b8b32a2\") " pod="openshift-image-registry/node-ca-t5l5t" Nov 03 22:21:38.250946 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249430 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a90468d5-e140-4ae4-a1cb-c8f9dec579ff-iptables-alerter-script\") pod \"iptables-alerter-p55z4\" (UID: \"a90468d5-e140-4ae4-a1cb-c8f9dec579ff\") " pod="openshift-network-operator/iptables-alerter-p55z4" Nov 03 22:21:38.250946 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249456 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a90468d5-e140-4ae4-a1cb-c8f9dec579ff-host-slash\") pod \"iptables-alerter-p55z4\" (UID: \"a90468d5-e140-4ae4-a1cb-c8f9dec579ff\") " pod="openshift-network-operator/iptables-alerter-p55z4" Nov 03 22:21:38.250946 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249480 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69c5d\" (UniqueName: \"kubernetes.io/projected/3f63f886-379d-4636-8fb3-21b932a67831-kube-api-access-69c5d\") pod \"node-resolver-k4h5g\" (UID: \"3f63f886-379d-4636-8fb3-21b932a67831\") " pod="openshift-dns/node-resolver-k4h5g" Nov 03 22:21:38.250946 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249402 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-host-run-k8s-cni-cncf-io\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.250946 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249508 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-sysctl-d\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.250946 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249533 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-sysctl-conf\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.250946 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249560 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-host\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.250946 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249590 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f63f886-379d-4636-8fb3-21b932a67831-tmp-dir\") pod \"node-resolver-k4h5g\" (UID: \"3f63f886-379d-4636-8fb3-21b932a67831\") " pod="openshift-dns/node-resolver-k4h5g" Nov 03 22:21:38.250946 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249601 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1b2df441-74d4-435f-a56a-5eff3ec4575d-env-overrides\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.250946 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249627 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b2df441-74d4-435f-a56a-5eff3ec4575d-ovn-node-metrics-cert\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.250946 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249660 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4n2l\" (UniqueName: \"kubernetes.io/projected/1b2df441-74d4-435f-a56a-5eff3ec4575d-kube-api-access-s4n2l\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.250946 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249685 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e68337a-9274-466d-a238-082fdf86f4cb-cni-binary-copy\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.251734 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249714 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-sysctl-d\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.251734 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249715 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0ba09d3-25f2-422b-b5e2-357aabcaf81c-konnectivity-ca\") pod \"konnectivity-agent-g6qjz\" (UID: \"b0ba09d3-25f2-422b-b5e2-357aabcaf81c\") " pod="kube-system/konnectivity-agent-g6qjz" Nov 03 22:21:38.251734 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249748 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-sys\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.251734 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249768 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3f63f886-379d-4636-8fb3-21b932a67831-hosts-file\") pod \"node-resolver-k4h5g\" (UID: \"3f63f886-379d-4636-8fb3-21b932a67831\") " pod="openshift-dns/node-resolver-k4h5g" Nov 03 22:21:38.251734 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249789 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-cnibin\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.251734 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249796 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/514a860a-f1e3-4714-8493-367b6b8b32a2-serviceca\") pod \"node-ca-t5l5t\" (UID: \"514a860a-f1e3-4714-8493-367b6b8b32a2\") " pod="openshift-image-registry/node-ca-t5l5t" Nov 03 22:21:38.251734 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249809 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxz2n\" (UniqueName: \"kubernetes.io/projected/c6b58b28-fb4b-4635-a93a-0bd23222670b-kube-api-access-wxz2n\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:21:38.251734 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249838 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/514a860a-f1e3-4714-8493-367b6b8b32a2-host\") pod \"node-ca-t5l5t\" (UID: \"514a860a-f1e3-4714-8493-367b6b8b32a2\") " pod="openshift-image-registry/node-ca-t5l5t" Nov 03 22:21:38.251734 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249866 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d4425733-4528-4caa-99cb-e94072a00f78-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sq9mk\" (UID: \"d4425733-4528-4caa-99cb-e94072a00f78\") " pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:21:38.251734 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249891 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlg2q\" (UniqueName: \"kubernetes.io/projected/d4425733-4528-4caa-99cb-e94072a00f78-kube-api-access-zlg2q\") pod \"multus-additional-cni-plugins-sq9mk\" (UID: \"d4425733-4528-4caa-99cb-e94072a00f78\") " pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:21:38.251734 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249915 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-cni-bin\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.251734 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249963 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-cni-netd\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.251734 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.249986 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-host-run-netns\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.251734 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.250011 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b6154c5a-4925-4c74-8db4-0acdb525b743-kubelet-config\") pod \"global-pull-secret-syncer-h8jxb\" (UID: \"b6154c5a-4925-4c74-8db4-0acdb525b743\") " pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:21:38.251734 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.250056 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6b58b28-fb4b-4635-a93a-0bd23222670b-socket-dir\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:21:38.251734 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.250185 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0ba09d3-25f2-422b-b5e2-357aabcaf81c-konnectivity-ca\") pod \"konnectivity-agent-g6qjz\" (UID: \"b0ba09d3-25f2-422b-b5e2-357aabcaf81c\") " pod="kube-system/konnectivity-agent-g6qjz" Nov 03 22:21:38.251734 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.250194 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a90468d5-e140-4ae4-a1cb-c8f9dec579ff-iptables-alerter-script\") pod \"iptables-alerter-p55z4\" (UID: \"a90468d5-e140-4ae4-a1cb-c8f9dec579ff\") " pod="openshift-network-operator/iptables-alerter-p55z4" Nov 03 22:21:38.252488 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.250249 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-cni-bin\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.252488 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.250284 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/514a860a-f1e3-4714-8493-367b6b8b32a2-host\") pod \"node-ca-t5l5t\" (UID: \"514a860a-f1e3-4714-8493-367b6b8b32a2\") " pod="openshift-image-registry/node-ca-t5l5t" Nov 03 22:21:38.252488 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.250319 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-sysctl-conf\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.252488 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.250352 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-sys\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.252488 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.250387 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-host\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.252488 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.250439 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3f63f886-379d-4636-8fb3-21b932a67831-hosts-file\") pod \"node-resolver-k4h5g\" (UID: \"3f63f886-379d-4636-8fb3-21b932a67831\") " pod="openshift-dns/node-resolver-k4h5g" Nov 03 22:21:38.252488 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.250485 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-cnibin\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.252488 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.250784 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1b2df441-74d4-435f-a56a-5eff3ec4575d-env-overrides\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.252488 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.250828 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d4425733-4528-4caa-99cb-e94072a00f78-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sq9mk\" (UID: \"d4425733-4528-4caa-99cb-e94072a00f78\") " pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:21:38.252488 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.250836 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a90468d5-e140-4ae4-a1cb-c8f9dec579ff-host-slash\") pod \"iptables-alerter-p55z4\" (UID: \"a90468d5-e140-4ae4-a1cb-c8f9dec579ff\") " pod="openshift-network-operator/iptables-alerter-p55z4" Nov 03 22:21:38.252488 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.251077 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-host-run-netns\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.252488 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.251188 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-cni-netd\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.252488 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.251283 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e68337a-9274-466d-a238-082fdf86f4cb-cni-binary-copy\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.252488 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.251373 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b6154c5a-4925-4c74-8db4-0acdb525b743-kubelet-config\") pod \"global-pull-secret-syncer-h8jxb\" (UID: \"b6154c5a-4925-4c74-8db4-0acdb525b743\") " pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:21:38.252488 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.251396 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6b58b28-fb4b-4635-a93a-0bd23222670b-socket-dir\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:21:38.253027 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.252716 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-tuned\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.253027 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.252746 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1618fd8a-9a1b-4437-8559-86690d19cc9d-tmp\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.253027 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.252822 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0ba09d3-25f2-422b-b5e2-357aabcaf81c-agent-certs\") pod \"konnectivity-agent-g6qjz\" (UID: \"b0ba09d3-25f2-422b-b5e2-357aabcaf81c\") " pod="kube-system/konnectivity-agent-g6qjz" Nov 03 22:21:38.253027 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.252982 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b2df441-74d4-435f-a56a-5eff3ec4575d-ovn-node-metrics-cert\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.253154 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253069 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6b58b28-fb4b-4635-a93a-0bd23222670b-registration-dir\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:21:38.253154 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253088 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-systemd-units\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.253154 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253110 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-etc-openvswitch\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.253154 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253133 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-system-cni-dir\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.253154 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253137 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-systemd-units\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.253154 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253145 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6b58b28-fb4b-4635-a93a-0bd23222670b-registration-dir\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:21:38.253414 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253156 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-var-lib-openvswitch\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.253414 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253171 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-etc-openvswitch\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.253414 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253190 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-var-lib-openvswitch\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.253414 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253230 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-system-cni-dir\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.253414 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253258 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-log-socket\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.253414 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253277 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1b2df441-74d4-435f-a56a-5eff3ec4575d-ovnkube-config\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.253414 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253310 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-host-run-multus-certs\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.253414 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253317 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-log-socket\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.253414 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253334 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-etc-kubernetes\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.253414 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253370 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-host-run-multus-certs\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.253414 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253384 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-etc-kubernetes\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.253752 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253427 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-modprobe-d\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.253752 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253445 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d4425733-4528-4caa-99cb-e94072a00f78-system-cni-dir\") pod \"multus-additional-cni-plugins-sq9mk\" (UID: \"d4425733-4528-4caa-99cb-e94072a00f78\") " pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:21:38.253752 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253475 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d4425733-4528-4caa-99cb-e94072a00f78-system-cni-dir\") pod \"multus-additional-cni-plugins-sq9mk\" (UID: \"d4425733-4528-4caa-99cb-e94072a00f78\") " pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:21:38.253752 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253506 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-modprobe-d\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.253752 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253530 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-run-systemd\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.253752 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253565 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-run-systemd\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.253752 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253554 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-run-openvswitch\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.253752 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253604 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-multus-conf-dir\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.253752 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253623 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-kubernetes\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.253752 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253648 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-run-openvswitch\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.253752 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253652 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-slash\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.253752 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253668 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-multus-conf-dir\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.253752 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253682 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-slash\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.253752 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253688 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6b58b28-fb4b-4635-a93a-0bd23222670b-sys-fs\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:21:38.253752 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253697 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-kubernetes\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.253752 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253720 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jqsx\" (UniqueName: \"kubernetes.io/projected/a90468d5-e140-4ae4-a1cb-c8f9dec579ff-kube-api-access-9jqsx\") pod \"iptables-alerter-p55z4\" (UID: \"a90468d5-e140-4ae4-a1cb-c8f9dec579ff\") " pod="openshift-network-operator/iptables-alerter-p55z4" Nov 03 22:21:38.253752 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253732 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6b58b28-fb4b-4635-a93a-0bd23222670b-sys-fs\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:21:38.254446 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253747 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-var-lib-kubelet\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.254446 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253770 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d4425733-4528-4caa-99cb-e94072a00f78-cnibin\") pod \"multus-additional-cni-plugins-sq9mk\" (UID: \"d4425733-4528-4caa-99cb-e94072a00f78\") " pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:21:38.254446 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253786 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-node-log\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.254446 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253801 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d4425733-4528-4caa-99cb-e94072a00f78-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sq9mk\" (UID: \"d4425733-4528-4caa-99cb-e94072a00f78\") " pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:21:38.254446 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253825 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d4425733-4528-4caa-99cb-e94072a00f78-cnibin\") pod \"multus-additional-cni-plugins-sq9mk\" (UID: \"d4425733-4528-4caa-99cb-e94072a00f78\") " pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:21:38.254446 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253826 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1b2df441-74d4-435f-a56a-5eff3ec4575d-ovnkube-script-lib\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.254446 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253841 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-node-log\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.254446 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.253867 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-var-lib-kubelet\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.254446 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.254027 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d4425733-4528-4caa-99cb-e94072a00f78-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sq9mk\" (UID: \"d4425733-4528-4caa-99cb-e94072a00f78\") " pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:21:38.254446 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.254285 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1b2df441-74d4-435f-a56a-5eff3ec4575d-ovnkube-script-lib\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.254446 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.254331 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1b2df441-74d4-435f-a56a-5eff3ec4575d-ovnkube-config\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.307403 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:38.307301 2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:21:38.307403 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:38.307340 2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:21:38.307403 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:38.307352 2541 projected.go:196] Error preparing data for projected volume kube-api-access-g5zc6 for pod openshift-network-diagnostics/network-check-target-jkjnj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:21:38.307652 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:38.307438 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29b82faa-1aa1-4760-a9f6-1cbeb9427094-kube-api-access-g5zc6 podName:29b82faa-1aa1-4760-a9f6-1cbeb9427094 nodeName:}" failed. No retries permitted until 2026-11-03 22:21:38.807423625 +0000 UTC m=+2.328118393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-g5zc6" (UniqueName: "kubernetes.io/projected/29b82faa-1aa1-4760-a9f6-1cbeb9427094-kube-api-access-g5zc6") pod "network-check-target-jkjnj" (UID: "29b82faa-1aa1-4760-a9f6-1cbeb9427094") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:21:38.310001 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.309967 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9swx\" (UniqueName: \"kubernetes.io/projected/4e68337a-9274-466d-a238-082fdf86f4cb-kube-api-access-p9swx\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.310601 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.310577 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqdbp\" (UniqueName: \"kubernetes.io/projected/c027d694-64cc-46a2-b4e7-49b117f696d3-kube-api-access-sqdbp\") pod \"network-metrics-daemon-vqppk\" (UID: \"c027d694-64cc-46a2-b4e7-49b117f696d3\") " pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:21:38.310747 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.310724 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmjhw\" (UniqueName: \"kubernetes.io/projected/514a860a-f1e3-4714-8493-367b6b8b32a2-kube-api-access-mmjhw\") pod \"node-ca-t5l5t\" (UID: \"514a860a-f1e3-4714-8493-367b6b8b32a2\") " pod="openshift-image-registry/node-ca-t5l5t" Nov 03 22:21:38.310871 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.310851 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69c5d\" (UniqueName: \"kubernetes.io/projected/3f63f886-379d-4636-8fb3-21b932a67831-kube-api-access-69c5d\") pod \"node-resolver-k4h5g\" (UID: \"3f63f886-379d-4636-8fb3-21b932a67831\") " pod="openshift-dns/node-resolver-k4h5g" Nov 03 22:21:38.310960 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.310936 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9988v\" (UniqueName: \"kubernetes.io/projected/1618fd8a-9a1b-4437-8559-86690d19cc9d-kube-api-access-9988v\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.311671 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.311653 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4n2l\" (UniqueName: \"kubernetes.io/projected/1b2df441-74d4-435f-a56a-5eff3ec4575d-kube-api-access-s4n2l\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.311888 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.311863 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxz2n\" (UniqueName: \"kubernetes.io/projected/c6b58b28-fb4b-4635-a93a-0bd23222670b-kube-api-access-wxz2n\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:21:38.312188 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.312169 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlg2q\" (UniqueName: \"kubernetes.io/projected/d4425733-4528-4caa-99cb-e94072a00f78-kube-api-access-zlg2q\") pod \"multus-additional-cni-plugins-sq9mk\" (UID: \"d4425733-4528-4caa-99cb-e94072a00f78\") " pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:21:38.313780 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.313759 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jqsx\" (UniqueName: \"kubernetes.io/projected/a90468d5-e140-4ae4-a1cb-c8f9dec579ff-kube-api-access-9jqsx\") pod \"iptables-alerter-p55z4\" (UID: \"a90468d5-e140-4ae4-a1cb-c8f9dec579ff\") " pod="openshift-network-operator/iptables-alerter-p55z4" Nov 03 22:21:38.411598 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.411570 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:21:38.451050 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.451013 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t5l5t" Nov 03 22:21:38.467733 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.467711 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4gfds" Nov 03 22:21:38.488170 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.488152 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-p55z4" Nov 03 22:21:38.499243 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.499228 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g6qjz" Nov 03 22:21:38.505156 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:38.505130 2541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda90468d5_e140_4ae4_a1cb_c8f9dec579ff.slice/crio-73a265ef5f45b3ac3d94f657dfcb36bea82a6fc504e1212b5f7b2134f096ce40 WatchSource:0}: Error finding container 73a265ef5f45b3ac3d94f657dfcb36bea82a6fc504e1212b5f7b2134f096ce40: Status 404 returned error can't find the container with id 73a265ef5f45b3ac3d94f657dfcb36bea82a6fc504e1212b5f7b2134f096ce40 Nov 03 22:21:38.506391 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:38.506350 2541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb86a6441b6931e82229a352ca0e493.slice/crio-fd424519b7120717bc3041a2154a7338c2064d0aa6855d0a355f48237626b29a WatchSource:0}: Error finding container fd424519b7120717bc3041a2154a7338c2064d0aa6855d0a355f48237626b29a: Status 404 returned error can't find the container with id fd424519b7120717bc3041a2154a7338c2064d0aa6855d0a355f48237626b29a Nov 03 22:21:38.507297 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:38.507275 2541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd09a9d6123011e89fe0e21d9dfdc2d8.slice/crio-0d307ea1d74a23a33ed4b95e70c747f17dfebf86248ec7bbaf3c0d67eb8b95e9 WatchSource:0}: Error finding container 0d307ea1d74a23a33ed4b95e70c747f17dfebf86248ec7bbaf3c0d67eb8b95e9: Status 404 returned error can't find the container with id 0d307ea1d74a23a33ed4b95e70c747f17dfebf86248ec7bbaf3c0d67eb8b95e9 Nov 03 22:21:38.510802 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.510788 2541 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:21:38.512121 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:38.511992 2541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ba09d3_25f2_422b_b5e2_357aabcaf81c.slice/crio-2f31c2f531cff99e34c0b986817d4291f63faa877f61fb1abb177d3a325faeb5 WatchSource:0}: Error finding container 2f31c2f531cff99e34c0b986817d4291f63faa877f61fb1abb177d3a325faeb5: Status 404 returned error can't find the container with id 2f31c2f531cff99e34c0b986817d4291f63faa877f61fb1abb177d3a325faeb5 Nov 03 22:21:38.525179 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.525163 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:21:38.533782 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:38.533764 2541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6b58b28_fb4b_4635_a93a_0bd23222670b.slice/crio-2127e840e13547e934d97137d91178a5271c7d343a979272861e66faf1fd80f6 WatchSource:0}: Error finding container 2127e840e13547e934d97137d91178a5271c7d343a979272861e66faf1fd80f6: Status 404 returned error can't find the container with id 2127e840e13547e934d97137d91178a5271c7d343a979272861e66faf1fd80f6 Nov 03 22:21:38.541041 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.541027 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:21:38.545809 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.545793 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k4h5g" Nov 03 22:21:38.548998 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:38.548980 2541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1618fd8a_9a1b_4437_8559_86690d19cc9d.slice/crio-112b78ae0dbb28af03e915408e4014c75a9dc87adb37820f39b80054a3b16ae1 WatchSource:0}: Error finding container 112b78ae0dbb28af03e915408e4014c75a9dc87adb37820f39b80054a3b16ae1: Status 404 returned error can't find the container with id 112b78ae0dbb28af03e915408e4014c75a9dc87adb37820f39b80054a3b16ae1 Nov 03 22:21:38.553560 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:38.553540 2541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f63f886_379d_4636_8fb3_21b932a67831.slice/crio-8f4e96c3a6e48b918986d2ecd8b46c7fc4a7de6ec45a32278cc33c191b90d10d WatchSource:0}: Error finding container 8f4e96c3a6e48b918986d2ecd8b46c7fc4a7de6ec45a32278cc33c191b90d10d: Status 404 returned error can't find the container with id 8f4e96c3a6e48b918986d2ecd8b46c7fc4a7de6ec45a32278cc33c191b90d10d Nov 03 22:21:38.572534 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.572488 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:21:38.581421 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:38.581401 2541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4425733_4528_4caa_99cb_e94072a00f78.slice/crio-cc5da91da75f5484b0e4a722d92d72b7058291a8db65f386076b30594d68ade0 WatchSource:0}: Error finding container cc5da91da75f5484b0e4a722d92d72b7058291a8db65f386076b30594d68ade0: Status 404 returned error can't find the container with id cc5da91da75f5484b0e4a722d92d72b7058291a8db65f386076b30594d68ade0 Nov 03 22:21:38.653920 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.653900 2541 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:21:38.757881 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.757862 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b6154c5a-4925-4c74-8db4-0acdb525b743-original-pull-secret\") pod \"global-pull-secret-syncer-h8jxb\" (UID: \"b6154c5a-4925-4c74-8db4-0acdb525b743\") " pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:21:38.757987 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:38.757888 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c027d694-64cc-46a2-b4e7-49b117f696d3-metrics-certs\") pod \"network-metrics-daemon-vqppk\" (UID: \"c027d694-64cc-46a2-b4e7-49b117f696d3\") " pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:21:38.758029 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:38.757998 2541 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:21:38.758029 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:38.757999 2541 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:21:38.758086 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:38.758041 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c027d694-64cc-46a2-b4e7-49b117f696d3-metrics-certs podName:c027d694-64cc-46a2-b4e7-49b117f696d3 nodeName:}" failed. No retries permitted until 2026-11-03 22:21:39.758029968 +0000 UTC m=+3.278724731 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c027d694-64cc-46a2-b4e7-49b117f696d3-metrics-certs") pod "network-metrics-daemon-vqppk" (UID: "c027d694-64cc-46a2-b4e7-49b117f696d3") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:21:38.758086 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:38.758054 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6154c5a-4925-4c74-8db4-0acdb525b743-original-pull-secret podName:b6154c5a-4925-4c74-8db4-0acdb525b743 nodeName:}" failed. No retries permitted until 2026-11-03 22:21:39.758048045 +0000 UTC m=+3.278742795 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b6154c5a-4925-4c74-8db4-0acdb525b743-original-pull-secret") pod "global-pull-secret-syncer-h8jxb" (UID: "b6154c5a-4925-4c74-8db4-0acdb525b743") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:21:39.862207 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:39.862181 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5zc6\" (UniqueName: \"kubernetes.io/projected/29b82faa-1aa1-4760-a9f6-1cbeb9427094-kube-api-access-g5zc6\") pod \"network-check-target-jkjnj\" (UID: \"29b82faa-1aa1-4760-a9f6-1cbeb9427094\") " pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:21:39.862779 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:39.862279 2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:21:39.862779 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:39.862291 2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:21:39.862779 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:39.862301 2541 projected.go:196] Error preparing data for projected volume kube-api-access-g5zc6 for pod openshift-network-diagnostics/network-check-target-jkjnj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:21:39.862779 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:39.862342 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29b82faa-1aa1-4760-a9f6-1cbeb9427094-kube-api-access-g5zc6 podName:29b82faa-1aa1-4760-a9f6-1cbeb9427094 nodeName:}" failed. No retries permitted until 2026-11-03 22:21:40.862330737 +0000 UTC m=+3.379527286 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-g5zc6" (UniqueName: "kubernetes.io/projected/29b82faa-1aa1-4760-a9f6-1cbeb9427094-kube-api-access-g5zc6") pod "network-check-target-jkjnj" (UID: "29b82faa-1aa1-4760-a9f6-1cbeb9427094") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:21:39.891606 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:39.891587 2541 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:21:39.981613 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:39.981565 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k4h5g" event={"ID":"3f63f886-379d-4636-8fb3-21b932a67831","Type":"ContainerStarted","Data":"8f4e96c3a6e48b918986d2ecd8b46c7fc4a7de6ec45a32278cc33c191b90d10d"} Nov 03 22:21:39.982570 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:39.982547 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j9x58" event={"ID":"1618fd8a-9a1b-4437-8559-86690d19cc9d","Type":"ContainerStarted","Data":"112b78ae0dbb28af03e915408e4014c75a9dc87adb37820f39b80054a3b16ae1"} Nov 03 22:21:39.983533 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:39.983513 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" event={"ID":"c6b58b28-fb4b-4635-a93a-0bd23222670b","Type":"ContainerStarted","Data":"2127e840e13547e934d97137d91178a5271c7d343a979272861e66faf1fd80f6"} Nov 03 22:21:39.984583 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:39.984564 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g6qjz" event={"ID":"b0ba09d3-25f2-422b-b5e2-357aabcaf81c","Type":"ContainerStarted","Data":"2f31c2f531cff99e34c0b986817d4291f63faa877f61fb1abb177d3a325faeb5"} Nov 03 22:21:39.985572 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:39.985554 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-p55z4" event={"ID":"a90468d5-e140-4ae4-a1cb-c8f9dec579ff","Type":"ContainerStarted","Data":"73a265ef5f45b3ac3d94f657dfcb36bea82a6fc504e1212b5f7b2134f096ce40"} Nov 03 22:21:39.989213 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:39.989170 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-132.us-west-1.compute.internal" event={"ID":"bd09a9d6123011e89fe0e21d9dfdc2d8","Type":"ContainerStarted","Data":"0d307ea1d74a23a33ed4b95e70c747f17dfebf86248ec7bbaf3c0d67eb8b95e9"} Nov 03 22:21:39.990697 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:39.990676 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal" event={"ID":"9bb86a6441b6931e82229a352ca0e493","Type":"ContainerStarted","Data":"fd424519b7120717bc3041a2154a7338c2064d0aa6855d0a355f48237626b29a"} Nov 03 22:21:39.992206 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:39.992187 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sq9mk" event={"ID":"d4425733-4528-4caa-99cb-e94072a00f78","Type":"ContainerStarted","Data":"cc5da91da75f5484b0e4a722d92d72b7058291a8db65f386076b30594d68ade0"} Nov 03 22:21:40.158042 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:40.157989 2541 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:21:40.199125 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:40.199085 2541 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 22:16:38 +0000 UTC" deadline="2028-07-09 05:33:27.098210907 +0000 UTC" Nov 03 22:21:40.199125 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:40.199118 2541 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14719h11m46.899094786s" Nov 03 22:21:40.300410 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:40.300375 2541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b2df441_74d4_435f_a56a_5eff3ec4575d.slice/crio-73a5922cdfee41d93928be7c593ec6ee40292e4888ec656fb0de1ea3834c8062 WatchSource:0}: Error finding container 73a5922cdfee41d93928be7c593ec6ee40292e4888ec656fb0de1ea3834c8062: Status 404 returned error can't find the container with id 73a5922cdfee41d93928be7c593ec6ee40292e4888ec656fb0de1ea3834c8062 Nov 03 22:21:40.328849 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:40.328826 2541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e68337a_9274_466d_a238_082fdf86f4cb.slice/crio-20100914eddea7d757ce5f2aa3d713348c92743c7a31753896979a9a188e758c WatchSource:0}: Error finding container 20100914eddea7d757ce5f2aa3d713348c92743c7a31753896979a9a188e758c: Status 404 returned error can't find the container with id 20100914eddea7d757ce5f2aa3d713348c92743c7a31753896979a9a188e758c Nov 03 22:21:40.352532 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:21:40.352496 2541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod514a860a_f1e3_4714_8493_367b6b8b32a2.slice/crio-11620f2ef1ce4776d1fe1a60b27ba56db85c0fb6d4c806344ac114bb54456f59 WatchSource:0}: Error finding container 11620f2ef1ce4776d1fe1a60b27ba56db85c0fb6d4c806344ac114bb54456f59: Status 404 returned error can't find the container with id 11620f2ef1ce4776d1fe1a60b27ba56db85c0fb6d4c806344ac114bb54456f59 Nov 03 22:21:40.769067 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:40.769028 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b6154c5a-4925-4c74-8db4-0acdb525b743-original-pull-secret\") pod \"global-pull-secret-syncer-h8jxb\" (UID: \"b6154c5a-4925-4c74-8db4-0acdb525b743\") " pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:21:40.769310 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:40.769078 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c027d694-64cc-46a2-b4e7-49b117f696d3-metrics-certs\") pod \"network-metrics-daemon-vqppk\" (UID: \"c027d694-64cc-46a2-b4e7-49b117f696d3\") " pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:21:40.769310 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:40.769195 2541 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:21:40.769310 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:40.769251 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c027d694-64cc-46a2-b4e7-49b117f696d3-metrics-certs podName:c027d694-64cc-46a2-b4e7-49b117f696d3 nodeName:}" failed. No retries permitted until 2026-11-03 22:21:42.769233998 +0000 UTC m=+5.286430561 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c027d694-64cc-46a2-b4e7-49b117f696d3-metrics-certs") pod "network-metrics-daemon-vqppk" (UID: "c027d694-64cc-46a2-b4e7-49b117f696d3") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:21:40.769310 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:40.769310 2541 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:21:40.769534 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:40.769338 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6154c5a-4925-4c74-8db4-0acdb525b743-original-pull-secret podName:b6154c5a-4925-4c74-8db4-0acdb525b743 nodeName:}" failed. No retries permitted until 2026-11-03 22:21:42.769331667 +0000 UTC m=+5.286528230 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b6154c5a-4925-4c74-8db4-0acdb525b743-original-pull-secret") pod "global-pull-secret-syncer-h8jxb" (UID: "b6154c5a-4925-4c74-8db4-0acdb525b743") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:21:40.869528 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:40.869490 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5zc6\" (UniqueName: \"kubernetes.io/projected/29b82faa-1aa1-4760-a9f6-1cbeb9427094-kube-api-access-g5zc6\") pod \"network-check-target-jkjnj\" (UID: \"29b82faa-1aa1-4760-a9f6-1cbeb9427094\") " pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:21:40.870071 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:40.869728 2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:21:40.870071 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:40.869753 2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:21:40.870071 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:40.869767 2541 projected.go:196] Error preparing data for projected volume kube-api-access-g5zc6 for pod openshift-network-diagnostics/network-check-target-jkjnj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:21:40.870071 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:40.869827 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29b82faa-1aa1-4760-a9f6-1cbeb9427094-kube-api-access-g5zc6 podName:29b82faa-1aa1-4760-a9f6-1cbeb9427094 nodeName:}" failed. No retries permitted until 2026-11-03 22:21:42.869808899 +0000 UTC m=+5.387005464 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-g5zc6" (UniqueName: "kubernetes.io/projected/29b82faa-1aa1-4760-a9f6-1cbeb9427094-kube-api-access-g5zc6") pod "network-check-target-jkjnj" (UID: "29b82faa-1aa1-4760-a9f6-1cbeb9427094") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:21:40.978098 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:40.978061 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:21:40.978273 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:40.978209 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqppk" podUID="c027d694-64cc-46a2-b4e7-49b117f696d3" Nov 03 22:21:40.978342 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:40.978287 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:21:40.978395 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:40.978361 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8jxb" podUID="b6154c5a-4925-4c74-8db4-0acdb525b743" Nov 03 22:21:40.978445 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:40.978417 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:21:40.978497 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:40.978483 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjnj" podUID="29b82faa-1aa1-4760-a9f6-1cbeb9427094" Nov 03 22:21:40.997253 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:40.997218 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4gfds" event={"ID":"4e68337a-9274-466d-a238-082fdf86f4cb","Type":"ContainerStarted","Data":"20100914eddea7d757ce5f2aa3d713348c92743c7a31753896979a9a188e758c"} Nov 03 22:21:41.000487 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:41.000457 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" event={"ID":"1b2df441-74d4-435f-a56a-5eff3ec4575d","Type":"ContainerStarted","Data":"73a5922cdfee41d93928be7c593ec6ee40292e4888ec656fb0de1ea3834c8062"} Nov 03 22:21:41.006836 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:41.006800 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t5l5t" event={"ID":"514a860a-f1e3-4714-8493-367b6b8b32a2","Type":"ContainerStarted","Data":"11620f2ef1ce4776d1fe1a60b27ba56db85c0fb6d4c806344ac114bb54456f59"} Nov 03 22:21:41.200137 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:41.200095 2541 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 22:16:38 +0000 UTC" deadline="2028-06-10 15:20:07.165392995 +0000 UTC" Nov 03 22:21:41.200137 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:41.200137 2541 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14032h58m25.965261205s" Nov 03 22:21:42.786023 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:42.785805 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b6154c5a-4925-4c74-8db4-0acdb525b743-original-pull-secret\") pod \"global-pull-secret-syncer-h8jxb\" (UID: \"b6154c5a-4925-4c74-8db4-0acdb525b743\") " pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:21:42.786023 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:42.785849 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c027d694-64cc-46a2-b4e7-49b117f696d3-metrics-certs\") pod \"network-metrics-daemon-vqppk\" (UID: \"c027d694-64cc-46a2-b4e7-49b117f696d3\") " pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:21:42.786023 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:42.786000 2541 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:21:42.786480 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:42.786060 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c027d694-64cc-46a2-b4e7-49b117f696d3-metrics-certs podName:c027d694-64cc-46a2-b4e7-49b117f696d3 nodeName:}" failed. No retries permitted until 2026-11-03 22:21:46.786042167 +0000 UTC m=+9.303238739 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c027d694-64cc-46a2-b4e7-49b117f696d3-metrics-certs") pod "network-metrics-daemon-vqppk" (UID: "c027d694-64cc-46a2-b4e7-49b117f696d3") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:21:42.786480 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:42.786125 2541 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:21:42.786480 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:42.786159 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6154c5a-4925-4c74-8db4-0acdb525b743-original-pull-secret podName:b6154c5a-4925-4c74-8db4-0acdb525b743 nodeName:}" failed. No retries permitted until 2026-11-03 22:21:46.786147618 +0000 UTC m=+9.303344181 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b6154c5a-4925-4c74-8db4-0acdb525b743-original-pull-secret") pod "global-pull-secret-syncer-h8jxb" (UID: "b6154c5a-4925-4c74-8db4-0acdb525b743") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:21:42.887194 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:42.887155 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5zc6\" (UniqueName: \"kubernetes.io/projected/29b82faa-1aa1-4760-a9f6-1cbeb9427094-kube-api-access-g5zc6\") pod \"network-check-target-jkjnj\" (UID: \"29b82faa-1aa1-4760-a9f6-1cbeb9427094\") " pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:21:42.887384 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:42.887363 2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:21:42.887455 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:42.887397 2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:21:42.887455 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:42.887419 2541 projected.go:196] Error preparing data for projected volume kube-api-access-g5zc6 for pod openshift-network-diagnostics/network-check-target-jkjnj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:21:42.887554 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:42.887478 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29b82faa-1aa1-4760-a9f6-1cbeb9427094-kube-api-access-g5zc6 podName:29b82faa-1aa1-4760-a9f6-1cbeb9427094 nodeName:}" failed. No retries permitted until 2026-11-03 22:21:46.887459237 +0000 UTC m=+9.404655804 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-g5zc6" (UniqueName: "kubernetes.io/projected/29b82faa-1aa1-4760-a9f6-1cbeb9427094-kube-api-access-g5zc6") pod "network-check-target-jkjnj" (UID: "29b82faa-1aa1-4760-a9f6-1cbeb9427094") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:21:42.978561 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:42.978518 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:21:42.978726 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:42.978663 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqppk" podUID="c027d694-64cc-46a2-b4e7-49b117f696d3" Nov 03 22:21:42.978822 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:42.978747 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:21:42.978924 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:42.978837 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8jxb" podUID="b6154c5a-4925-4c74-8db4-0acdb525b743" Nov 03 22:21:42.978924 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:42.978906 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:21:42.979037 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:42.978972 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjnj" podUID="29b82faa-1aa1-4760-a9f6-1cbeb9427094" Nov 03 22:21:44.020999 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:44.020005 2541 generic.go:356] "Generic (PLEG): container finished" podID="d4425733-4528-4caa-99cb-e94072a00f78" containerID="03b241e13e64283f332bf3a4364cc62616f9f35012899a92dcf12f3144b16665" exitCode=0 Nov 03 22:21:44.020999 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:44.020252 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sq9mk" event={"ID":"d4425733-4528-4caa-99cb-e94072a00f78","Type":"ContainerDied","Data":"03b241e13e64283f332bf3a4364cc62616f9f35012899a92dcf12f3144b16665"} Nov 03 22:21:44.978270 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:44.977786 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:21:44.978270 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:44.977946 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqppk" podUID="c027d694-64cc-46a2-b4e7-49b117f696d3" Nov 03 22:21:44.978270 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:44.978033 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:21:44.978270 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:44.978112 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8jxb" podUID="b6154c5a-4925-4c74-8db4-0acdb525b743" Nov 03 22:21:44.978270 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:44.978168 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:21:44.978270 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:44.978231 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjnj" podUID="29b82faa-1aa1-4760-a9f6-1cbeb9427094" Nov 03 22:21:45.029036 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:45.028954 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-132.us-west-1.compute.internal" event={"ID":"bd09a9d6123011e89fe0e21d9dfdc2d8","Type":"ContainerStarted","Data":"e510afdbebdf9514515f2a59d6d3c10d55632221b86784f23141b6582e4a1dcc"} Nov 03 22:21:45.032544 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:45.032033 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g6qjz" event={"ID":"b0ba09d3-25f2-422b-b5e2-357aabcaf81c","Type":"ContainerStarted","Data":"a3ad883b829281ba3e966546d936131bbd26577f1f1bbe30d008acf0077978be"} Nov 03 22:21:45.035887 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:45.035778 2541 generic.go:356] "Generic (PLEG): container finished" podID="9bb86a6441b6931e82229a352ca0e493" containerID="fca4dce96915bea16e8548eda8b5fbecb51a93451c1e8ae54245b67c3bb88da2" exitCode=0 Nov 03 22:21:45.035887 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:45.035821 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal" event={"ID":"9bb86a6441b6931e82229a352ca0e493","Type":"ContainerDied","Data":"fca4dce96915bea16e8548eda8b5fbecb51a93451c1e8ae54245b67c3bb88da2"} Nov 03 22:21:45.147118 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:45.146298 2541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-132.us-west-1.compute.internal" podStartSLOduration=7.146279756 podStartE2EDuration="7.146279756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:45.145603077 +0000 UTC m=+7.662799649" watchObservedRunningTime="2026-11-03 22:21:45.146279756 +0000 UTC m=+7.663476330" Nov 03 22:21:46.040870 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:46.040822 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" event={"ID":"c6b58b28-fb4b-4635-a93a-0bd23222670b","Type":"ContainerStarted","Data":"620fa51973b2136920c01ea804660e0197b620749da34d1be864b2512ec9d9b0"} Nov 03 22:21:46.043893 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:46.043851 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal" event={"ID":"9bb86a6441b6931e82229a352ca0e493","Type":"ContainerStarted","Data":"1b04557f1b6393af227bb855f4b3208bff8e8433af562489183ee6bbc4262199"} Nov 03 22:21:46.159792 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:46.159738 2541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-g6qjz" podStartSLOduration=3.7340983359999997 podStartE2EDuration="8.159720264s" totalImagesPullingTime="4.425621928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:45.376519451 +0000 UTC m=+7.893716042" watchObservedRunningTime="2026-11-03 22:21:46.159720264 +0000 UTC m=+8.676916836" Nov 03 22:21:46.160193 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:46.160159 2541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal" podStartSLOduration=8.160150351 podStartE2EDuration="8.160150351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:46.159272196 +0000 UTC m=+8.676468768" watchObservedRunningTime="2026-11-03 22:21:46.160150351 +0000 UTC m=+8.677346924" Nov 03 22:21:46.826099 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:46.826061 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b6154c5a-4925-4c74-8db4-0acdb525b743-original-pull-secret\") pod \"global-pull-secret-syncer-h8jxb\" (UID: \"b6154c5a-4925-4c74-8db4-0acdb525b743\") " pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:21:46.826099 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:46.826105 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c027d694-64cc-46a2-b4e7-49b117f696d3-metrics-certs\") pod \"network-metrics-daemon-vqppk\" (UID: \"c027d694-64cc-46a2-b4e7-49b117f696d3\") " pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:21:46.826319 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:46.826231 2541 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:21:46.826319 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:46.826295 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c027d694-64cc-46a2-b4e7-49b117f696d3-metrics-certs podName:c027d694-64cc-46a2-b4e7-49b117f696d3 nodeName:}" failed. No retries permitted until 2026-11-03 22:21:54.826275306 +0000 UTC m=+17.343471875 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c027d694-64cc-46a2-b4e7-49b117f696d3-metrics-certs") pod "network-metrics-daemon-vqppk" (UID: "c027d694-64cc-46a2-b4e7-49b117f696d3") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:21:46.826439 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:46.826377 2541 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:21:46.826439 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:46.826424 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6154c5a-4925-4c74-8db4-0acdb525b743-original-pull-secret podName:b6154c5a-4925-4c74-8db4-0acdb525b743 nodeName:}" failed. No retries permitted until 2026-11-03 22:21:54.82640197 +0000 UTC m=+17.343598534 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b6154c5a-4925-4c74-8db4-0acdb525b743-original-pull-secret") pod "global-pull-secret-syncer-h8jxb" (UID: "b6154c5a-4925-4c74-8db4-0acdb525b743") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:21:46.927138 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:46.927102 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5zc6\" (UniqueName: \"kubernetes.io/projected/29b82faa-1aa1-4760-a9f6-1cbeb9427094-kube-api-access-g5zc6\") pod \"network-check-target-jkjnj\" (UID: \"29b82faa-1aa1-4760-a9f6-1cbeb9427094\") " pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:21:46.927326 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:46.927306 2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:21:46.927453 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:46.927332 2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:21:46.927453 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:46.927345 2541 projected.go:196] Error preparing data for projected volume kube-api-access-g5zc6 for pod openshift-network-diagnostics/network-check-target-jkjnj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:21:46.927453 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:46.927402 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29b82faa-1aa1-4760-a9f6-1cbeb9427094-kube-api-access-g5zc6 podName:29b82faa-1aa1-4760-a9f6-1cbeb9427094 nodeName:}" failed. No retries permitted until 2026-11-03 22:21:54.927385612 +0000 UTC m=+17.444582166 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-g5zc6" (UniqueName: "kubernetes.io/projected/29b82faa-1aa1-4760-a9f6-1cbeb9427094-kube-api-access-g5zc6") pod "network-check-target-jkjnj" (UID: "29b82faa-1aa1-4760-a9f6-1cbeb9427094") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:21:46.978518 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:46.978403 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:21:46.978685 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:46.978534 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjnj" podUID="29b82faa-1aa1-4760-a9f6-1cbeb9427094" Nov 03 22:21:46.978685 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:46.978629 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:21:46.978802 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:46.978714 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqppk" podUID="c027d694-64cc-46a2-b4e7-49b117f696d3" Nov 03 22:21:46.978802 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:46.978773 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:21:46.978928 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:46.978848 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8jxb" podUID="b6154c5a-4925-4c74-8db4-0acdb525b743" Nov 03 22:21:47.053591 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:47.053553 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t5l5t" event={"ID":"514a860a-f1e3-4714-8493-367b6b8b32a2","Type":"ContainerStarted","Data":"d3068ccda5c64e9f0f57a93d03ef36240071748088029a346c4aa59d939fda70"} Nov 03 22:21:47.169810 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:47.169463 2541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-t5l5t" podStartSLOduration=3.396735321 podStartE2EDuration="9.169446561s" totalImagesPullingTime="5.77271124s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:47.16889582 +0000 UTC m=+9.686092391" watchObservedRunningTime="2026-11-03 22:21:47.169446561 +0000 UTC m=+9.686643133" Nov 03 22:21:47.450583 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:47.450106 2541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-g6qjz" Nov 03 22:21:47.451292 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:47.451259 2541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-g6qjz" Nov 03 22:21:48.055321 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:48.055237 2541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-g6qjz" Nov 03 22:21:48.056341 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:48.056244 2541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-g6qjz" Nov 03 22:21:48.087122 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:48.087052 2541 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 22:21:48.913694 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:48.913591 2541 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T22:21:48.087069082Z","UUID":"000e6b40-f069-43bf-9bab-de170239a3df","Handler":null,"Name":"","Endpoint":""} Nov 03 22:21:48.915420 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:48.915264 2541 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 22:21:48.915420 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:48.915426 2541 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 22:21:48.978479 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:48.978441 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:21:48.978642 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:48.978559 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:21:48.978642 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:48.978575 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8jxb" podUID="b6154c5a-4925-4c74-8db4-0acdb525b743" Nov 03 22:21:48.978725 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:48.978682 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqppk" podUID="c027d694-64cc-46a2-b4e7-49b117f696d3" Nov 03 22:21:48.978785 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:48.978767 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:21:48.978938 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:48.978896 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjnj" podUID="29b82faa-1aa1-4760-a9f6-1cbeb9427094" Nov 03 22:21:49.061538 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:49.061467 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sq9mk" event={"ID":"d4425733-4528-4caa-99cb-e94072a00f78","Type":"ContainerStarted","Data":"d0cb86b6ed4b868e4aec1dd10ee10b30b70bafe8d8e21a083a8bb80dcc39197f"} Nov 03 22:21:49.064925 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:49.064850 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" event={"ID":"c6b58b28-fb4b-4635-a93a-0bd23222670b","Type":"ContainerStarted","Data":"afa375adb08743fa40ee646d453431361ba751c1f0b398e8a4395eba3a09130a"} Nov 03 22:21:49.067794 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:49.067315 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4gfds" event={"ID":"4e68337a-9274-466d-a238-082fdf86f4cb","Type":"ContainerStarted","Data":"eb611f07c7ff5d3811fc5e5ac59dd9bb5c14714277a9942dae3b701e4110e1f0"} Nov 03 22:21:49.070530 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:49.070510 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k4h5g" event={"ID":"3f63f886-379d-4636-8fb3-21b932a67831","Type":"ContainerStarted","Data":"b43ef26ad15c338061e1f88c86e00c19f3d986b6a3a11177dfe92ce6cb1f4e17"} Nov 03 22:21:49.301749 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:49.300428 2541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-k4h5g" podStartSLOduration=2.706505754 podStartE2EDuration="11.300409265s" totalImagesPullingTime="8.593903511s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:49.29669413 +0000 UTC m=+11.813890699" watchObservedRunningTime="2026-11-03 22:21:49.300409265 +0000 UTC m=+11.817605837" Nov 03 22:21:50.076421 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:50.076386 2541 generic.go:356] "Generic (PLEG): container finished" podID="d4425733-4528-4caa-99cb-e94072a00f78" containerID="d0cb86b6ed4b868e4aec1dd10ee10b30b70bafe8d8e21a083a8bb80dcc39197f" exitCode=0 Nov 03 22:21:50.076905 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:50.076465 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sq9mk" event={"ID":"d4425733-4528-4caa-99cb-e94072a00f78","Type":"ContainerDied","Data":"d0cb86b6ed4b868e4aec1dd10ee10b30b70bafe8d8e21a083a8bb80dcc39197f"} Nov 03 22:21:50.079346 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:50.079318 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" event={"ID":"c6b58b28-fb4b-4635-a93a-0bd23222670b","Type":"ContainerStarted","Data":"634e5e2dd185c3c6b4134b42c6ad2585e654a98cb18174bacd9d11e4a79e3c08"} Nov 03 22:21:50.081433 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:50.081405 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-p55z4" event={"ID":"a90468d5-e140-4ae4-a1cb-c8f9dec579ff","Type":"ContainerStarted","Data":"99f07b3d16f2d0e4fac6ae1cbe6e0cc505e1793629c6a08a67069416f60053b5"} Nov 03 22:21:50.198156 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:50.198086 2541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" podStartSLOduration=2.645696658 podStartE2EDuration="12.198068271s" totalImagesPullingTime="9.552371613s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:21:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:50.197757202 +0000 UTC m=+12.714953784" watchObservedRunningTime="2026-11-03 22:21:50.198068271 +0000 UTC m=+12.715264843" Nov 03 22:21:50.198779 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:50.198738 2541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-4gfds" podStartSLOduration=3.932056886 podStartE2EDuration="12.198728501s" totalImagesPullingTime="8.266671615s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:49.419786752 +0000 UTC m=+11.936983323" watchObservedRunningTime="2026-11-03 22:21:50.198728501 +0000 UTC m=+12.715925081" Nov 03 22:21:50.312946 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:50.312633 2541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-p55z4" podStartSLOduration=3.439773042 podStartE2EDuration="12.312615141s" totalImagesPullingTime="8.872842099s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:50.312071924 +0000 UTC m=+12.829268495" watchObservedRunningTime="2026-11-03 22:21:50.312615141 +0000 UTC m=+12.829811714" Nov 03 22:21:50.978217 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:50.978193 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:21:50.978217 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:50.978203 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:21:50.978217 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:50.978221 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:21:50.978516 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:50.978295 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjnj" podUID="29b82faa-1aa1-4760-a9f6-1cbeb9427094" Nov 03 22:21:50.978516 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:50.978358 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqppk" podUID="c027d694-64cc-46a2-b4e7-49b117f696d3" Nov 03 22:21:50.978516 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:50.978458 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8jxb" podUID="b6154c5a-4925-4c74-8db4-0acdb525b743" Nov 03 22:21:51.086365 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:51.086332 2541 generic.go:356] "Generic (PLEG): container finished" podID="d4425733-4528-4caa-99cb-e94072a00f78" containerID="087a1ae1427d3f7988ab424a8b7cbfc16d90a36dd835fa2c14b2f31e87d20790" exitCode=0 Nov 03 22:21:51.087115 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:51.086417 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sq9mk" event={"ID":"d4425733-4528-4caa-99cb-e94072a00f78","Type":"ContainerDied","Data":"087a1ae1427d3f7988ab424a8b7cbfc16d90a36dd835fa2c14b2f31e87d20790"} Nov 03 22:21:51.088101 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:51.088070 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j9x58" event={"ID":"1618fd8a-9a1b-4437-8559-86690d19cc9d","Type":"ContainerStarted","Data":"2b7a67a5bd9545a26445d927216d4ee77cbef0bc4895fa94d523ef973b863753"} Nov 03 22:21:52.978118 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:52.977919 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:21:52.978731 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:52.977976 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:21:52.978731 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:52.978214 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqppk" podUID="c027d694-64cc-46a2-b4e7-49b117f696d3" Nov 03 22:21:52.978731 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:52.978299 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjnj" podUID="29b82faa-1aa1-4760-a9f6-1cbeb9427094" Nov 03 22:21:52.978731 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:52.977988 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:21:52.978731 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:52.978410 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8jxb" podUID="b6154c5a-4925-4c74-8db4-0acdb525b743" Nov 03 22:21:53.095239 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:53.095209 2541 generic.go:356] "Generic (PLEG): container finished" podID="d4425733-4528-4caa-99cb-e94072a00f78" containerID="46697433e9e4a240671e4b7536319819b42ebf66560f3389b56c455073f17eaf" exitCode=0 Nov 03 22:21:53.095393 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:53.095262 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sq9mk" event={"ID":"d4425733-4528-4caa-99cb-e94072a00f78","Type":"ContainerDied","Data":"46697433e9e4a240671e4b7536319819b42ebf66560f3389b56c455073f17eaf"} Nov 03 22:21:53.317516 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:53.317380 2541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-j9x58" podStartSLOduration=4.495639931 podStartE2EDuration="15.317364776s" totalImagesPullingTime="10.821724845s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:51.325472138 +0000 UTC m=+13.842668710" watchObservedRunningTime="2026-11-03 22:21:53.317364776 +0000 UTC m=+15.834561347" Nov 03 22:21:54.883971 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:54.883939 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b6154c5a-4925-4c74-8db4-0acdb525b743-original-pull-secret\") pod \"global-pull-secret-syncer-h8jxb\" (UID: \"b6154c5a-4925-4c74-8db4-0acdb525b743\") " pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:21:54.884405 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:54.883978 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c027d694-64cc-46a2-b4e7-49b117f696d3-metrics-certs\") pod \"network-metrics-daemon-vqppk\" (UID: \"c027d694-64cc-46a2-b4e7-49b117f696d3\") " pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:21:54.884405 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:54.884108 2541 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:21:54.884405 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:54.884113 2541 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:21:54.884405 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:54.884167 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c027d694-64cc-46a2-b4e7-49b117f696d3-metrics-certs podName:c027d694-64cc-46a2-b4e7-49b117f696d3 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:10.884155066 +0000 UTC m=+33.401351615 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c027d694-64cc-46a2-b4e7-49b117f696d3-metrics-certs") pod "network-metrics-daemon-vqppk" (UID: "c027d694-64cc-46a2-b4e7-49b117f696d3") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:21:54.884405 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:54.884185 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6154c5a-4925-4c74-8db4-0acdb525b743-original-pull-secret podName:b6154c5a-4925-4c74-8db4-0acdb525b743 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:10.884176037 +0000 UTC m=+33.401372590 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b6154c5a-4925-4c74-8db4-0acdb525b743-original-pull-secret") pod "global-pull-secret-syncer-h8jxb" (UID: "b6154c5a-4925-4c74-8db4-0acdb525b743") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:21:54.978166 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:54.978137 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:21:54.978166 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:54.978159 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:21:54.978379 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:54.978148 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:21:54.978379 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:54.978266 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjnj" podUID="29b82faa-1aa1-4760-a9f6-1cbeb9427094" Nov 03 22:21:54.978447 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:54.978381 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqppk" podUID="c027d694-64cc-46a2-b4e7-49b117f696d3" Nov 03 22:21:54.978447 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:54.978432 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8jxb" podUID="b6154c5a-4925-4c74-8db4-0acdb525b743" Nov 03 22:21:54.984292 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:54.984270 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5zc6\" (UniqueName: \"kubernetes.io/projected/29b82faa-1aa1-4760-a9f6-1cbeb9427094-kube-api-access-g5zc6\") pod \"network-check-target-jkjnj\" (UID: \"29b82faa-1aa1-4760-a9f6-1cbeb9427094\") " pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:21:54.984452 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:54.984435 2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:21:54.984498 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:54.984460 2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:21:54.984498 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:54.984473 2541 projected.go:196] Error preparing data for projected volume kube-api-access-g5zc6 for pod openshift-network-diagnostics/network-check-target-jkjnj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:21:54.984569 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:54.984524 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29b82faa-1aa1-4760-a9f6-1cbeb9427094-kube-api-access-g5zc6 podName:29b82faa-1aa1-4760-a9f6-1cbeb9427094 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:10.984511467 +0000 UTC m=+33.501708015 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-g5zc6" (UniqueName: "kubernetes.io/projected/29b82faa-1aa1-4760-a9f6-1cbeb9427094-kube-api-access-g5zc6") pod "network-check-target-jkjnj" (UID: "29b82faa-1aa1-4760-a9f6-1cbeb9427094") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:21:56.105040 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:56.104868 2541 generic.go:356] "Generic (PLEG): container finished" podID="d4425733-4528-4caa-99cb-e94072a00f78" containerID="79927d94fab030ee9dba1fa91448445f8199385057ce029c1b0d8ed2b8818b2c" exitCode=0 Nov 03 22:21:56.105471 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:56.104954 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sq9mk" event={"ID":"d4425733-4528-4caa-99cb-e94072a00f78","Type":"ContainerDied","Data":"79927d94fab030ee9dba1fa91448445f8199385057ce029c1b0d8ed2b8818b2c"} Nov 03 22:21:56.110974 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:56.110951 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" event={"ID":"1b2df441-74d4-435f-a56a-5eff3ec4575d","Type":"ContainerStarted","Data":"c5f83338edea88075f3ab3793219fcf7fbcd93c618a6c9398f20fe071a5c05b7"} Nov 03 22:21:56.111072 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:56.111015 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" event={"ID":"1b2df441-74d4-435f-a56a-5eff3ec4575d","Type":"ContainerStarted","Data":"24c303dfe5fe1a8820c86661b081cda9114158e3fd532c0f61e8ea908010ccad"} Nov 03 22:21:56.111072 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:56.111055 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" event={"ID":"1b2df441-74d4-435f-a56a-5eff3ec4575d","Type":"ContainerStarted","Data":"157802195c6b330f58c534fbbbe1cbbf61c180ab6fe4f2083ab38506808263d7"} Nov 03 22:21:56.111150 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:56.111083 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" event={"ID":"1b2df441-74d4-435f-a56a-5eff3ec4575d","Type":"ContainerStarted","Data":"1eb96e95abdfa0f87833f00f63f3b77ca8af145ae24d1d104c39a4c21c369886"} Nov 03 22:21:56.111150 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:56.111113 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" event={"ID":"1b2df441-74d4-435f-a56a-5eff3ec4575d","Type":"ContainerStarted","Data":"418485658c1e151986164f71adf7010840f2807a9334f27838a7676af2f436d3"} Nov 03 22:21:56.111150 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:56.111147 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" event={"ID":"1b2df441-74d4-435f-a56a-5eff3ec4575d","Type":"ContainerStarted","Data":"d34cb8c8a93b08d151ae831d25182e1c411c5dd63bf5630b5bfe690b2362b490"} Nov 03 22:21:56.978500 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:56.978461 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:21:56.978500 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:56.978487 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:21:56.978718 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:56.978605 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8jxb" podUID="b6154c5a-4925-4c74-8db4-0acdb525b743" Nov 03 22:21:56.978718 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:56.978679 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqppk" podUID="c027d694-64cc-46a2-b4e7-49b117f696d3" Nov 03 22:21:56.978838 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:56.978732 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:21:56.978910 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:56.978840 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjnj" podUID="29b82faa-1aa1-4760-a9f6-1cbeb9427094" Nov 03 22:21:57.115829 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:57.115798 2541 generic.go:356] "Generic (PLEG): container finished" podID="d4425733-4528-4caa-99cb-e94072a00f78" containerID="c92df619635d1a29a4909fe0ed9549ca9a6a64d8db7bbbfeff74404ae7ae1078" exitCode=0 Nov 03 22:21:57.116236 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:57.115877 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sq9mk" event={"ID":"d4425733-4528-4caa-99cb-e94072a00f78","Type":"ContainerDied","Data":"c92df619635d1a29a4909fe0ed9549ca9a6a64d8db7bbbfeff74404ae7ae1078"} Nov 03 22:21:57.216078 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:57.216054 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:57.411142 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:57.411103 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:58.120835 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:58.120736 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" event={"ID":"1b2df441-74d4-435f-a56a-5eff3ec4575d","Type":"ContainerStarted","Data":"f0bdbf0eab8b061dceff2d97ff79d8cd7d524d4d923638e5dad59ce3a0c38f55"} Nov 03 22:21:58.124043 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:58.124018 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sq9mk" event={"ID":"d4425733-4528-4caa-99cb-e94072a00f78","Type":"ContainerStarted","Data":"59aa6440b8666a11d5176c320e3db29dcb4a799ce911466024eb18703e595a00"} Nov 03 22:21:58.246911 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:58.246839 2541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-sq9mk" podStartSLOduration=7.25614502 podStartE2EDuration="20.246827228s" totalImagesPullingTime="12.990682208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:21:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:58.246506635 +0000 UTC m=+20.763703207" watchObservedRunningTime="2026-11-03 22:21:58.246827228 +0000 UTC m=+20.764023799" Nov 03 22:21:58.642588 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:58.642560 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:21:58.978437 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:58.978354 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:21:58.978582 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:58.978444 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:21:58.978582 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:58.978458 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:21:58.978582 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:58.978538 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqppk" podUID="c027d694-64cc-46a2-b4e7-49b117f696d3" Nov 03 22:21:58.978669 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:58.978613 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8jxb" podUID="b6154c5a-4925-4c74-8db4-0acdb525b743" Nov 03 22:21:58.978729 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:21:58.978713 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjnj" podUID="29b82faa-1aa1-4760-a9f6-1cbeb9427094" Nov 03 22:21:59.110319 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:21:59.110290 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:00.131671 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:00.131630 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" event={"ID":"1b2df441-74d4-435f-a56a-5eff3ec4575d","Type":"ContainerStarted","Data":"859cf8bc58736d7791064c51928b893c45b856bb2049c45912fe1704ddc89951"} Nov 03 22:22:00.132561 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:00.131848 2541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:00.147729 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:00.147577 2541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:00.410843 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:00.410795 2541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" podStartSLOduration=7.300817301 podStartE2EDuration="22.410777694s" totalImagesPullingTime="15.109960393s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:00.410634543 +0000 UTC m=+22.927831114" watchObservedRunningTime="2026-11-03 22:22:00.410777694 +0000 UTC m=+22.927974264" Nov 03 22:22:00.978243 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:00.978214 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:22:00.978496 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:00.978292 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:22:00.978496 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:00.978380 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:22:00.978496 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:00.978397 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqppk" podUID="c027d694-64cc-46a2-b4e7-49b117f696d3" Nov 03 22:22:00.978496 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:00.978442 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8jxb" podUID="b6154c5a-4925-4c74-8db4-0acdb525b743" Nov 03 22:22:00.978646 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:00.978514 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjnj" podUID="29b82faa-1aa1-4760-a9f6-1cbeb9427094" Nov 03 22:22:01.134309 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:01.134280 2541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:01.134762 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:01.134340 2541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:01.151209 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:01.151182 2541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:02.117888 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:02.116925 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:02.259980 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:02.259952 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:02.439476 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:02.439443 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:02.590741 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:02.590706 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:02.729983 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:02.729903 2541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h8jxb"] Nov 03 22:22:02.730125 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:02.730069 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:22:02.730203 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:02.730179 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8jxb" podUID="b6154c5a-4925-4c74-8db4-0acdb525b743" Nov 03 22:22:02.732891 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:02.732852 2541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jkjnj"] Nov 03 22:22:02.733012 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:02.732951 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:22:02.733074 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:02.733015 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjnj" podUID="29b82faa-1aa1-4760-a9f6-1cbeb9427094" Nov 03 22:22:02.737977 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:02.737955 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:02.779917 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:02.779892 2541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vqppk"] Nov 03 22:22:02.780005 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:02.779990 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:22:02.780085 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:02.780069 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqppk" podUID="c027d694-64cc-46a2-b4e7-49b117f696d3" Nov 03 22:22:02.877576 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:02.877546 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:03.032254 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:03.032168 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:04.329227 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:04.329163 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:04.515037 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:04.515009 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:04.745292 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:04.745261 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:04.978233 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:04.978149 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:22:04.978233 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:04.978177 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:22:04.978414 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:04.978235 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8jxb" podUID="b6154c5a-4925-4c74-8db4-0acdb525b743" Nov 03 22:22:04.978414 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:04.978291 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:22:04.978414 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:04.978387 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqppk" podUID="c027d694-64cc-46a2-b4e7-49b117f696d3" Nov 03 22:22:04.978523 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:04.978456 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjnj" podUID="29b82faa-1aa1-4760-a9f6-1cbeb9427094" Nov 03 22:22:05.619504 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:05.619476 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:05.760256 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:05.760226 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:06.259657 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:06.259625 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:06.395162 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:06.395139 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k4h5g_3f63f886-379d-4636-8fb3-21b932a67831/dns-node-resolver/0.log" Nov 03 22:22:06.977982 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:06.977951 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:22:06.977982 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:06.977977 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:22:06.978419 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:06.978056 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8jxb" podUID="b6154c5a-4925-4c74-8db4-0acdb525b743" Nov 03 22:22:06.978419 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:06.978095 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:22:06.978419 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:06.978158 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjnj" podUID="29b82faa-1aa1-4760-a9f6-1cbeb9427094" Nov 03 22:22:06.978419 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:06.978250 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqppk" podUID="c027d694-64cc-46a2-b4e7-49b117f696d3" Nov 03 22:22:07.591771 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:07.591746 2541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t5l5t_514a860a-f1e3-4714-8493-367b6b8b32a2/node-ca/0.log" Nov 03 22:22:08.197109 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:08.197080 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:08.340648 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:08.340618 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:08.486021 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:08.485993 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:08.977692 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:08.977667 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:22:08.977852 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:08.977698 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:22:08.977852 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:08.977757 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:22:08.977852 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:08.977769 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjnj" podUID="29b82faa-1aa1-4760-a9f6-1cbeb9427094" Nov 03 22:22:08.977852 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:08.977831 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8jxb" podUID="b6154c5a-4925-4c74-8db4-0acdb525b743" Nov 03 22:22:08.978005 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:08.977924 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqppk" podUID="c027d694-64cc-46a2-b4e7-49b117f696d3" Nov 03 22:22:09.131906 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:09.131873 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:09.349572 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:09.349534 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:09.489344 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:09.489314 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:09.672684 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:09.672608 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:09.867008 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:09.866966 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:10.010373 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:10.010300 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:10.162247 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:10.162222 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:10.900489 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:10.900459 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b6154c5a-4925-4c74-8db4-0acdb525b743-original-pull-secret\") pod \"global-pull-secret-syncer-h8jxb\" (UID: \"b6154c5a-4925-4c74-8db4-0acdb525b743\") " pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:22:10.900489 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:10.900493 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c027d694-64cc-46a2-b4e7-49b117f696d3-metrics-certs\") pod \"network-metrics-daemon-vqppk\" (UID: \"c027d694-64cc-46a2-b4e7-49b117f696d3\") " pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:22:10.900929 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:10.900606 2541 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:22:10.900929 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:10.900606 2541 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:22:10.900929 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:10.900667 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6154c5a-4925-4c74-8db4-0acdb525b743-original-pull-secret podName:b6154c5a-4925-4c74-8db4-0acdb525b743 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:42.90064947 +0000 UTC m=+65.417846033 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b6154c5a-4925-4c74-8db4-0acdb525b743-original-pull-secret") pod "global-pull-secret-syncer-h8jxb" (UID: "b6154c5a-4925-4c74-8db4-0acdb525b743") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:22:10.900929 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:10.900682 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c027d694-64cc-46a2-b4e7-49b117f696d3-metrics-certs podName:c027d694-64cc-46a2-b4e7-49b117f696d3 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:42.90067651 +0000 UTC m=+65.417873059 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c027d694-64cc-46a2-b4e7-49b117f696d3-metrics-certs") pod "network-metrics-daemon-vqppk" (UID: "c027d694-64cc-46a2-b4e7-49b117f696d3") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:22:10.978479 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:10.978452 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:22:10.978479 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:10.978480 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:22:10.978665 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:10.978553 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8jxb" podUID="b6154c5a-4925-4c74-8db4-0acdb525b743" Nov 03 22:22:10.978665 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:10.978600 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:22:10.978767 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:10.978662 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjnj" podUID="29b82faa-1aa1-4760-a9f6-1cbeb9427094" Nov 03 22:22:10.978767 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:10.978731 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqppk" podUID="c027d694-64cc-46a2-b4e7-49b117f696d3" Nov 03 22:22:11.000988 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:11.000966 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5zc6\" (UniqueName: \"kubernetes.io/projected/29b82faa-1aa1-4760-a9f6-1cbeb9427094-kube-api-access-g5zc6\") pod \"network-check-target-jkjnj\" (UID: \"29b82faa-1aa1-4760-a9f6-1cbeb9427094\") " pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:22:11.001124 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:11.001110 2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:22:11.001170 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:11.001127 2541 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:22:11.001170 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:11.001137 2541 projected.go:196] Error preparing data for projected volume kube-api-access-g5zc6 for pod openshift-network-diagnostics/network-check-target-jkjnj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:22:11.001227 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:11.001185 2541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29b82faa-1aa1-4760-a9f6-1cbeb9427094-kube-api-access-g5zc6 podName:29b82faa-1aa1-4760-a9f6-1cbeb9427094 nodeName:}" failed. No retries permitted until 2026-11-03 22:22:43.001174075 +0000 UTC m=+65.518370624 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-g5zc6" (UniqueName: "kubernetes.io/projected/29b82faa-1aa1-4760-a9f6-1cbeb9427094-kube-api-access-g5zc6") pod "network-check-target-jkjnj" (UID: "29b82faa-1aa1-4760-a9f6-1cbeb9427094") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:22:11.056772 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:11.056749 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:11.259371 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:11.259295 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:11.456717 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:11.456683 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:11.659898 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:11.659867 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:11.854032 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:11.854001 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:12.977653 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:12.977620 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:22:12.978047 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:12.977626 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:22:12.978047 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:12.977722 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8jxb" podUID="b6154c5a-4925-4c74-8db4-0acdb525b743" Nov 03 22:22:12.978047 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:12.977809 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqppk" podUID="c027d694-64cc-46a2-b4e7-49b117f696d3" Nov 03 22:22:12.978047 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:12.977626 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:22:12.978047 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:12.977904 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjnj" podUID="29b82faa-1aa1-4760-a9f6-1cbeb9427094" Nov 03 22:22:12.983001 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:12.982982 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:13.122354 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:13.122323 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:13.266216 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:13.266143 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:13.461356 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:13.461330 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:13.605272 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:13.605243 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:13.744154 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:13.744125 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:13.882016 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:13.881947 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:14.978347 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:14.978315 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:22:14.978837 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:14.978384 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:22:14.978837 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:14.978501 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:22:14.978837 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:14.978516 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqppk" podUID="c027d694-64cc-46a2-b4e7-49b117f696d3" Nov 03 22:22:14.978837 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:14.978612 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjnj" podUID="29b82faa-1aa1-4760-a9f6-1cbeb9427094" Nov 03 22:22:14.978837 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:14.978675 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8jxb" podUID="b6154c5a-4925-4c74-8db4-0acdb525b743" Nov 03 22:22:15.055587 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:15.055564 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:15.261757 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:15.261680 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:15.457889 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:15.457828 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:15.674494 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:15.674465 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:15.847766 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:15.847736 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:15.985990 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:15.985917 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:16.700788 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:16.700762 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:16.978606 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:16.978531 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:22:16.978606 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:16.978561 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:22:16.978816 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:16.978635 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8jxb" podUID="b6154c5a-4925-4c74-8db4-0acdb525b743" Nov 03 22:22:16.978816 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:16.978660 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:22:16.978816 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:16.978741 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqppk" podUID="c027d694-64cc-46a2-b4e7-49b117f696d3" Nov 03 22:22:16.978816 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:16.978808 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjnj" podUID="29b82faa-1aa1-4760-a9f6-1cbeb9427094" Nov 03 22:22:17.060234 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:17.060207 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:17.260051 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:17.259975 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:17.457093 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:17.457068 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:17.659015 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:17.658991 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:17.855544 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:17.855514 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:18.014389 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:18.014321 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:18.977933 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:18.977894 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:22:18.977933 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:18.977931 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:22:18.978461 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:18.978012 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:22:18.978461 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:18.978030 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqppk" podUID="c027d694-64cc-46a2-b4e7-49b117f696d3" Nov 03 22:22:18.978461 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:18.978080 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjnj" podUID="29b82faa-1aa1-4760-a9f6-1cbeb9427094" Nov 03 22:22:18.978461 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:18.978150 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8jxb" podUID="b6154c5a-4925-4c74-8db4-0acdb525b743" Nov 03 22:22:19.056526 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:19.056496 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:19.200848 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:19.200802 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:19.342767 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:19.342738 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:19.484823 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:19.484773 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:19.654694 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:19.654610 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:19.855990 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:19.855952 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:19.994418 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:19.994344 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:20.465142 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:20.465118 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:20.657997 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:20.657971 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:20.978363 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:20.978338 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:22:20.978363 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:20.978369 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:22:20.978573 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:20.978450 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:22:20.978573 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:20.978457 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqppk" podUID="c027d694-64cc-46a2-b4e7-49b117f696d3" Nov 03 22:22:20.978573 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:20.978519 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8jxb" podUID="b6154c5a-4925-4c74-8db4-0acdb525b743" Nov 03 22:22:20.978703 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:20.978614 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjnj" podUID="29b82faa-1aa1-4760-a9f6-1cbeb9427094" Nov 03 22:22:22.978576 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:22.978544 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:22:22.978576 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:22.978571 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:22:22.979161 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:22.978637 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8jxb" podUID="b6154c5a-4925-4c74-8db4-0acdb525b743" Nov 03 22:22:22.979161 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:22.978727 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:22:22.979161 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:22.978837 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjnj" podUID="29b82faa-1aa1-4760-a9f6-1cbeb9427094" Nov 03 22:22:22.979161 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:22.978958 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqppk" podUID="c027d694-64cc-46a2-b4e7-49b117f696d3" Nov 03 22:22:24.978238 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:24.978200 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:22:24.978739 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:24.978354 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:22:24.978739 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:24.978381 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:22:24.978739 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:24.978373 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqppk" podUID="c027d694-64cc-46a2-b4e7-49b117f696d3" Nov 03 22:22:24.978739 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:24.978463 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8jxb" podUID="b6154c5a-4925-4c74-8db4-0acdb525b743" Nov 03 22:22:24.978739 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:24.978532 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjnj" podUID="29b82faa-1aa1-4760-a9f6-1cbeb9427094" Nov 03 22:22:26.627582 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:26.627543 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:26.834077 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:26.834044 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:26.978338 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:26.978252 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:22:26.978338 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:26.978303 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:22:26.978549 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:26.978260 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:22:26.978549 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:26.978385 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vqppk" podUID="c027d694-64cc-46a2-b4e7-49b117f696d3" Nov 03 22:22:26.978549 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:26.978463 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8jxb" podUID="b6154c5a-4925-4c74-8db4-0acdb525b743" Nov 03 22:22:26.978549 ip-10-0-143-132 kubenswrapper[2541]: E1103 22:22:26.978532 2541 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jkjnj" podUID="29b82faa-1aa1-4760-a9f6-1cbeb9427094" Nov 03 22:22:26.980194 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:26.980169 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:27.199312 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:27.199282 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:27.346065 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:27.346037 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:27.711962 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:27.711932 2541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeReady" Nov 03 22:22:27.712335 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:27.712053 2541 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 22:22:27.867971 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:27.867934 2541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6d49h"] Nov 03 22:22:27.869537 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:27.869521 2541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jwm6k"] Nov 03 22:22:27.869673 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:27.869658 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6d49h" Nov 03 22:22:27.871416 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:27.871392 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jwm6k" Nov 03 22:22:27.878762 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:27.878740 2541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6d49h"] Nov 03 22:22:27.880791 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:27.880766 2541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jwm6k"] Nov 03 22:22:27.912268 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:27.912246 2541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gw5gm"] Nov 03 22:22:27.913785 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:27.913771 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gw5gm" Nov 03 22:22:27.922469 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:27.922441 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 22:22:27.922567 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:27.922485 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vc5c5\"" Nov 03 22:22:27.922567 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:27.922516 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 22:22:27.923484 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:27.923470 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-94lr2\"" Nov 03 22:22:27.923837 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:27.923823 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 22:22:27.929832 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:27.929819 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 22:22:27.929946 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:27.929933 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 22:22:27.930062 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:27.930051 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 22:22:27.981069 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:27.981022 2541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gw5gm"] Nov 03 22:22:28.019556 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.019526 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b5730579-cca5-46be-9fbb-da9f7ae58d14-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jwm6k\" (UID: \"b5730579-cca5-46be-9fbb-da9f7ae58d14\") " pod="openshift-insights/insights-runtime-extractor-jwm6k" Nov 03 22:22:28.019556 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.019556 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5730579-cca5-46be-9fbb-da9f7ae58d14-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jwm6k\" (UID: \"b5730579-cca5-46be-9fbb-da9f7ae58d14\") " pod="openshift-insights/insights-runtime-extractor-jwm6k" Nov 03 22:22:28.019698 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.019579 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b5730579-cca5-46be-9fbb-da9f7ae58d14-crio-socket\") pod \"insights-runtime-extractor-jwm6k\" (UID: \"b5730579-cca5-46be-9fbb-da9f7ae58d14\") " pod="openshift-insights/insights-runtime-extractor-jwm6k" Nov 03 22:22:28.019698 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.019602 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/254d9855-203d-43f2-9c75-ed2eb84c4471-config-volume\") pod \"dns-default-6d49h\" (UID: \"254d9855-203d-43f2-9c75-ed2eb84c4471\") " pod="openshift-dns/dns-default-6d49h" Nov 03 22:22:28.019698 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.019685 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/254d9855-203d-43f2-9c75-ed2eb84c4471-metrics-tls\") pod \"dns-default-6d49h\" (UID: \"254d9855-203d-43f2-9c75-ed2eb84c4471\") " pod="openshift-dns/dns-default-6d49h" Nov 03 22:22:28.019788 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.019727 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b5730579-cca5-46be-9fbb-da9f7ae58d14-data-volume\") pod \"insights-runtime-extractor-jwm6k\" (UID: \"b5730579-cca5-46be-9fbb-da9f7ae58d14\") " pod="openshift-insights/insights-runtime-extractor-jwm6k" Nov 03 22:22:28.019935 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.019790 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/254d9855-203d-43f2-9c75-ed2eb84c4471-tmp-dir\") pod \"dns-default-6d49h\" (UID: \"254d9855-203d-43f2-9c75-ed2eb84c4471\") " pod="openshift-dns/dns-default-6d49h" Nov 03 22:22:28.020176 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.020153 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf4a34cf-5e64-407a-aaf7-0d211a6be6f6-cert\") pod \"ingress-canary-gw5gm\" (UID: \"bf4a34cf-5e64-407a-aaf7-0d211a6be6f6\") " pod="openshift-ingress-canary/ingress-canary-gw5gm" Nov 03 22:22:28.022666 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.020330 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5vnj\" (UniqueName: \"kubernetes.io/projected/b5730579-cca5-46be-9fbb-da9f7ae58d14-kube-api-access-q5vnj\") pod \"insights-runtime-extractor-jwm6k\" (UID: \"b5730579-cca5-46be-9fbb-da9f7ae58d14\") " pod="openshift-insights/insights-runtime-extractor-jwm6k" Nov 03 22:22:28.022666 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.020389 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fhh6\" (UniqueName: \"kubernetes.io/projected/bf4a34cf-5e64-407a-aaf7-0d211a6be6f6-kube-api-access-4fhh6\") pod \"ingress-canary-gw5gm\" (UID: \"bf4a34cf-5e64-407a-aaf7-0d211a6be6f6\") " pod="openshift-ingress-canary/ingress-canary-gw5gm" Nov 03 22:22:28.022666 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.020431 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfsqx\" (UniqueName: \"kubernetes.io/projected/254d9855-203d-43f2-9c75-ed2eb84c4471-kube-api-access-kfsqx\") pod \"dns-default-6d49h\" (UID: \"254d9855-203d-43f2-9c75-ed2eb84c4471\") " pod="openshift-dns/dns-default-6d49h" Nov 03 22:22:28.023145 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.023128 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 22:22:28.023145 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.023141 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 22:22:28.023278 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.023164 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 22:22:28.023278 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.023220 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-rrf2w\"" Nov 03 22:22:28.121372 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.121340 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/254d9855-203d-43f2-9c75-ed2eb84c4471-metrics-tls\") pod \"dns-default-6d49h\" (UID: \"254d9855-203d-43f2-9c75-ed2eb84c4471\") " pod="openshift-dns/dns-default-6d49h" Nov 03 22:22:28.121372 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.121375 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b5730579-cca5-46be-9fbb-da9f7ae58d14-data-volume\") pod \"insights-runtime-extractor-jwm6k\" (UID: \"b5730579-cca5-46be-9fbb-da9f7ae58d14\") " pod="openshift-insights/insights-runtime-extractor-jwm6k" Nov 03 22:22:28.121599 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.121401 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/254d9855-203d-43f2-9c75-ed2eb84c4471-tmp-dir\") pod \"dns-default-6d49h\" (UID: \"254d9855-203d-43f2-9c75-ed2eb84c4471\") " pod="openshift-dns/dns-default-6d49h" Nov 03 22:22:28.121599 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.121427 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf4a34cf-5e64-407a-aaf7-0d211a6be6f6-cert\") pod \"ingress-canary-gw5gm\" (UID: \"bf4a34cf-5e64-407a-aaf7-0d211a6be6f6\") " pod="openshift-ingress-canary/ingress-canary-gw5gm" Nov 03 22:22:28.121599 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.121494 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5vnj\" (UniqueName: \"kubernetes.io/projected/b5730579-cca5-46be-9fbb-da9f7ae58d14-kube-api-access-q5vnj\") pod \"insights-runtime-extractor-jwm6k\" (UID: \"b5730579-cca5-46be-9fbb-da9f7ae58d14\") " pod="openshift-insights/insights-runtime-extractor-jwm6k" Nov 03 22:22:28.121743 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.121620 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fhh6\" (UniqueName: \"kubernetes.io/projected/bf4a34cf-5e64-407a-aaf7-0d211a6be6f6-kube-api-access-4fhh6\") pod \"ingress-canary-gw5gm\" (UID: \"bf4a34cf-5e64-407a-aaf7-0d211a6be6f6\") " pod="openshift-ingress-canary/ingress-canary-gw5gm" Nov 03 22:22:28.121743 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.121665 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfsqx\" (UniqueName: \"kubernetes.io/projected/254d9855-203d-43f2-9c75-ed2eb84c4471-kube-api-access-kfsqx\") pod \"dns-default-6d49h\" (UID: \"254d9855-203d-43f2-9c75-ed2eb84c4471\") " pod="openshift-dns/dns-default-6d49h" Nov 03 22:22:28.121743 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.121693 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b5730579-cca5-46be-9fbb-da9f7ae58d14-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jwm6k\" (UID: \"b5730579-cca5-46be-9fbb-da9f7ae58d14\") " pod="openshift-insights/insights-runtime-extractor-jwm6k" Nov 03 22:22:28.121743 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.121716 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5730579-cca5-46be-9fbb-da9f7ae58d14-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jwm6k\" (UID: \"b5730579-cca5-46be-9fbb-da9f7ae58d14\") " pod="openshift-insights/insights-runtime-extractor-jwm6k" Nov 03 22:22:28.121972 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.121745 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b5730579-cca5-46be-9fbb-da9f7ae58d14-crio-socket\") pod \"insights-runtime-extractor-jwm6k\" (UID: \"b5730579-cca5-46be-9fbb-da9f7ae58d14\") " pod="openshift-insights/insights-runtime-extractor-jwm6k" Nov 03 22:22:28.121972 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.121779 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/254d9855-203d-43f2-9c75-ed2eb84c4471-config-volume\") pod \"dns-default-6d49h\" (UID: \"254d9855-203d-43f2-9c75-ed2eb84c4471\") " pod="openshift-dns/dns-default-6d49h" Nov 03 22:22:28.121972 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.121791 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b5730579-cca5-46be-9fbb-da9f7ae58d14-data-volume\") pod \"insights-runtime-extractor-jwm6k\" (UID: \"b5730579-cca5-46be-9fbb-da9f7ae58d14\") " pod="openshift-insights/insights-runtime-extractor-jwm6k" Nov 03 22:22:28.121972 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.121816 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/254d9855-203d-43f2-9c75-ed2eb84c4471-tmp-dir\") pod \"dns-default-6d49h\" (UID: \"254d9855-203d-43f2-9c75-ed2eb84c4471\") " pod="openshift-dns/dns-default-6d49h" Nov 03 22:22:28.122167 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.122080 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b5730579-cca5-46be-9fbb-da9f7ae58d14-crio-socket\") pod \"insights-runtime-extractor-jwm6k\" (UID: \"b5730579-cca5-46be-9fbb-da9f7ae58d14\") " pod="openshift-insights/insights-runtime-extractor-jwm6k" Nov 03 22:22:28.122331 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.122310 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b5730579-cca5-46be-9fbb-da9f7ae58d14-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jwm6k\" (UID: \"b5730579-cca5-46be-9fbb-da9f7ae58d14\") " pod="openshift-insights/insights-runtime-extractor-jwm6k" Nov 03 22:22:28.122393 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.122346 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/254d9855-203d-43f2-9c75-ed2eb84c4471-config-volume\") pod \"dns-default-6d49h\" (UID: \"254d9855-203d-43f2-9c75-ed2eb84c4471\") " pod="openshift-dns/dns-default-6d49h" Nov 03 22:22:28.125121 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.125098 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/254d9855-203d-43f2-9c75-ed2eb84c4471-metrics-tls\") pod \"dns-default-6d49h\" (UID: \"254d9855-203d-43f2-9c75-ed2eb84c4471\") " pod="openshift-dns/dns-default-6d49h" Nov 03 22:22:28.125205 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.125140 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5730579-cca5-46be-9fbb-da9f7ae58d14-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jwm6k\" (UID: \"b5730579-cca5-46be-9fbb-da9f7ae58d14\") " pod="openshift-insights/insights-runtime-extractor-jwm6k" Nov 03 22:22:28.125275 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.125259 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf4a34cf-5e64-407a-aaf7-0d211a6be6f6-cert\") pod \"ingress-canary-gw5gm\" (UID: \"bf4a34cf-5e64-407a-aaf7-0d211a6be6f6\") " pod="openshift-ingress-canary/ingress-canary-gw5gm" Nov 03 22:22:28.182919 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.182875 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfsqx\" (UniqueName: \"kubernetes.io/projected/254d9855-203d-43f2-9c75-ed2eb84c4471-kube-api-access-kfsqx\") pod \"dns-default-6d49h\" (UID: \"254d9855-203d-43f2-9c75-ed2eb84c4471\") " pod="openshift-dns/dns-default-6d49h" Nov 03 22:22:28.182919 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.182907 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5vnj\" (UniqueName: \"kubernetes.io/projected/b5730579-cca5-46be-9fbb-da9f7ae58d14-kube-api-access-q5vnj\") pod \"insights-runtime-extractor-jwm6k\" (UID: \"b5730579-cca5-46be-9fbb-da9f7ae58d14\") " pod="openshift-insights/insights-runtime-extractor-jwm6k" Nov 03 22:22:28.183091 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.183031 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fhh6\" (UniqueName: \"kubernetes.io/projected/bf4a34cf-5e64-407a-aaf7-0d211a6be6f6-kube-api-access-4fhh6\") pod \"ingress-canary-gw5gm\" (UID: \"bf4a34cf-5e64-407a-aaf7-0d211a6be6f6\") " pod="openshift-ingress-canary/ingress-canary-gw5gm" Nov 03 22:22:28.184705 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.184693 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6d49h" Nov 03 22:22:28.190019 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.189996 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jwm6k" Nov 03 22:22:28.223933 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.223899 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gw5gm" Nov 03 22:22:28.550206 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.550093 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:28.608382 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.608339 2541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6d49h"] Nov 03 22:22:28.614305 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:22:28.614279 2541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod254d9855_203d_43f2_9c75_ed2eb84c4471.slice/crio-843ca26127fa43d9fd34ef88d0fae215a9fb7bcb3e65d1925511dc7d0fa9e843 WatchSource:0}: Error finding container 843ca26127fa43d9fd34ef88d0fae215a9fb7bcb3e65d1925511dc7d0fa9e843: Status 404 returned error can't find the container with id 843ca26127fa43d9fd34ef88d0fae215a9fb7bcb3e65d1925511dc7d0fa9e843 Nov 03 22:22:28.624973 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.624950 2541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gw5gm"] Nov 03 22:22:28.628392 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:22:28.628367 2541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf4a34cf_5e64_407a_aaf7_0d211a6be6f6.slice/crio-2cd7eb9d637d8de82bc72f1c4d6cf376f2e35ba9e47be08ddd28756f0239bc7f WatchSource:0}: Error finding container 2cd7eb9d637d8de82bc72f1c4d6cf376f2e35ba9e47be08ddd28756f0239bc7f: Status 404 returned error can't find the container with id 2cd7eb9d637d8de82bc72f1c4d6cf376f2e35ba9e47be08ddd28756f0239bc7f Nov 03 22:22:28.628480 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.628453 2541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jwm6k"] Nov 03 22:22:28.644078 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:22:28.644054 2541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5730579_cca5_46be_9fbb_da9f7ae58d14.slice/crio-34f8c7a6a612c6b18d7e00d44eaeccd5bf769ade6933c792f4c3565d04ab8dde WatchSource:0}: Error finding container 34f8c7a6a612c6b18d7e00d44eaeccd5bf769ade6933c792f4c3565d04ab8dde: Status 404 returned error can't find the container with id 34f8c7a6a612c6b18d7e00d44eaeccd5bf769ade6933c792f4c3565d04ab8dde Nov 03 22:22:28.978491 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.978466 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:22:28.978957 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.978498 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:22:28.978957 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:28.978465 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:22:29.031257 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:29.031233 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 22:22:29.031357 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:29.031330 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 22:22:29.031720 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:29.031699 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ft6gs\"" Nov 03 22:22:29.031822 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:29.031718 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dnsfs\"" Nov 03 22:22:29.031822 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:29.031752 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 22:22:29.031822 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:29.031798 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 22:22:29.031954 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:29.031838 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-j25tw\"" Nov 03 22:22:29.193457 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:29.193421 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gw5gm" event={"ID":"bf4a34cf-5e64-407a-aaf7-0d211a6be6f6","Type":"ContainerStarted","Data":"2cd7eb9d637d8de82bc72f1c4d6cf376f2e35ba9e47be08ddd28756f0239bc7f"} Nov 03 22:22:29.194685 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:29.194659 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6d49h" event={"ID":"254d9855-203d-43f2-9c75-ed2eb84c4471","Type":"ContainerStarted","Data":"843ca26127fa43d9fd34ef88d0fae215a9fb7bcb3e65d1925511dc7d0fa9e843"} Nov 03 22:22:29.196110 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:29.196091 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jwm6k" event={"ID":"b5730579-cca5-46be-9fbb-da9f7ae58d14","Type":"ContainerStarted","Data":"4a7031e15261021f800347f7ceb45c7ad0d481dbe859309c3c31393df7bc4102"} Nov 03 22:22:29.196168 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:29.196131 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jwm6k" event={"ID":"b5730579-cca5-46be-9fbb-da9f7ae58d14","Type":"ContainerStarted","Data":"34f8c7a6a612c6b18d7e00d44eaeccd5bf769ade6933c792f4c3565d04ab8dde"} Nov 03 22:22:30.201192 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:30.201152 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jwm6k" event={"ID":"b5730579-cca5-46be-9fbb-da9f7ae58d14","Type":"ContainerStarted","Data":"eaebbee20febd0bc3c0b0752d1a478dc0a8741ce3ff5ad589c49ceb1720119b8"} Nov 03 22:22:30.985487 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:30.985384 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:31.130705 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:31.130656 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:31.206567 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:31.206528 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6d49h" event={"ID":"254d9855-203d-43f2-9c75-ed2eb84c4471","Type":"ContainerStarted","Data":"ff7449ad118bf6ea0daf701da27f8d3a6d1bf7ec2eb9e4cc96c7f2970924801b"} Nov 03 22:22:31.207066 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:31.206586 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6d49h" event={"ID":"254d9855-203d-43f2-9c75-ed2eb84c4471","Type":"ContainerStarted","Data":"eb138193b013ad34be1ea97e64e553668cdec46e69f4ca691f1c352c040030aa"} Nov 03 22:22:31.207066 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:31.206689 2541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6d49h" Nov 03 22:22:31.279631 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:31.279555 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:31.382536 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:31.382474 2541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-6d49h" podStartSLOduration=2.262867007 podStartE2EDuration="4.382457444s" totalImagesPullingTime="2.119590437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:31.325909404 +0000 UTC m=+53.843105978" watchObservedRunningTime="2026-11-03 22:22:31.382457444 +0000 UTC m=+53.899654009" Nov 03 22:22:31.382732 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:31.382628 2541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-qb4qm"] Nov 03 22:22:31.384739 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:31.384716 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qb4qm" Nov 03 22:22:31.428343 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:31.428312 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:31.438269 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:31.438243 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 03 22:22:31.438476 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:31.438323 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 22:22:31.438476 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:31.438328 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 22:22:31.438476 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:31.438371 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 03 22:22:31.438476 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:31.438410 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-9gf2r\"" Nov 03 22:22:31.487759 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:31.487703 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 22:22:31.488726 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:31.488694 2541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-qb4qm"] Nov 03 22:22:31.553132 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:31.553053 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6690a9b-44fa-4b1f-9523-173d8481dbea-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qb4qm\" (UID: \"e6690a9b-44fa-4b1f-9523-173d8481dbea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qb4qm" Nov 03 22:22:31.553303 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:31.553170 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6690a9b-44fa-4b1f-9523-173d8481dbea-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qb4qm\" (UID: \"e6690a9b-44fa-4b1f-9523-173d8481dbea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qb4qm" Nov 03 22:22:31.553303 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:31.553212 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e6690a9b-44fa-4b1f-9523-173d8481dbea-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qb4qm\" (UID: \"e6690a9b-44fa-4b1f-9523-173d8481dbea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qb4qm" Nov 03 22:22:31.553303 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:31.553256 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cst2m\" (UniqueName: \"kubernetes.io/projected/e6690a9b-44fa-4b1f-9523-173d8481dbea-kube-api-access-cst2m\") pod \"prometheus-operator-9cd89b74c-qb4qm\" (UID: \"e6690a9b-44fa-4b1f-9523-173d8481dbea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qb4qm" Nov 03 22:22:31.653990 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:31.653945 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e6690a9b-44fa-4b1f-9523-173d8481dbea-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qb4qm\" (UID: \"e6690a9b-44fa-4b1f-9523-173d8481dbea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qb4qm" Nov 03 22:22:31.654169 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:31.654011 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cst2m\" (UniqueName: \"kubernetes.io/projected/e6690a9b-44fa-4b1f-9523-173d8481dbea-kube-api-access-cst2m\") pod \"prometheus-operator-9cd89b74c-qb4qm\" (UID: \"e6690a9b-44fa-4b1f-9523-173d8481dbea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qb4qm" Nov 03 22:22:31.654169 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:31.654035 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6690a9b-44fa-4b1f-9523-173d8481dbea-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qb4qm\" (UID: \"e6690a9b-44fa-4b1f-9523-173d8481dbea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qb4qm" Nov 03 22:22:31.654169 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:31.654123 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6690a9b-44fa-4b1f-9523-173d8481dbea-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qb4qm\" (UID: \"e6690a9b-44fa-4b1f-9523-173d8481dbea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qb4qm" Nov 03 22:22:31.655326 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:31.655229 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6690a9b-44fa-4b1f-9523-173d8481dbea-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qb4qm\" (UID: \"e6690a9b-44fa-4b1f-9523-173d8481dbea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qb4qm" Nov 03 22:22:31.657934 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:31.657901 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e6690a9b-44fa-4b1f-9523-173d8481dbea-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qb4qm\" (UID: \"e6690a9b-44fa-4b1f-9523-173d8481dbea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qb4qm" Nov 03 22:22:31.658018 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:31.657936 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6690a9b-44fa-4b1f-9523-173d8481dbea-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qb4qm\" (UID: \"e6690a9b-44fa-4b1f-9523-173d8481dbea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qb4qm" Nov 03 22:22:31.716674 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:31.716645 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cst2m\" (UniqueName: \"kubernetes.io/projected/e6690a9b-44fa-4b1f-9523-173d8481dbea-kube-api-access-cst2m\") pod \"prometheus-operator-9cd89b74c-qb4qm\" (UID: \"e6690a9b-44fa-4b1f-9523-173d8481dbea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qb4qm" Nov 03 22:22:31.998208 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:31.998176 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qb4qm" Nov 03 22:22:32.211654 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:32.211559 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jwm6k" event={"ID":"b5730579-cca5-46be-9fbb-da9f7ae58d14","Type":"ContainerStarted","Data":"6b60ca37ec25bbf220ea57e24567d984ff94948442e0897508dc3df488b35120"} Nov 03 22:22:32.213111 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:32.213081 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gw5gm" event={"ID":"bf4a34cf-5e64-407a-aaf7-0d211a6be6f6","Type":"ContainerStarted","Data":"d8e7ccf49fbcaa4acf58637d6083e03a8b0ec67fb23c083b158c4adee7e41d6a"} Nov 03 22:22:32.334355 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:32.334279 2541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jwm6k" podStartSLOduration=2.405792296 podStartE2EDuration="5.33426265s" totalImagesPullingTime="2.928470354s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:22:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:32.333837233 +0000 UTC m=+54.851033803" watchObservedRunningTime="2026-11-03 22:22:32.33426265 +0000 UTC m=+54.851459223" Nov 03 22:22:32.371496 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:32.371470 2541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-qb4qm"] Nov 03 22:22:32.412004 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:22:32.411977 2541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6690a9b_44fa_4b1f_9523_173d8481dbea.slice/crio-36442bbfb7f8ef0153c97d3e49504fbf236257d4a7b120ce8403585766bb4b69 WatchSource:0}: Error finding container 36442bbfb7f8ef0153c97d3e49504fbf236257d4a7b120ce8403585766bb4b69: Status 404 returned error can't find the container with id 36442bbfb7f8ef0153c97d3e49504fbf236257d4a7b120ce8403585766bb4b69 Nov 03 22:22:32.449450 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:32.449410 2541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gw5gm" podStartSLOduration=2.236638625 podStartE2EDuration="5.449397268s" totalImagesPullingTime="3.212758643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:32.448500089 +0000 UTC m=+54.965696660" watchObservedRunningTime="2026-11-03 22:22:32.449397268 +0000 UTC m=+54.966593838" Nov 03 22:22:32.523398 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:32.523363 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:32.669164 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:32.669132 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:33.149466 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:33.149436 2541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:33.216756 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:33.216719 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qb4qm" event={"ID":"e6690a9b-44fa-4b1f-9523-173d8481dbea","Type":"ContainerStarted","Data":"36442bbfb7f8ef0153c97d3e49504fbf236257d4a7b120ce8403585766bb4b69"} Nov 03 22:22:34.221359 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:34.221328 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qb4qm" event={"ID":"e6690a9b-44fa-4b1f-9523-173d8481dbea","Type":"ContainerStarted","Data":"70dc80d6e09706f415b4e762cb16d00d17e83615951e6d319b3764f331818cdf"} Nov 03 22:22:34.221693 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:34.221391 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qb4qm" event={"ID":"e6690a9b-44fa-4b1f-9523-173d8481dbea","Type":"ContainerStarted","Data":"da74c2722e921045dfd623c7ff075205e54281722c223f370901b038f1b7f28e"} Nov 03 22:22:34.340830 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:34.340785 2541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qb4qm" podStartSLOduration=1.675435565 podStartE2EDuration="3.340771846s" totalImagesPullingTime="1.665336281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:34.340092493 +0000 UTC m=+56.857289065" watchObservedRunningTime="2026-11-03 22:22:34.340771846 +0000 UTC m=+56.857968416" Nov 03 22:22:35.434373 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:35.434341 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:36.257398 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:36.257366 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:36.434624 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:36.434590 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:36.661676 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:36.661647 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:36.854532 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:36.854496 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:37.059484 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:37.059402 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:37.092458 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:37.092428 2541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zt4dd"] Nov 03 22:22:37.094437 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:37.094423 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:37.147557 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:37.147531 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 22:22:37.197783 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:37.197753 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 22:22:37.197921 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:37.197841 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-q5zlz\"" Nov 03 22:22:37.197921 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:37.197842 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 22:22:37.215371 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:37.215348 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:37.287949 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:37.287919 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0e68894d-ed4f-4bc5-b36e-691e557883bd-node-exporter-tls\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:37.288089 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:37.287954 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e68894d-ed4f-4bc5-b36e-691e557883bd-metrics-client-ca\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:37.288089 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:37.287975 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0e68894d-ed4f-4bc5-b36e-691e557883bd-node-exporter-accelerators-collector-config\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:37.288163 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:37.288087 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e68894d-ed4f-4bc5-b36e-691e557883bd-sys\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:37.288163 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:37.288107 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0e68894d-ed4f-4bc5-b36e-691e557883bd-root\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:37.288163 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:37.288124 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0e68894d-ed4f-4bc5-b36e-691e557883bd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:37.288163 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:37.288154 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0e68894d-ed4f-4bc5-b36e-691e557883bd-node-exporter-textfile\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:37.288269 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:37.288191 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0e68894d-ed4f-4bc5-b36e-691e557883bd-node-exporter-wtmp\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:37.288303 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:37.288275 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7qg9\" (UniqueName: \"kubernetes.io/projected/0e68894d-ed4f-4bc5-b36e-691e557883bd-kube-api-access-z7qg9\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:37.367541 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:37.367511 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:37.389318 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:37.389290 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0e68894d-ed4f-4bc5-b36e-691e557883bd-node-exporter-textfile\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:37.389318 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:37.389319 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0e68894d-ed4f-4bc5-b36e-691e557883bd-node-exporter-wtmp\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:37.389457 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:37.389336 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7qg9\" (UniqueName: \"kubernetes.io/projected/0e68894d-ed4f-4bc5-b36e-691e557883bd-kube-api-access-z7qg9\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:37.389506 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:37.389453 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0e68894d-ed4f-4bc5-b36e-691e557883bd-node-exporter-tls\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:37.389506 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:37.389491 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0e68894d-ed4f-4bc5-b36e-691e557883bd-node-exporter-wtmp\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:37.389586 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:37.389493 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e68894d-ed4f-4bc5-b36e-691e557883bd-metrics-client-ca\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:37.389586 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:37.389541 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0e68894d-ed4f-4bc5-b36e-691e557883bd-node-exporter-accelerators-collector-config\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:37.389663 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:37.389585 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e68894d-ed4f-4bc5-b36e-691e557883bd-sys\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:37.389663 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:37.389610 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0e68894d-ed4f-4bc5-b36e-691e557883bd-root\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:37.389663 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:37.389636 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e68894d-ed4f-4bc5-b36e-691e557883bd-sys\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:37.389771 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:37.389639 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0e68894d-ed4f-4bc5-b36e-691e557883bd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:37.389771 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:37.389687 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0e68894d-ed4f-4bc5-b36e-691e557883bd-root\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:37.389771 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:37.389684 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0e68894d-ed4f-4bc5-b36e-691e557883bd-node-exporter-textfile\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:37.390005 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:37.389989 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e68894d-ed4f-4bc5-b36e-691e557883bd-metrics-client-ca\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:37.390130 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:37.390109 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0e68894d-ed4f-4bc5-b36e-691e557883bd-node-exporter-accelerators-collector-config\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:37.392033 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:37.392010 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0e68894d-ed4f-4bc5-b36e-691e557883bd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:37.392127 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:37.392098 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0e68894d-ed4f-4bc5-b36e-691e557883bd-node-exporter-tls\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:37.450739 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:37.450709 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7qg9\" (UniqueName: \"kubernetes.io/projected/0e68894d-ed4f-4bc5-b36e-691e557883bd-kube-api-access-z7qg9\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:37.547987 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:37.547950 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:37.705792 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:37.705717 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:37.715984 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:22:37.715962 2541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e68894d_ed4f_4bc5_b36e_691e557883bd.slice/crio-c8c68876c533e2f9e7aa85688046333b962cf0f2909987e8eb0a3e99c7afe8ab WatchSource:0}: Error finding container c8c68876c533e2f9e7aa85688046333b962cf0f2909987e8eb0a3e99c7afe8ab: Status 404 returned error can't find the container with id c8c68876c533e2f9e7aa85688046333b962cf0f2909987e8eb0a3e99c7afe8ab Nov 03 22:22:38.236487 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:38.236454 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zt4dd" event={"ID":"0e68894d-ed4f-4bc5-b36e-691e557883bd","Type":"ContainerStarted","Data":"c8c68876c533e2f9e7aa85688046333b962cf0f2909987e8eb0a3e99c7afe8ab"} Nov 03 22:22:38.278123 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:38.278093 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:38.468307 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:38.468274 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:38.623332 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:38.623298 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:38.769832 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:38.769801 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:39.142310 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:39.142287 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:39.242172 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:39.242132 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zt4dd" event={"ID":"0e68894d-ed4f-4bc5-b36e-691e557883bd","Type":"ContainerStarted","Data":"43a9341e30c80bc69e6aadaeb0894b714daec9c722a189d10ebe8ed6f0cedd12"} Nov 03 22:22:39.346826 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:39.346791 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:39.494284 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:39.494205 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:40.246047 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:40.246009 2541 generic.go:356] "Generic (PLEG): container finished" podID="0e68894d-ed4f-4bc5-b36e-691e557883bd" containerID="43a9341e30c80bc69e6aadaeb0894b714daec9c722a189d10ebe8ed6f0cedd12" exitCode=0 Nov 03 22:22:40.246213 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:40.246097 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zt4dd" event={"ID":"0e68894d-ed4f-4bc5-b36e-691e557883bd","Type":"ContainerDied","Data":"43a9341e30c80bc69e6aadaeb0894b714daec9c722a189d10ebe8ed6f0cedd12"} Nov 03 22:22:41.086113 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:41.086083 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:41.215058 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:41.215008 2541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6d49h" Nov 03 22:22:41.257923 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:41.257889 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zt4dd" event={"ID":"0e68894d-ed4f-4bc5-b36e-691e557883bd","Type":"ContainerStarted","Data":"b4b46c12b94477ed8456f73d147603cf07430415ec2649f774213ccc1590ed39"} Nov 03 22:22:41.258066 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:41.257968 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zt4dd" event={"ID":"0e68894d-ed4f-4bc5-b36e-691e557883bd","Type":"ContainerStarted","Data":"8e02cd23948b2cda656dfbd99fd4aa600591f05dad79df9c6010bb32ca378c46"} Nov 03 22:22:41.315026 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:41.314987 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:41.454042 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:41.453996 2541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zt4dd" podStartSLOduration=2.956671026 podStartE2EDuration="4.453982639s" totalImagesPullingTime="1.420453834s" totalInitContainerRuntime="76.857779ms" isStatefulPod=true podCreationTimestamp="2026-11-03 22:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:41.453437055 +0000 UTC m=+63.970633639" watchObservedRunningTime="2026-11-03 22:22:41.453982639 +0000 UTC m=+63.971179209" Nov 03 22:22:41.464376 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:41.464347 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:41.633112 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:41.633078 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:41.708334 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:41.708251 2541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-hmr98"] Nov 03 22:22:41.710289 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:41.710268 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hmr98" Nov 03 22:22:41.718417 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:41.718394 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5b40d10d-8dd1-46f6-b897-82780c392c84-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hmr98\" (UID: \"5b40d10d-8dd1-46f6-b897-82780c392c84\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hmr98" Nov 03 22:22:41.718498 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:41.718423 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lrhw\" (UniqueName: \"kubernetes.io/projected/5b40d10d-8dd1-46f6-b897-82780c392c84-kube-api-access-6lrhw\") pod \"monitoring-plugin-77f4f76594-hmr98\" (UID: \"5b40d10d-8dd1-46f6-b897-82780c392c84\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hmr98" Nov 03 22:22:41.762832 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:41.762811 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-5ghk8\"" Nov 03 22:22:41.762832 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:41.762827 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 03 22:22:41.770043 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:41.770020 2541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-hmr98"] Nov 03 22:22:41.787260 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:41.787233 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:41.819612 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:41.819581 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5b40d10d-8dd1-46f6-b897-82780c392c84-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hmr98\" (UID: \"5b40d10d-8dd1-46f6-b897-82780c392c84\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hmr98" Nov 03 22:22:41.819700 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:41.819624 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lrhw\" (UniqueName: \"kubernetes.io/projected/5b40d10d-8dd1-46f6-b897-82780c392c84-kube-api-access-6lrhw\") pod \"monitoring-plugin-77f4f76594-hmr98\" (UID: \"5b40d10d-8dd1-46f6-b897-82780c392c84\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hmr98" Nov 03 22:22:41.822194 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:41.822174 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5b40d10d-8dd1-46f6-b897-82780c392c84-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hmr98\" (UID: \"5b40d10d-8dd1-46f6-b897-82780c392c84\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hmr98" Nov 03 22:22:41.886516 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:41.886488 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lrhw\" (UniqueName: \"kubernetes.io/projected/5b40d10d-8dd1-46f6-b897-82780c392c84-kube-api-access-6lrhw\") pod \"monitoring-plugin-77f4f76594-hmr98\" (UID: \"5b40d10d-8dd1-46f6-b897-82780c392c84\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hmr98" Nov 03 22:22:42.015559 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:42.015482 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:42.020619 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:42.020600 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hmr98" Nov 03 22:22:42.187305 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:42.187271 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:42.333965 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:42.333881 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:42.389179 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:42.389152 2541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-hmr98"] Nov 03 22:22:42.392136 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:22:42.392112 2541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b40d10d_8dd1_46f6_b897_82780c392c84.slice/crio-3ccc68b5b925803a4907d8bb570385c4da297fe156d3cccbc13f5b5393bfe0cb WatchSource:0}: Error finding container 3ccc68b5b925803a4907d8bb570385c4da297fe156d3cccbc13f5b5393bfe0cb: Status 404 returned error can't find the container with id 3ccc68b5b925803a4907d8bb570385c4da297fe156d3cccbc13f5b5393bfe0cb Nov 03 22:22:42.573031 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:42.573002 2541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-lqwmt"] Nov 03 22:22:42.575256 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:42.575242 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-lqwmt" Nov 03 22:22:42.597738 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:42.597692 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:42.626897 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:42.626851 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8f1371d-471f-496f-ac95-19eeb163c2c7-tmp\") pod \"downloads-6dd789c8b7-lqwmt\" (UID: \"d8f1371d-471f-496f-ac95-19eeb163c2c7\") " pod="openshift-console/downloads-6dd789c8b7-lqwmt" Nov 03 22:22:42.626990 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:42.626908 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbjsn\" (UniqueName: \"kubernetes.io/projected/d8f1371d-471f-496f-ac95-19eeb163c2c7-kube-api-access-mbjsn\") pod \"downloads-6dd789c8b7-lqwmt\" (UID: \"d8f1371d-471f-496f-ac95-19eeb163c2c7\") " pod="openshift-console/downloads-6dd789c8b7-lqwmt" Nov 03 22:22:42.631248 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:42.631232 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 22:22:42.631395 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:42.631380 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-2kpmz\"" Nov 03 22:22:42.631443 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:42.631395 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 22:22:42.680091 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:42.680069 2541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-lqwmt"] Nov 03 22:22:42.727245 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:42.727223 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8f1371d-471f-496f-ac95-19eeb163c2c7-tmp\") pod \"downloads-6dd789c8b7-lqwmt\" (UID: \"d8f1371d-471f-496f-ac95-19eeb163c2c7\") " pod="openshift-console/downloads-6dd789c8b7-lqwmt" Nov 03 22:22:42.727373 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:42.727257 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbjsn\" (UniqueName: \"kubernetes.io/projected/d8f1371d-471f-496f-ac95-19eeb163c2c7-kube-api-access-mbjsn\") pod \"downloads-6dd789c8b7-lqwmt\" (UID: \"d8f1371d-471f-496f-ac95-19eeb163c2c7\") " pod="openshift-console/downloads-6dd789c8b7-lqwmt" Nov 03 22:22:42.727614 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:42.727596 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8f1371d-471f-496f-ac95-19eeb163c2c7-tmp\") pod \"downloads-6dd789c8b7-lqwmt\" (UID: \"d8f1371d-471f-496f-ac95-19eeb163c2c7\") " pod="openshift-console/downloads-6dd789c8b7-lqwmt" Nov 03 22:22:42.743931 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:42.743909 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:42.790304 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:42.790279 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbjsn\" (UniqueName: \"kubernetes.io/projected/d8f1371d-471f-496f-ac95-19eeb163c2c7-kube-api-access-mbjsn\") pod \"downloads-6dd789c8b7-lqwmt\" (UID: \"d8f1371d-471f-496f-ac95-19eeb163c2c7\") " pod="openshift-console/downloads-6dd789c8b7-lqwmt" Nov 03 22:22:42.885065 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:42.885014 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-lqwmt" Nov 03 22:22:42.912556 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:42.912530 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:42.928483 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:42.928444 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b6154c5a-4925-4c74-8db4-0acdb525b743-original-pull-secret\") pod \"global-pull-secret-syncer-h8jxb\" (UID: \"b6154c5a-4925-4c74-8db4-0acdb525b743\") " pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:22:42.928608 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:42.928483 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c027d694-64cc-46a2-b4e7-49b117f696d3-metrics-certs\") pod \"network-metrics-daemon-vqppk\" (UID: \"c027d694-64cc-46a2-b4e7-49b117f696d3\") " pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:22:42.980390 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:42.980365 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 22:22:42.980523 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:42.980452 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 22:22:42.991748 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:42.991713 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c027d694-64cc-46a2-b4e7-49b117f696d3-metrics-certs\") pod \"network-metrics-daemon-vqppk\" (UID: \"c027d694-64cc-46a2-b4e7-49b117f696d3\") " pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:22:42.991873 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:42.991774 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b6154c5a-4925-4c74-8db4-0acdb525b743-original-pull-secret\") pod \"global-pull-secret-syncer-h8jxb\" (UID: \"b6154c5a-4925-4c74-8db4-0acdb525b743\") " pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:22:43.029099 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:43.029064 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5zc6\" (UniqueName: \"kubernetes.io/projected/29b82faa-1aa1-4760-a9f6-1cbeb9427094-kube-api-access-g5zc6\") pod \"network-check-target-jkjnj\" (UID: \"29b82faa-1aa1-4760-a9f6-1cbeb9427094\") " pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:22:43.080878 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:43.080828 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 22:22:43.141774 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:43.141713 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 22:22:43.141906 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:43.141789 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dnsfs\"" Nov 03 22:22:43.150445 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:43.150416 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:22:43.151919 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:43.151903 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ft6gs\"" Nov 03 22:22:43.153070 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:43.152935 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5zc6\" (UniqueName: \"kubernetes.io/projected/29b82faa-1aa1-4760-a9f6-1cbeb9427094-kube-api-access-g5zc6\") pod \"network-check-target-jkjnj\" (UID: \"29b82faa-1aa1-4760-a9f6-1cbeb9427094\") " pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:22:43.160770 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:43.160752 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:22:43.265507 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:43.265474 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hmr98" event={"ID":"5b40d10d-8dd1-46f6-b897-82780c392c84","Type":"ContainerStarted","Data":"3ccc68b5b925803a4907d8bb570385c4da297fe156d3cccbc13f5b5393bfe0cb"} Nov 03 22:22:43.304166 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:43.304136 2541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-lqwmt"] Nov 03 22:22:43.307279 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:22:43.307252 2541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8f1371d_471f_496f_ac95_19eeb163c2c7.slice/crio-cde676372f389b72eb7b34513e4f61f2ab6261fbf9e9aad50e5348722e7d51ba WatchSource:0}: Error finding container cde676372f389b72eb7b34513e4f61f2ab6261fbf9e9aad50e5348722e7d51ba: Status 404 returned error can't find the container with id cde676372f389b72eb7b34513e4f61f2ab6261fbf9e9aad50e5348722e7d51ba Nov 03 22:22:43.446670 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:43.446593 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-j25tw\"" Nov 03 22:22:43.455424 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:43.455391 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:22:43.522581 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:43.522555 2541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h8jxb"] Nov 03 22:22:43.526384 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:22:43.526357 2541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6154c5a_4925_4c74_8db4_0acdb525b743.slice/crio-31e115bc65dc1b6ad686c8a800c4e28b7daa4c4ce1e67ab05d1218ce18aef29b WatchSource:0}: Error finding container 31e115bc65dc1b6ad686c8a800c4e28b7daa4c4ce1e67ab05d1218ce18aef29b: Status 404 returned error can't find the container with id 31e115bc65dc1b6ad686c8a800c4e28b7daa4c4ce1e67ab05d1218ce18aef29b Nov 03 22:22:43.536631 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:43.536610 2541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vqppk"] Nov 03 22:22:43.540765 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:22:43.540742 2541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc027d694_64cc_46a2_b4e7_49b117f696d3.slice/crio-2e8ff8210cf78b240869c8813a6efa43f0cc8e97887c0161838e754a0aabb833 WatchSource:0}: Error finding container 2e8ff8210cf78b240869c8813a6efa43f0cc8e97887c0161838e754a0aabb833: Status 404 returned error can't find the container with id 2e8ff8210cf78b240869c8813a6efa43f0cc8e97887c0161838e754a0aabb833 Nov 03 22:22:43.831554 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:43.831486 2541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jkjnj"] Nov 03 22:22:43.835257 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:22:43.835197 2541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29b82faa_1aa1_4760_a9f6_1cbeb9427094.slice/crio-6fcda1ab02615ec2ec6ae197eda91031ad640509ee6d23b4425b6ade0757214a WatchSource:0}: Error finding container 6fcda1ab02615ec2ec6ae197eda91031ad640509ee6d23b4425b6ade0757214a: Status 404 returned error can't find the container with id 6fcda1ab02615ec2ec6ae197eda91031ad640509ee6d23b4425b6ade0757214a Nov 03 22:22:44.270063 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:44.270030 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-lqwmt" event={"ID":"d8f1371d-471f-496f-ac95-19eeb163c2c7","Type":"ContainerStarted","Data":"cde676372f389b72eb7b34513e4f61f2ab6261fbf9e9aad50e5348722e7d51ba"} Nov 03 22:22:44.271790 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:44.271756 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hmr98" event={"ID":"5b40d10d-8dd1-46f6-b897-82780c392c84","Type":"ContainerStarted","Data":"9fbe4b3a215bfcd743a2649f11436751739bc9c6f7c89c0930cd3534bd377122"} Nov 03 22:22:44.271915 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:44.271899 2541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hmr98" Nov 03 22:22:44.273348 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:44.273323 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jkjnj" event={"ID":"29b82faa-1aa1-4760-a9f6-1cbeb9427094","Type":"ContainerStarted","Data":"6fcda1ab02615ec2ec6ae197eda91031ad640509ee6d23b4425b6ade0757214a"} Nov 03 22:22:44.274743 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:44.274718 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h8jxb" event={"ID":"b6154c5a-4925-4c74-8db4-0acdb525b743","Type":"ContainerStarted","Data":"31e115bc65dc1b6ad686c8a800c4e28b7daa4c4ce1e67ab05d1218ce18aef29b"} Nov 03 22:22:44.276160 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:44.276109 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vqppk" event={"ID":"c027d694-64cc-46a2-b4e7-49b117f696d3","Type":"ContainerStarted","Data":"2e8ff8210cf78b240869c8813a6efa43f0cc8e97887c0161838e754a0aabb833"} Nov 03 22:22:44.278236 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:44.278214 2541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hmr98" Nov 03 22:22:44.387886 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:44.387827 2541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hmr98" podStartSLOduration=1.6177213529999999 podStartE2EDuration="3.387813467s" totalImagesPullingTime="1.770092114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:44.386665309 +0000 UTC m=+66.903861878" watchObservedRunningTime="2026-11-03 22:22:44.387813467 +0000 UTC m=+66.905010039" Nov 03 22:22:44.469918 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:44.469828 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:44.621496 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:44.621431 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:46.087631 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.087598 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:46.174018 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.173981 2541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 22:22:46.180062 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.178052 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.224766 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.224724 2541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 22:22:46.244554 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.244521 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 22:22:46.244764 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.244641 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 22:22:46.244764 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.244727 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-t9khm\"" Nov 03 22:22:46.244951 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.244816 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 22:22:46.244951 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.244898 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 22:22:46.245054 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.244963 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 22:22:46.250385 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.250356 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.250502 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.250399 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-config\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.250502 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.250427 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.250502 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.250473 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.250651 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.250501 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.250651 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.250524 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.250651 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.250560 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.250651 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.250595 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.250651 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.250625 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-config-out\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.250651 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.250649 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.250968 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.250674 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.250968 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.250701 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-web-config\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.250968 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.250760 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqg7s\" (UniqueName: \"kubernetes.io/projected/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-kube-api-access-gqg7s\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.250968 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.250794 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.250968 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.250819 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.250968 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.250845 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.250968 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.250890 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.250968 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.250937 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.250968 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.250960 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.251392 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.250996 2541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.273277 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.273216 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 22:22:46.273402 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.273387 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-26t3ivs4b9nmd\"" Nov 03 22:22:46.285531 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.285139 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 22:22:46.285531 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.285300 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 22:22:46.285531 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.285348 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 22:22:46.285531 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.285374 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 22:22:46.297018 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.296915 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vqppk" event={"ID":"c027d694-64cc-46a2-b4e7-49b117f696d3","Type":"ContainerStarted","Data":"a8a876a31a6d6d8d4d3f8ca9262380189d95d837af1dd8dff907fe4b8ac206aa"} Nov 03 22:22:46.297018 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.296974 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vqppk" event={"ID":"c027d694-64cc-46a2-b4e7-49b117f696d3","Type":"ContainerStarted","Data":"d66106c0cf224869dbb690deb272b178d9ceb0ad9f62bd3f6d8a0692974351c8"} Nov 03 22:22:46.351794 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.351758 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.351970 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.351797 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.351970 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.351824 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.351970 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.351848 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.351970 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.351932 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.351970 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.351953 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.352231 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.351980 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.352231 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.352009 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.352231 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.352033 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-config\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.352231 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.352063 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.352231 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.352088 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.352231 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.352115 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.352231 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.352137 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.352231 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.352169 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.352231 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.352204 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.352231 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.352231 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-config-out\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.352680 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.352254 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.352680 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.352281 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.352680 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.352303 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-web-config\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.352680 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.352320 2541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqg7s\" (UniqueName: \"kubernetes.io/projected/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-kube-api-access-gqg7s\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.353851 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.353817 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.356189 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.354788 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.356189 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.355362 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.356189 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.355385 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.356189 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.355901 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.361431 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.361407 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.361600 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.361576 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-web-config\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.363152 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.363131 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.372380 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.372355 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-config-out\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.372526 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.372508 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.372732 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.372696 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.372842 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.372802 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.372952 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.372932 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-config\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.382934 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.379182 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.385619 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.385597 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 22:22:46.385702 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.385597 2541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 22:22:46.386797 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.386778 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 22:22:46.387032 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.387013 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 22:22:46.387032 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.387023 2541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 22:22:46.394756 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.394738 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.398517 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.398497 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.399580 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.399556 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.402128 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.402056 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.403237 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.403213 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.415901 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.415872 2541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqg7s\" (UniqueName: \"kubernetes.io/projected/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-kube-api-access-gqg7s\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.418873 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.418839 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:46.500624 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.500594 2541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:46.508204 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.507399 2541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vqppk" podStartSLOduration=66.749103908 podStartE2EDuration="1m8.507383102s" totalImagesPullingTime="1.758279194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:46.505105536 +0000 UTC m=+69.022302133" watchObservedRunningTime="2026-11-03 22:22:46.507383102 +0000 UTC m=+69.024579675" Nov 03 22:22:46.580377 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.580302 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:46.891473 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:46.891404 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:47.023810 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:47.023525 2541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 22:22:47.027003 ip-10-0-143-132 kubenswrapper[2541]: W1103 22:22:47.026971 2541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83dfb48_acb6_4cbc_bfc0_61103dafa90f.slice/crio-2b4d8589545194c7a73b28d8b87d606ffcb4749772644f101dd0d5956cf6cb4f WatchSource:0}: Error finding container 2b4d8589545194c7a73b28d8b87d606ffcb4749772644f101dd0d5956cf6cb4f: Status 404 returned error can't find the container with id 2b4d8589545194c7a73b28d8b87d606ffcb4749772644f101dd0d5956cf6cb4f Nov 03 22:22:47.303073 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:47.302985 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a83dfb48-acb6-4cbc-bfc0-61103dafa90f","Type":"ContainerStarted","Data":"2b4d8589545194c7a73b28d8b87d606ffcb4749772644f101dd0d5956cf6cb4f"} Nov 03 22:22:48.874274 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:48.874226 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:49.024801 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:49.024765 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:49.171625 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:49.171585 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:49.314160 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:49.314114 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jkjnj" event={"ID":"29b82faa-1aa1-4760-a9f6-1cbeb9427094","Type":"ContainerStarted","Data":"665995f5927fbab59f9ac590d9ffdbacfed83afdf88bcb6cfcafb26f7e080ffe"} Nov 03 22:22:49.314343 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:49.314239 2541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:22:49.316191 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:49.316084 2541 generic.go:356] "Generic (PLEG): container finished" podID="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" containerID="1bbcf6f7eeeec5392123a3b9318980d2d5b05a92d70b2c8aebf5d0ad0780c635" exitCode=0 Nov 03 22:22:49.316191 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:49.316134 2541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a83dfb48-acb6-4cbc-bfc0-61103dafa90f","Type":"ContainerDied","Data":"1bbcf6f7eeeec5392123a3b9318980d2d5b05a92d70b2c8aebf5d0ad0780c635"} Nov 03 22:22:49.351209 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:49.351137 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:49.448361 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:49.448224 2541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jkjnj" podStartSLOduration=66.713116357 podStartE2EDuration="1m11.448208358s" totalImagesPullingTime="4.735092001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:49.445744789 +0000 UTC m=+71.962941360" watchObservedRunningTime="2026-11-03 22:22:49.448208358 +0000 UTC m=+71.965404929" Nov 03 22:22:50.503714 ip-10-0-143-132 kubenswrapper[2541]: I1103 22:22:50.503674 2541 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:50.616019 ip-10-0-143-132 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 22:22:50.633389 ip-10-0-143-132 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 22:22:50.633665 ip-10-0-143-132 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 22:22:50.633727 ip-10-0-143-132 systemd[1]: kubelet.service: Consumed 3.911s CPU time. Nov 03 22:22:50.644426 ip-10-0-143-132 systemd[1]: Starting Kubernetes Kubelet... Nov 03 22:22:50.720044 ip-10-0-143-132 kubenswrapper[5196]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:22:50.720044 ip-10-0-143-132 kubenswrapper[5196]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 22:22:50.720044 ip-10-0-143-132 kubenswrapper[5196]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:22:50.720044 ip-10-0-143-132 kubenswrapper[5196]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:22:50.722265 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722169 5196 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:22:50.722265 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722264 5196 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:22:50.722265 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722268 5196 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:22:50.722377 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722272 5196 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:22:50.722377 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722275 5196 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:22:50.722377 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722278 5196 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:22:50.722377 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722281 5196 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:22:50.722377 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722283 5196 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:22:50.722377 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722286 5196 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:22:50.722377 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722288 5196 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:22:50.722377 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722291 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:22:50.722377 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722294 5196 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:22:50.722377 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722296 5196 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:22:50.722377 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722298 5196 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:22:50.722377 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722301 5196 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:22:50.722377 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722303 5196 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:22:50.722377 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722306 5196 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:22:50.722377 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722308 5196 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:22:50.722377 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722311 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:22:50.722377 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722313 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:22:50.722377 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722317 5196 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:22:50.722377 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722322 5196 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:22:50.722871 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722325 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:22:50.722871 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722328 5196 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:22:50.722871 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722330 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:22:50.722871 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722333 5196 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:22:50.722871 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722335 5196 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:22:50.722871 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722338 5196 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:22:50.722871 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722341 5196 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:22:50.722871 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722343 5196 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:22:50.722871 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722346 5196 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:22:50.722871 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722349 5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:22:50.722871 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722352 5196 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:22:50.722871 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722354 5196 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:22:50.722871 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722357 5196 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:22:50.722871 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722360 5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:22:50.722871 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722363 5196 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:22:50.722871 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722366 5196 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:22:50.722871 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722369 5196 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:22:50.722871 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722371 5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:22:50.722871 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722374 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:22:50.722871 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722377 5196 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:22:50.723356 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722380 5196 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:22:50.723356 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722383 5196 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:22:50.723356 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722386 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:22:50.723356 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722388 5196 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:22:50.723356 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722391 5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:22:50.723356 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722394 5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:22:50.723356 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722396 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:22:50.723356 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722399 5196 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:22:50.723356 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722402 5196 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:22:50.723356 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722404 5196 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:22:50.723356 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722407 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:22:50.723356 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722411 5196 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:22:50.723356 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722413 5196 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:22:50.723356 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722416 5196 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:22:50.723356 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722419 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:22:50.723356 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722421 5196 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:22:50.723356 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722424 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:22:50.723356 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722427 5196 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:22:50.723356 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722429 5196 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:22:50.723810 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722432 5196 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:22:50.723810 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722435 5196 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:22:50.723810 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722438 5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:22:50.723810 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722440 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:22:50.723810 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722443 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:22:50.723810 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722446 5196 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:22:50.723810 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722448 5196 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:22:50.723810 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722451 5196 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:22:50.723810 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722455 5196 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:22:50.723810 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722458 5196 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:22:50.723810 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722461 5196 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:22:50.723810 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722465 5196 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:22:50.723810 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722468 5196 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:22:50.723810 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722471 5196 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:22:50.723810 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722474 5196 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:22:50.723810 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722476 5196 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:22:50.723810 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722479 5196 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:22:50.723810 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722481 5196 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:22:50.723810 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722484 5196 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:22:50.723810 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722486 5196 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:22:50.724326 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722489 5196 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:22:50.724326 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722491 5196 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:22:50.724326 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722494 5196 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:22:50.724326 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722496 5196 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:22:50.724326 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722499 5196 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:22:50.724326 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722502 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:22:50.724326 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722504 5196 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:22:50.724326 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722507 5196 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:22:50.724326 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722510 5196 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:22:50.724326 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722512 5196 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:22:50.724326 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722515 5196 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:22:50.724326 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722517 5196 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:22:50.724326 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722520 5196 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:22:50.724326 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722522 5196 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:22:50.724326 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722525 5196 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:22:50.724326 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722527 5196 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:22:50.724326 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722531 5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:22:50.724326 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722534 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:22:50.724326 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722536 5196 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:22:50.724326 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722538 5196 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:22:50.724847 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722541 5196 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:22:50.724847 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722543 5196 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:22:50.724847 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722546 5196 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:22:50.724847 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722548 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:22:50.724847 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.722551 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:22:50.724847 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723333 5196 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:22:50.724847 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723342 5196 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:22:50.724847 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723345 5196 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:22:50.724847 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723348 5196 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:22:50.724847 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723351 5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:22:50.724847 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723354 5196 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:22:50.724847 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723356 5196 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:22:50.724847 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723359 5196 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:22:50.724847 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723362 5196 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:22:50.724847 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723365 5196 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:22:50.724847 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723368 5196 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:22:50.724847 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723371 5196 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:22:50.724847 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723374 5196 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:22:50.724847 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723377 5196 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:22:50.725339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723380 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:22:50.725339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723382 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:22:50.725339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723385 5196 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:22:50.725339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723387 5196 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:22:50.725339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723390 5196 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:22:50.725339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723393 5196 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:22:50.725339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723395 5196 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:22:50.725339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723398 5196 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:22:50.725339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723400 5196 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:22:50.725339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723403 5196 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:22:50.725339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723405 5196 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:22:50.725339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723408 5196 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:22:50.725339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723410 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:22:50.725339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723414 5196 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:22:50.725339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723418 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:22:50.725339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723421 5196 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:22:50.725339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723424 5196 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:22:50.725339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723427 5196 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:22:50.725339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723430 5196 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:22:50.725790 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723433 5196 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:22:50.725790 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723435 5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:22:50.725790 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723438 5196 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:22:50.725790 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723440 5196 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:22:50.725790 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723443 5196 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:22:50.725790 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723445 5196 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:22:50.725790 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723448 5196 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:22:50.725790 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723453 5196 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:22:50.725790 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723455 5196 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:22:50.725790 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723458 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:22:50.725790 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723461 5196 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:22:50.725790 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723463 5196 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:22:50.725790 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723466 5196 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:22:50.725790 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723469 5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:22:50.725790 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723472 5196 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:22:50.725790 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723475 5196 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:22:50.725790 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723478 5196 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:22:50.725790 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723480 5196 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:22:50.725790 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723483 5196 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:22:50.725790 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723485 5196 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:22:50.726289 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723488 5196 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:22:50.726289 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723491 5196 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:22:50.726289 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723493 5196 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:22:50.726289 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723496 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:22:50.726289 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723498 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:22:50.726289 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723501 5196 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:22:50.726289 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723503 5196 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:22:50.726289 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723506 5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:22:50.726289 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723509 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:22:50.726289 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723511 5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:22:50.726289 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723513 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:22:50.726289 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723516 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:22:50.726289 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723519 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:22:50.726289 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723522 5196 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:22:50.726289 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723526 5196 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:22:50.726289 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723528 5196 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:22:50.726289 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723531 5196 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:22:50.726289 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723533 5196 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:22:50.726289 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723536 5196 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:22:50.726763 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723540 5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:22:50.726763 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723543 5196 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:22:50.726763 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723545 5196 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:22:50.726763 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723563 5196 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:22:50.726763 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723567 5196 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:22:50.726763 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723570 5196 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:22:50.726763 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723573 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:22:50.726763 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723577 5196 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:22:50.726763 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723580 5196 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:22:50.726763 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723583 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:22:50.726763 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723586 5196 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:22:50.726763 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723588 5196 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:22:50.726763 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723591 5196 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:22:50.726763 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723594 5196 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:22:50.726763 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723597 5196 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:22:50.726763 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723600 5196 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:22:50.726763 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723602 5196 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:22:50.726763 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723604 5196 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:22:50.726763 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723607 5196 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:22:50.726763 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723609 5196 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:22:50.727274 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723612 5196 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:22:50.727274 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723615 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:22:50.727274 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723617 5196 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:22:50.727274 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723620 5196 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:22:50.727274 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723622 5196 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:22:50.727274 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723625 5196 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:22:50.727274 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723634 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:22:50.727274 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723636 5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:22:50.727274 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723639 5196 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:22:50.727274 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723641 5196 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:22:50.727274 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723644 5196 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:22:50.727274 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723647 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:22:50.727274 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723651 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:22:50.727274 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.723654 5196 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:22:50.727274 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724150 5196 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:22:50.727274 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724156 5196 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:22:50.727274 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724159 5196 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:22:50.727274 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724162 5196 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:22:50.727274 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724164 5196 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:22:50.727722 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724167 5196 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:22:50.727722 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724170 5196 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:22:50.727722 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724173 5196 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:22:50.727722 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724175 5196 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:22:50.727722 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724178 5196 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:22:50.727722 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724181 5196 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:22:50.727722 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724183 5196 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:22:50.727722 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724186 5196 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:22:50.727722 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724188 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:22:50.727722 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724191 5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:22:50.727722 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724193 5196 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:22:50.727722 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724196 5196 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:22:50.727722 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724198 5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:22:50.727722 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724201 5196 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:22:50.727722 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724204 5196 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:22:50.727722 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724206 5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:22:50.727722 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724209 5196 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:22:50.727722 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724211 5196 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:22:50.727722 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724214 5196 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:22:50.727722 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724217 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:22:50.728245 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724220 5196 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:22:50.728245 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724223 5196 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:22:50.728245 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724226 5196 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:22:50.728245 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724231 5196 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:22:50.728245 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724235 5196 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:22:50.728245 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724240 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:22:50.728245 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724243 5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:22:50.728245 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724245 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:22:50.728245 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724248 5196 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:22:50.728245 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724250 5196 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:22:50.728245 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724253 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:22:50.728245 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724255 5196 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:22:50.728245 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724258 5196 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:22:50.728245 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724261 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:22:50.728245 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724263 5196 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:22:50.728245 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724266 5196 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:22:50.728245 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724268 5196 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:22:50.728245 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724271 5196 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:22:50.728787 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724273 5196 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:22:50.728787 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724276 5196 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:22:50.728787 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724278 5196 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:22:50.728787 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724281 5196 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:22:50.728787 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724283 5196 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:22:50.728787 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724286 5196 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:22:50.728787 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724288 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:22:50.728787 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724291 5196 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:22:50.728787 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724293 5196 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:22:50.728787 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724296 5196 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:22:50.728787 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724298 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:22:50.728787 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724301 5196 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:22:50.728787 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724304 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:22:50.728787 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724306 5196 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:22:50.728787 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724309 5196 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:22:50.728787 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724312 5196 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:22:50.728787 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724314 5196 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:22:50.728787 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724317 5196 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:22:50.728787 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724320 5196 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:22:50.728787 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724324 5196 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:22:50.729319 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724326 5196 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:22:50.729319 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724329 5196 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:22:50.729319 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724331 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:22:50.729319 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724334 5196 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:22:50.729319 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724337 5196 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:22:50.729319 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724339 5196 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:22:50.729319 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724342 5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:22:50.729319 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724344 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:22:50.729319 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724347 5196 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:22:50.729319 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724349 5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:22:50.729319 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724352 5196 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:22:50.729319 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724354 5196 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:22:50.729319 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724357 5196 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:22:50.729319 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724360 5196 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:22:50.729319 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724362 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:22:50.729319 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724365 5196 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:22:50.729319 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724367 5196 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:22:50.729319 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724370 5196 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:22:50.729319 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724373 5196 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:22:50.729319 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724375 5196 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:22:50.729798 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724377 5196 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:22:50.729798 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724380 5196 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:22:50.729798 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724382 5196 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:22:50.729798 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724385 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:22:50.729798 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724387 5196 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:22:50.729798 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724390 5196 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:22:50.729798 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724392 5196 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:22:50.729798 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724395 5196 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:22:50.729798 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724397 5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:22:50.729798 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724400 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:22:50.729798 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724404 5196 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:22:50.729798 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724407 5196 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:22:50.729798 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724409 5196 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:22:50.729798 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724412 5196 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:22:50.729798 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724414 5196 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:22:50.729798 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724417 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:22:50.729798 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724419 5196 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:22:50.729798 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724422 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:22:50.729798 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724425 5196 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:22:50.729798 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724428 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:22:50.730301 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724430 5196 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:22:50.730301 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724432 5196 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:22:50.730301 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.724435 5196 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:22:50.730301 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724525 5196 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 22:22:50.730301 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724536 5196 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 22:22:50.730301 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724541 5196 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 22:22:50.730301 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724546 5196 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 22:22:50.730301 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724551 5196 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 22:22:50.730301 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724554 5196 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 22:22:50.730301 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724558 5196 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 22:22:50.730301 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724562 5196 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 22:22:50.730301 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724565 5196 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 22:22:50.730301 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724568 5196 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 22:22:50.730301 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724571 5196 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 22:22:50.730301 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724574 5196 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 22:22:50.730301 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724577 5196 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 22:22:50.730301 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724580 5196 flags.go:64] FLAG: --cgroup-root="" Nov 03 22:22:50.730301 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724582 5196 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 22:22:50.730301 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724585 5196 flags.go:64] FLAG: --client-ca-file="" Nov 03 22:22:50.730301 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724588 5196 flags.go:64] FLAG: --cloud-provider="external" Nov 03 22:22:50.730301 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724597 5196 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 22:22:50.730301 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724603 5196 flags.go:64] FLAG: --cluster-domain="" Nov 03 22:22:50.730301 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724608 5196 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 22:22:50.730301 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724611 5196 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 22:22:50.730936 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724614 5196 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 22:22:50.730936 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724617 5196 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 22:22:50.730936 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724621 5196 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 22:22:50.730936 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724624 5196 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 22:22:50.730936 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724627 5196 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 22:22:50.730936 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724631 5196 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 22:22:50.730936 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724633 5196 flags.go:64] FLAG: --contention-profiling="false" Nov 03 22:22:50.730936 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724636 5196 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 22:22:50.730936 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724639 5196 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 22:22:50.730936 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724642 5196 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 22:22:50.730936 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724645 5196 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 22:22:50.730936 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724650 5196 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 22:22:50.730936 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724652 5196 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 22:22:50.730936 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724655 5196 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 22:22:50.730936 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724658 5196 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 22:22:50.730936 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724661 5196 flags.go:64] FLAG: --enable-server="true" Nov 03 22:22:50.730936 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724664 5196 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 22:22:50.730936 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724670 5196 flags.go:64] FLAG: --event-burst="100" Nov 03 22:22:50.730936 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724674 5196 flags.go:64] FLAG: --event-qps="50" Nov 03 22:22:50.730936 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724676 5196 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 22:22:50.730936 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724679 5196 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 22:22:50.730936 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724682 5196 flags.go:64] FLAG: --eviction-hard="" Nov 03 22:22:50.730936 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724686 5196 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 22:22:50.730936 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724689 5196 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 22:22:50.731785 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724692 5196 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 22:22:50.731785 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724695 5196 flags.go:64] FLAG: --eviction-soft="" Nov 03 22:22:50.731785 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724698 5196 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 22:22:50.731785 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724701 5196 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 22:22:50.731785 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724704 5196 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 22:22:50.731785 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724708 5196 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 22:22:50.731785 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724712 5196 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 22:22:50.731785 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724716 5196 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 22:22:50.731785 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724718 5196 flags.go:64] FLAG: --feature-gates="" Nov 03 22:22:50.731785 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724722 5196 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 22:22:50.731785 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724725 5196 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 22:22:50.731785 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724728 5196 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 22:22:50.731785 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724731 5196 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 22:22:50.731785 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724734 5196 flags.go:64] FLAG: --healthz-port="10248" Nov 03 22:22:50.731785 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724737 5196 flags.go:64] FLAG: --help="false" Nov 03 22:22:50.731785 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724740 5196 flags.go:64] FLAG: --hostname-override="ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:22:50.731785 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724743 5196 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 22:22:50.731785 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724746 5196 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 22:22:50.731785 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724749 5196 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 22:22:50.731785 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724752 5196 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 22:22:50.731785 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724755 5196 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 22:22:50.731785 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724758 5196 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 22:22:50.731785 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724761 5196 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 22:22:50.732749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724764 5196 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 22:22:50.732749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724766 5196 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 22:22:50.732749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724770 5196 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 22:22:50.732749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724773 5196 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 22:22:50.732749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724776 5196 flags.go:64] FLAG: --kube-reserved="" Nov 03 22:22:50.732749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724779 5196 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 22:22:50.732749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724782 5196 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 22:22:50.732749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724785 5196 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 22:22:50.732749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724788 5196 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 22:22:50.732749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724790 5196 flags.go:64] FLAG: --lock-file="" Nov 03 22:22:50.732749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724793 5196 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 22:22:50.732749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724796 5196 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 22:22:50.732749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724799 5196 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 22:22:50.732749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724804 5196 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 22:22:50.732749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724807 5196 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 22:22:50.732749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724812 5196 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 22:22:50.732749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724814 5196 flags.go:64] FLAG: --logging-format="text" Nov 03 22:22:50.732749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724817 5196 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 22:22:50.732749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724821 5196 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 22:22:50.732749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724823 5196 flags.go:64] FLAG: --manifest-url="" Nov 03 22:22:50.732749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724826 5196 flags.go:64] FLAG: --manifest-url-header="" Nov 03 22:22:50.732749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724831 5196 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 22:22:50.732749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724834 5196 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 22:22:50.732749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724837 5196 flags.go:64] FLAG: --max-pods="110" Nov 03 22:22:50.732749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724840 5196 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 22:22:50.733382 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724843 5196 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 22:22:50.733382 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724846 5196 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 22:22:50.733382 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724850 5196 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 22:22:50.733382 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724870 5196 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 22:22:50.733382 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724874 5196 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 22:22:50.733382 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724877 5196 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 22:22:50.733382 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724884 5196 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 22:22:50.733382 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724887 5196 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 22:22:50.733382 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724890 5196 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 22:22:50.733382 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724893 5196 flags.go:64] FLAG: --pod-cidr="" Nov 03 22:22:50.733382 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724896 5196 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 22:22:50.733382 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724898 5196 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 22:22:50.733382 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724901 5196 flags.go:64] FLAG: --pods-per-core="0" Nov 03 22:22:50.733382 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724904 5196 flags.go:64] FLAG: --port="10250" Nov 03 22:22:50.733382 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724907 5196 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 22:22:50.733382 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724910 5196 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0a97294caf4c6a653" Nov 03 22:22:50.733382 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724913 5196 flags.go:64] FLAG: --qos-reserved="" Nov 03 22:22:50.733382 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724916 5196 flags.go:64] FLAG: --read-only-port="10255" Nov 03 22:22:50.733382 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724920 5196 flags.go:64] FLAG: --register-node="true" Nov 03 22:22:50.733382 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724923 5196 flags.go:64] FLAG: --register-with-taints="" Nov 03 22:22:50.733382 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724927 5196 flags.go:64] FLAG: --registry-burst="10" Nov 03 22:22:50.733382 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724929 5196 flags.go:64] FLAG: --registry-qps="5" Nov 03 22:22:50.733382 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724933 5196 flags.go:64] FLAG: --reserved-cpus="" Nov 03 22:22:50.733382 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724937 5196 flags.go:64] FLAG: --reserved-memory="" Nov 03 22:22:50.733382 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724940 5196 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 22:22:50.733382 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724943 5196 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 22:22:50.734011 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724946 5196 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 22:22:50.734011 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724950 5196 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 22:22:50.734011 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724952 5196 flags.go:64] FLAG: --runonce="false" Nov 03 22:22:50.734011 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724955 5196 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 22:22:50.734011 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724958 5196 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 22:22:50.734011 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724962 5196 flags.go:64] FLAG: --seccomp-default="false" Nov 03 22:22:50.734011 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724964 5196 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 22:22:50.734011 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724967 5196 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 22:22:50.734011 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724971 5196 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 22:22:50.734011 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724974 5196 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 22:22:50.734011 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724977 5196 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 22:22:50.734011 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724980 5196 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 22:22:50.734011 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724983 5196 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 22:22:50.734011 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724986 5196 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 22:22:50.734011 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724988 5196 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 22:22:50.734011 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724991 5196 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 22:22:50.734011 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724994 5196 flags.go:64] FLAG: --system-cgroups="" Nov 03 22:22:50.734011 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.724997 5196 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 22:22:50.734011 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.725002 5196 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 22:22:50.734011 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.725005 5196 flags.go:64] FLAG: --tls-cert-file="" Nov 03 22:22:50.734011 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.725008 5196 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 22:22:50.734011 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.725012 5196 flags.go:64] FLAG: --tls-min-version="" Nov 03 22:22:50.734011 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.725015 5196 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 22:22:50.734011 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.725017 5196 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 22:22:50.734011 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.725020 5196 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 22:22:50.734589 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.725023 5196 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 22:22:50.734589 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.725026 5196 flags.go:64] FLAG: --v="2" Nov 03 22:22:50.734589 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.725030 5196 flags.go:64] FLAG: --version="false" Nov 03 22:22:50.734589 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.725035 5196 flags.go:64] FLAG: --vmodule="" Nov 03 22:22:50.734589 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.725039 5196 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 22:22:50.734589 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.725043 5196 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 22:22:50.734589 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725175 5196 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:22:50.734589 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725179 5196 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:22:50.734589 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725182 5196 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:22:50.734589 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725185 5196 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:22:50.734589 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725188 5196 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:22:50.734589 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725191 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:22:50.734589 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725193 5196 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:22:50.734589 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725196 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:22:50.734589 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725198 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:22:50.734589 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725202 5196 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:22:50.734589 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725205 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:22:50.734589 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725207 5196 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:22:50.734589 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725210 5196 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:22:50.734589 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725213 5196 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:22:50.734589 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725215 5196 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:22:50.735114 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725218 5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:22:50.735114 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725220 5196 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:22:50.735114 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725223 5196 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:22:50.735114 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725225 5196 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:22:50.735114 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725228 5196 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:22:50.735114 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725230 5196 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:22:50.735114 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725233 5196 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:22:50.735114 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725235 5196 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:22:50.735114 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725238 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:22:50.735114 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725240 5196 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:22:50.735114 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725244 5196 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:22:50.735114 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725248 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:22:50.735114 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725251 5196 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:22:50.735114 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725254 5196 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:22:50.735114 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725258 5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:22:50.735114 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725261 5196 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:22:50.735114 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725264 5196 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:22:50.735114 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725267 5196 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:22:50.735114 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725269 5196 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:22:50.735681 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725272 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:22:50.735681 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725274 5196 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:22:50.735681 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725277 5196 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:22:50.735681 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725279 5196 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:22:50.735681 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725282 5196 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:22:50.735681 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725285 5196 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:22:50.735681 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725287 5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:22:50.735681 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725291 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:22:50.735681 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725294 5196 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:22:50.735681 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725296 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:22:50.735681 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725299 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:22:50.735681 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725301 5196 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:22:50.735681 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725304 5196 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:22:50.735681 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725307 5196 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:22:50.735681 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725309 5196 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:22:50.735681 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725312 5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:22:50.735681 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725315 5196 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:22:50.735681 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725317 5196 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:22:50.735681 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725320 5196 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:22:50.736208 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725322 5196 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:22:50.736208 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725325 5196 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:22:50.736208 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725327 5196 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:22:50.736208 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725331 5196 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:22:50.736208 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725335 5196 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:22:50.736208 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725338 5196 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:22:50.736208 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725341 5196 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:22:50.736208 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725343 5196 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:22:50.736208 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725348 5196 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:22:50.736208 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725350 5196 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:22:50.736208 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725353 5196 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:22:50.736208 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725356 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:22:50.736208 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725359 5196 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:22:50.736208 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725361 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:22:50.736208 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725364 5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:22:50.736208 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725366 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:22:50.736208 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725369 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:22:50.736208 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725371 5196 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:22:50.736208 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725374 5196 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:22:50.736208 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725376 5196 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:22:50.736716 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725380 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:22:50.736716 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725383 5196 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:22:50.736716 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725386 5196 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:22:50.736716 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725388 5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:22:50.736716 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725391 5196 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:22:50.736716 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725393 5196 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:22:50.736716 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725396 5196 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:22:50.736716 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725398 5196 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:22:50.736716 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725401 5196 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:22:50.736716 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725403 5196 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:22:50.736716 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725405 5196 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:22:50.736716 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725408 5196 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:22:50.736716 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725410 5196 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:22:50.736716 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725413 5196 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:22:50.736716 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725415 5196 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:22:50.736716 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725418 5196 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:22:50.736716 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725420 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:22:50.736716 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725423 5196 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:22:50.736716 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725425 5196 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:22:50.736716 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725428 5196 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:22:50.737310 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725432 5196 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:22:50.737310 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725434 5196 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:22:50.737310 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725437 5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:22:50.737310 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725439 5196 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:22:50.737310 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725442 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:22:50.737310 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725444 5196 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:22:50.737310 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725447 5196 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:22:50.737310 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725450 5196 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:22:50.737310 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725452 5196 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:22:50.737310 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725455 5196 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:22:50.737310 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725457 5196 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:22:50.737310 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725460 5196 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:22:50.737310 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725463 5196 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:22:50.737310 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.725489 5196 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:22:50.737310 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725522 5196 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:22:50.737310 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725526 5196 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:22:50.737963 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725530 5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:22:50.737963 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725533 5196 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:22:50.737963 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725536 5196 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:22:50.737963 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725538 5196 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:22:50.737963 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725541 5196 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:22:50.737963 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725544 5196 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:22:50.737963 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725546 5196 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:22:50.737963 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725549 5196 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:22:50.737963 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725551 5196 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:22:50.737963 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725554 5196 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:22:50.737963 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725557 5196 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:22:50.737963 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725559 5196 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:22:50.737963 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725562 5196 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:22:50.737963 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725564 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:22:50.737963 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725567 5196 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:22:50.737963 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725569 5196 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:22:50.737963 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725573 5196 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:22:50.737963 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725576 5196 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:22:50.737963 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725578 5196 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:22:50.737963 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725580 5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:22:50.738774 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725583 5196 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:22:50.738774 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725586 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:22:50.738774 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725589 5196 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:22:50.738774 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725592 5196 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:22:50.738774 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725595 5196 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:22:50.738774 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725597 5196 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:22:50.738774 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725599 5196 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:22:50.738774 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725602 5196 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:22:50.738774 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725605 5196 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:22:50.738774 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725608 5196 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:22:50.738774 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725611 5196 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:22:50.738774 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725613 5196 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:22:50.738774 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725616 5196 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:22:50.738774 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725618 5196 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:22:50.738774 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725621 5196 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:22:50.738774 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725623 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:22:50.738774 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725626 5196 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:22:50.738774 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725628 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:22:50.738774 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725631 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:22:50.739517 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725634 5196 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:22:50.739517 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725636 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:22:50.739517 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725638 5196 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:22:50.739517 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725641 5196 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:22:50.739517 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725644 5196 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:22:50.739517 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725647 5196 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:22:50.739517 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725649 5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:22:50.739517 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725652 5196 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:22:50.739517 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725654 5196 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:22:50.739517 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725658 5196 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:22:50.739517 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725660 5196 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:22:50.739517 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725663 5196 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:22:50.739517 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725665 5196 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:22:50.739517 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725667 5196 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:22:50.739517 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725670 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:22:50.739517 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725672 5196 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:22:50.739517 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725676 5196 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:22:50.739517 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725680 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:22:50.739517 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725683 5196 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:22:50.740163 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725686 5196 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:22:50.740163 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725688 5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:22:50.740163 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725691 5196 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:22:50.740163 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725695 5196 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:22:50.740163 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725698 5196 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:22:50.740163 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725700 5196 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:22:50.740163 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725703 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:22:50.740163 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725705 5196 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:22:50.740163 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725708 5196 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:22:50.740163 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725711 5196 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:22:50.740163 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725713 5196 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:22:50.740163 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725716 5196 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:22:50.740163 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725719 5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:22:50.740163 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725721 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:22:50.740163 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725724 5196 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:22:50.740163 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725726 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:22:50.740163 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725728 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:22:50.740163 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725731 5196 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:22:50.740163 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725734 5196 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:22:50.740759 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725736 5196 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:22:50.740759 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725738 5196 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:22:50.740759 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725741 5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:22:50.740759 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725745 5196 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:22:50.740759 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725748 5196 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:22:50.740759 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725750 5196 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:22:50.740759 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725752 5196 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:22:50.740759 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725755 5196 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:22:50.740759 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725757 5196 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:22:50.740759 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725761 5196 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:22:50.740759 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725764 5196 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:22:50.740759 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725767 5196 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:22:50.740759 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725770 5196 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:22:50.740759 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725773 5196 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:22:50.740759 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725775 5196 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:22:50.740759 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725777 5196 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:22:50.740759 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725781 5196 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:22:50.740759 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725784 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:22:50.740759 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725787 5196 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:22:50.741337 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725790 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:22:50.741337 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725793 5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:22:50.741337 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725796 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:22:50.741337 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725798 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:22:50.741337 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725802 5196 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:22:50.741337 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725804 5196 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:22:50.741337 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725807 5196 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:22:50.741337 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.725809 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:22:50.741337 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.729684 5196 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 22:22:50.741337 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.729698 5196 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 22:22:50.741337 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.729714 5196 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 22:22:50.741337 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.729718 5196 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 22:22:50.741337 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729799 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:22:50.741337 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729804 5196 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:22:50.741337 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729808 5196 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:22:50.741337 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729811 5196 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:22:50.741337 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729814 5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:22:50.741337 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729817 5196 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:22:50.741337 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729820 5196 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:22:50.741337 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729822 5196 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:22:50.741822 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729825 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:22:50.741822 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729827 5196 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:22:50.741822 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729830 5196 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:22:50.741822 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729833 5196 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:22:50.741822 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729841 5196 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:22:50.741822 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729844 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:22:50.741822 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729848 5196 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:22:50.741822 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729868 5196 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:22:50.741822 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729872 5196 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:22:50.741822 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729875 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:22:50.741822 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729878 5196 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:22:50.741822 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729880 5196 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:22:50.741822 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729883 5196 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:22:50.741822 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729886 5196 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:22:50.741822 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729889 5196 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:22:50.741822 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729891 5196 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:22:50.741822 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729894 5196 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:22:50.741822 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729896 5196 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:22:50.741822 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729899 5196 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:22:50.742577 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729902 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:22:50.742577 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729904 5196 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:22:50.742577 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729907 5196 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:22:50.742577 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729909 5196 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:22:50.742577 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729912 5196 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:22:50.742577 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729916 5196 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:22:50.742577 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729919 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:22:50.742577 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729922 5196 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:22:50.742577 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729925 5196 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:22:50.742577 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729927 5196 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:22:50.742577 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729930 5196 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:22:50.742577 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729932 5196 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:22:50.742577 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729935 5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:22:50.742577 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729937 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:22:50.742577 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729940 5196 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:22:50.742577 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729942 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:22:50.742577 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729945 5196 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:22:50.742577 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729947 5196 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:22:50.742577 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729956 5196 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:22:50.743083 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729959 5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:22:50.743083 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729961 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:22:50.743083 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729964 5196 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:22:50.743083 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729966 5196 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:22:50.743083 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729969 5196 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:22:50.743083 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729971 5196 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:22:50.743083 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729974 5196 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:22:50.743083 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729976 5196 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:22:50.743083 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729979 5196 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:22:50.743083 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729983 5196 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:22:50.743083 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729986 5196 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:22:50.743083 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729988 5196 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:22:50.743083 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729991 5196 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:22:50.743083 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729993 5196 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:22:50.743083 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729996 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:22:50.743083 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.729998 5196 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:22:50.743083 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730001 5196 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:22:50.743083 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730003 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:22:50.743083 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730006 5196 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:22:50.743083 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730008 5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:22:50.743547 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730011 5196 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:22:50.743547 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730013 5196 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:22:50.743547 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730016 5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:22:50.743547 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730019 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:22:50.743547 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730021 5196 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:22:50.743547 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730024 5196 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:22:50.743547 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730026 5196 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:22:50.743547 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730029 5196 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:22:50.743547 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730031 5196 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:22:50.743547 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730034 5196 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:22:50.743547 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730036 5196 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:22:50.743547 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730038 5196 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:22:50.743547 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730046 5196 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:22:50.743547 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730049 5196 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:22:50.743547 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730052 5196 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:22:50.743547 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730055 5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:22:50.743547 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730057 5196 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:22:50.743547 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730060 5196 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:22:50.743547 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730063 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:22:50.744013 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730065 5196 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:22:50.744013 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730068 5196 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:22:50.744013 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730070 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:22:50.744013 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730073 5196 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:22:50.744013 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730075 5196 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:22:50.744013 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730078 5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:22:50.744013 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730081 5196 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:22:50.744013 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730083 5196 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:22:50.744013 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730086 5196 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:22:50.744013 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730088 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:22:50.744013 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730091 5196 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:22:50.744013 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730093 5196 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:22:50.744013 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730096 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:22:50.744013 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730098 5196 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:22:50.744013 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730101 5196 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:22:50.744013 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730104 5196 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:22:50.744013 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730106 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:22:50.744013 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730108 5196 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:22:50.744013 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730111 5196 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:22:50.744460 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730113 5196 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:22:50.744460 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730116 5196 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:22:50.744460 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.730145 5196 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:22:50.744460 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730322 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:22:50.744460 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730328 5196 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:22:50.744460 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730330 5196 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:22:50.744460 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730334 5196 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:22:50.744460 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730344 5196 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:22:50.744460 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730348 5196 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:22:50.744460 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730351 5196 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:22:50.744460 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730354 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:22:50.744460 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730357 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:22:50.744460 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730359 5196 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:22:50.744460 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730362 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:22:50.744460 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730364 5196 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:22:50.744867 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730367 5196 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:22:50.744867 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730369 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:22:50.744867 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730372 5196 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:22:50.744867 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730374 5196 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:22:50.744867 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730377 5196 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:22:50.744867 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730379 5196 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:22:50.744867 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730382 5196 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:22:50.744867 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730384 5196 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:22:50.744867 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730387 5196 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:22:50.744867 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730389 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:22:50.744867 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730391 5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:22:50.744867 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730394 5196 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:22:50.744867 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730396 5196 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:22:50.744867 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730399 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:22:50.744867 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730401 5196 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:22:50.744867 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730404 5196 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:22:50.744867 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730406 5196 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:22:50.744867 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730409 5196 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:22:50.744867 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730411 5196 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:22:50.745325 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730414 5196 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:22:50.745325 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730416 5196 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:22:50.745325 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730419 5196 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:22:50.745325 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730421 5196 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:22:50.745325 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730424 5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:22:50.745325 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730426 5196 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:22:50.745325 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730436 5196 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:22:50.745325 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730439 5196 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:22:50.745325 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730441 5196 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:22:50.745325 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730444 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:22:50.745325 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730446 5196 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:22:50.745325 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730449 5196 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:22:50.745325 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730451 5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:22:50.745325 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730454 5196 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:22:50.745325 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730456 5196 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:22:50.745325 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730459 5196 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:22:50.745325 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730461 5196 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:22:50.745325 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730464 5196 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:22:50.745325 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730466 5196 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:22:50.745768 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730469 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:22:50.745768 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730471 5196 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:22:50.745768 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730474 5196 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:22:50.745768 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730476 5196 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:22:50.745768 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730485 5196 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:22:50.745768 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730488 5196 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:22:50.745768 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730490 5196 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:22:50.745768 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730493 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:22:50.745768 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730495 5196 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:22:50.745768 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730497 5196 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:22:50.745768 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730500 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:22:50.745768 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730502 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:22:50.745768 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730505 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:22:50.745768 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730507 5196 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:22:50.745768 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730510 5196 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:22:50.745768 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730513 5196 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:22:50.745768 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730515 5196 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:22:50.745768 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730517 5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:22:50.745768 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730520 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:22:50.745768 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730522 5196 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:22:50.746339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730530 5196 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:22:50.746339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730533 5196 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:22:50.746339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730535 5196 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:22:50.746339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730537 5196 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:22:50.746339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730540 5196 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:22:50.746339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730542 5196 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:22:50.746339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730545 5196 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:22:50.746339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730547 5196 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:22:50.746339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730550 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:22:50.746339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730552 5196 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:22:50.746339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730555 5196 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:22:50.746339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730557 5196 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:22:50.746339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730559 5196 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:22:50.746339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730562 5196 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:22:50.746339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730564 5196 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:22:50.746339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730567 5196 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:22:50.746339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730569 5196 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:22:50.746339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730572 5196 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:22:50.746339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730574 5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:22:50.746339 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730577 5196 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:22:50.746941 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730580 5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:22:50.746941 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730582 5196 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:22:50.746941 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730586 5196 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:22:50.746941 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730589 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:22:50.746941 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730592 5196 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:22:50.746941 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730595 5196 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:22:50.746941 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730597 5196 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:22:50.746941 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730600 5196 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:22:50.746941 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730602 5196 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:22:50.746941 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730604 5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:22:50.746941 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730607 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:22:50.746941 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730609 5196 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:22:50.746941 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730612 5196 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:22:50.746941 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730620 5196 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:22:50.746941 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730622 5196 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:22:50.746941 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730625 5196 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:22:50.747430 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.730647 5196 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:22:50.747430 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730694 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:22:50.747430 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730698 5196 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:22:50.747430 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730701 5196 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:22:50.747430 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730703 5196 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:22:50.747430 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730706 5196 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:22:50.747430 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730708 5196 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:22:50.747430 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730711 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:22:50.747430 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730713 5196 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:22:50.747430 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730716 5196 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:22:50.747430 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730720 5196 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:22:50.747430 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730724 5196 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:22:50.747430 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730726 5196 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:22:50.747430 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730730 5196 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:22:50.747430 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730733 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:22:50.747924 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730735 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:22:50.747924 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730738 5196 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:22:50.747924 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730740 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:22:50.747924 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730743 5196 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:22:50.747924 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730745 5196 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:22:50.747924 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730748 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:22:50.747924 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730750 5196 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:22:50.747924 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730753 5196 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:22:50.747924 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730755 5196 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:22:50.747924 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730758 5196 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:22:50.747924 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730760 5196 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:22:50.747924 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730763 5196 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:22:50.747924 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730765 5196 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:22:50.747924 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730768 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:22:50.747924 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730770 5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:22:50.747924 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730779 5196 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:22:50.747924 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730782 5196 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:22:50.747924 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730784 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:22:50.747924 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730787 5196 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:22:50.748450 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730789 5196 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:22:50.748450 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730792 5196 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:22:50.748450 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730794 5196 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:22:50.748450 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730796 5196 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:22:50.748450 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730799 5196 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:22:50.748450 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730801 5196 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:22:50.748450 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730804 5196 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:22:50.748450 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730806 5196 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:22:50.748450 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730808 5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:22:50.748450 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730811 5196 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:22:50.748450 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730813 5196 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:22:50.748450 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730816 5196 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:22:50.748450 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730818 5196 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:22:50.748450 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730823 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:22:50.748450 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730825 5196 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:22:50.748450 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730828 5196 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:22:50.748450 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730830 5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:22:50.748450 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730833 5196 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:22:50.748450 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730835 5196 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:22:50.749151 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730838 5196 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:22:50.749151 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730840 5196 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:22:50.749151 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730843 5196 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:22:50.749151 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730846 5196 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:22:50.749151 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730848 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:22:50.749151 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730850 5196 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:22:50.749151 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730877 5196 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:22:50.749151 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730881 5196 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:22:50.749151 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730883 5196 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:22:50.749151 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730886 5196 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:22:50.749151 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730901 5196 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:22:50.749151 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730905 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:22:50.749151 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730907 5196 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:22:50.749151 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730910 5196 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:22:50.749151 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730912 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:22:50.749151 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730915 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:22:50.749151 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730917 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:22:50.749151 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730920 5196 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:22:50.749151 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730922 5196 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:22:50.749151 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730925 5196 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:22:50.749801 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730927 5196 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:22:50.749801 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730930 5196 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:22:50.749801 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730932 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:22:50.749801 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730935 5196 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:22:50.749801 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730938 5196 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:22:50.749801 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730940 5196 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:22:50.749801 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730944 5196 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:22:50.749801 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730947 5196 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:22:50.749801 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730950 5196 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:22:50.749801 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730952 5196 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:22:50.749801 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730955 5196 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:22:50.749801 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730957 5196 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:22:50.749801 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730960 5196 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:22:50.749801 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730964 5196 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:22:50.749801 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730967 5196 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:22:50.749801 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730969 5196 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:22:50.749801 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730972 5196 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:22:50.749801 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730974 5196 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:22:50.749801 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730977 5196 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:22:50.749801 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730979 5196 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:22:50.750461 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730982 5196 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:22:50.750461 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730984 5196 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:22:50.750461 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730987 5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:22:50.750461 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730995 5196 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:22:50.750461 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.730998 5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:22:50.750461 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.731001 5196 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:22:50.750461 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.731004 5196 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:22:50.750461 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.731008 5196 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:22:50.750461 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.731011 5196 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:22:50.750461 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.731013 5196 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:22:50.750461 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.731016 5196 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:22:50.750461 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.731018 5196 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:22:50.750461 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.731021 5196 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:22:50.750461 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:22:50.731023 5196 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:22:50.750461 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.731074 5196 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 22:22:50.750461 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.733291 5196 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 22:22:50.750461 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.733348 5196 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 22:22:50.750461 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.733556 5196 server.go:1014] "Starting client certificate rotation" Nov 03 22:22:50.750461 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.733687 5196 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 22:22:50.751087 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.733745 5196 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 22:16:37 +0000 UTC" deadline="2028-04-06 22:46:11.653725505 +0000 UTC" Nov 03 22:22:50.751087 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.733782 5196 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12480h23m20.919945263s" Nov 03 22:22:50.751087 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.734041 5196 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:22:50.751087 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.735130 5196 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:22:50.751087 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.736367 5196 log.go:25] "Validated CRI v1 runtime API" Nov 03 22:22:50.751087 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.738306 5196 log.go:25] "Validated CRI v1 image API" Nov 03 22:22:50.751087 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.739167 5196 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 22:22:50.751087 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.741960 5196 fs.go:135] Filesystem UUIDs: map[26b5d738-b058-4f16-92ca-cc05b3dcad21:/dev/nvme0n1p3 27b847a8-9336-408c-88e1-9c4c017547c8:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 03 22:22:50.751298 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.741980 5196 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0d307ea1d74a23a33ed4b95e70c747f17dfebf86248ec7bbaf3c0d67eb8b95e9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0d307ea1d74a23a33ed4b95e70c747f17dfebf86248ec7bbaf3c0d67eb8b95e9/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/11620f2ef1ce4776d1fe1a60b27ba56db85c0fb6d4c806344ac114bb54456f59/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/11620f2ef1ce4776d1fe1a60b27ba56db85c0fb6d4c806344ac114bb54456f59/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/20100914eddea7d757ce5f2aa3d713348c92743c7a31753896979a9a188e758c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/20100914eddea7d757ce5f2aa3d713348c92743c7a31753896979a9a188e758c/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2127e840e13547e934d97137d91178a5271c7d343a979272861e66faf1fd80f6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2127e840e13547e934d97137d91178a5271c7d343a979272861e66faf1fd80f6/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2b4d8589545194c7a73b28d8b87d606ffcb4749772644f101dd0d5956cf6cb4f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2b4d8589545194c7a73b28d8b87d606ffcb4749772644f101dd0d5956cf6cb4f/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2cd7eb9d637d8de82bc72f1c4d6cf376f2e35ba9e47be08ddd28756f0239bc7f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2cd7eb9d637d8de82bc72f1c4d6cf376f2e35ba9e47be08ddd28756f0239bc7f/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e8ff8210cf78b240869c8813a6efa43f0cc8e97887c0161838e754a0aabb833/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e8ff8210cf78b240869c8813a6efa43f0cc8e97887c0161838e754a0aabb833/userdata/shm major:0 minor:299 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2f31c2f531cff99e34c0b986817d4291f63faa877f61fb1abb177d3a325faeb5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2f31c2f531cff99e34c0b986817d4291f63faa877f61fb1abb177d3a325faeb5/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/31e115bc65dc1b6ad686c8a800c4e28b7daa4c4ce1e67ab05d1218ce18aef29b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/31e115bc65dc1b6ad686c8a800c4e28b7daa4c4ce1e67ab05d1218ce18aef29b/userdata/shm major:0 minor:297 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/34f8c7a6a612c6b18d7e00d44eaeccd5bf769ade6933c792f4c3565d04ab8dde/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/34f8c7a6a612c6b18d7e00d44eaeccd5bf769ade6933c792f4c3565d04ab8dde/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/36442bbfb7f8ef0153c97d3e49504fbf236257d4a7b120ce8403585766bb4b69/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/36442bbfb7f8ef0153c97d3e49504fbf236257d4a7b120ce8403585766bb4b69/userdata/shm major:0 minor:249 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3ccc68b5b925803a4907d8bb570385c4da297fe156d3cccbc13f5b5393bfe0cb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3ccc68b5b925803a4907d8bb570385c4da297fe156d3cccbc13f5b5393bfe0cb/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6fcda1ab02615ec2ec6ae197eda91031ad640509ee6d23b4425b6ade0757214a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6fcda1ab02615ec2ec6ae197eda91031ad640509ee6d23b4425b6ade0757214a/userdata/shm major:0 minor:303 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/73a265ef5f45b3ac3d94f657dfcb36bea82a6fc504e1212b5f7b2134f096ce40/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/73a265ef5f45b3ac3d94f657dfcb36bea82a6fc504e1212b5f7b2134f096ce40/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/73a5922cdfee41d93928be7c593ec6ee40292e4888ec656fb0de1ea3834c8062/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/73a5922cdfee41d93928be7c593ec6ee40292e4888ec656fb0de1ea3834c8062/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/843ca26127fa43d9fd34ef88d0fae215a9fb7bcb3e65d1925511dc7d0fa9e843/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/843ca26127fa43d9fd34ef88d0fae215a9fb7bcb3e65d1925511dc7d0fa9e843/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8f4e96c3a6e48b918986d2ecd8b46c7fc4a7de6ec45a32278cc33c191b90d10d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8f4e96c3a6e48b918986d2ecd8b46c7fc4a7de6ec45a32278cc33c191b90d10d/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c8c68876c533e2f9e7aa85688046333b962cf0f2909987e8eb0a3e99c7afe8ab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c8c68876c533e2f9e7aa85688046333b962cf0f2909987e8eb0a3e99c7afe8ab/userdata/shm major:0 minor:271 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cc5da91da75f5484b0e4a722d92d72b7058291a8db65f386076b30594d68ade0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cc5da91da75f5484b0e4a722d92d72b7058291a8db65f386076b30594d68ade0/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cde676372f389b72eb7b34513e4f61f2ab6261fbf9e9aad50e5348722e7d51ba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cde676372f389b72eb7b34513e4f61f2ab6261fbf9e9aad50e5348722e7d51ba/userdata/shm major:0 minor:292 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fd424519b7120717bc3041a2154a7338c2064d0aa6855d0a355f48237626b29a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fd424519b7120717bc3041a2154a7338c2064d0aa6855d0a355f48237626b29a/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e68894d-ed4f-4bc5-b36e-691e557883bd/volumes/kubernetes.io~projected/kube-api-access-z7qg9:{mountpoint:/var/lib/kubelet/pods/0e68894d-ed4f-4bc5-b36e-691e557883bd/volumes/kubernetes.io~projected/kube-api-access-z7qg9 major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e68894d-ed4f-4bc5-b36e-691e557883bd/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/0e68894d-ed4f-4bc5-b36e-691e557883bd/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e68894d-ed4f-4bc5-b36e-691e557883bd/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/0e68894d-ed4f-4bc5-b36e-691e557883bd/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1618fd8a-9a1b-4437-8559-86690d19cc9d/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/1618fd8a-9a1b-4437-8559-86690d19cc9d/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1618fd8a-9a1b-4437-8559-86690d19cc9d/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/1618fd8a-9a1b-4437-8559-86690d19cc9d/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1618fd8a-9a1b-4437-8559-86690d19cc9d/volumes/kubernetes.io~projected/kube-api-access-9988v:{mountpoint:/var/lib/kubelet/pods/1618fd8a-9a1b-4437-8559-86690d19cc9d/volumes/kubernetes.io~projected/kube-api-access-9988v major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b2df441-74d4-435f-a56a-5eff3ec4575d/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/1b2df441-74d4-435f-a56a-5eff3ec4575d/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b2df441-74d4-435f-a56a-5eff3ec4575d/volumes/kubernetes.io~projected/kube-api-access-s4n2l:{mountpoint:/var/lib/kubelet/pods/1b2df441-74d4-435f-a56a-5eff3ec4575d/volumes/kubernetes.io~projected/kube-api-access-s4n2l major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b2df441-74d4-435f-a56a-5eff3ec4575d/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/1b2df441-74d4-435f-a56a-5eff3ec4575d/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/254d9855-203d-43f2-9c75-ed2eb84c4471/volumes/kubernetes.io~projected/kube-api-access-kfsqx:{mountpoint:/var/lib/kubelet/pods/254d9855-203d-43f2-9c75-ed2eb84c4471/volumes/kubernetes.io~projected/kube-api-access-kfsqx major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/254d9855-203d-43f2-9c75-ed2eb84c4471/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/254d9855-203d-43f2-9c75-ed2eb84c4471/volumes/kubernetes.io~secret/metrics-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29b82faa-1aa1-4760-a9f6-1cbeb9427094/volumes/kubernetes.io~projected/kube-api-access-g5zc6:{mountpoint:/var/lib/kubelet/pods/29b82faa-1aa1-4760-a9f6-1cbeb9427094/volumes/kubernetes.io~projected/kube-api-access-g5zc6 major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f63f886-379d-4636-8fb3-21b932a67831/volumes/kubernetes.io~projected/kube-api-access-69c5d:{mountpoint:/var/lib/kubelet/pods/3f63f886-379d-4636-8fb3-21b932a67831/volumes/kubernetes.io~projected/kube-api-access-69c5d major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e68337a-9274-466d-a238-082fdf86f4cb/volumes/kubernetes.io~projected/kube-api-access-p9swx:{mountpoint:/var/lib/kubelet/pods/4e68337a-9274-466d-a238-082fdf86f4cb/volumes/kubernetes.io~projected/kube-api-access-p9swx major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/514a860a-f1e3-4714-8493-367b6b8b32a2/volumes/kubernetes.io~projected/kube-api-access-mmjhw:{mountpoint:/var/lib/kubelet/pods/514a860a-f1e3-4714-8493-367b6b8b32a2/volumes/kubernetes.io~projected/kube-api-access-mmjhw major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b40d10d-8dd1-46f6-b897-82780c392c84/volumes/kubernetes.io~projected/kube-api-access-6lrhw:{mountpoint:/var/lib/kubelet/pods/5b40d10d-8dd1-46f6-b897-82780c392c84/volumes/kubernetes.io~projected/kube-api-access-6lrhw major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b40d10d-8dd1-46f6-b897-82780c392c84/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/5b40d10d-8dd1-46f6-b897-82780c392c84/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volumes/kubernetes.io~empty-dir/config-out major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volumes/kubernetes.io~projected/kube-api-access-gqg7s:{mountpoint:/var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volumes/kubernetes.io~projected/kube-api-access-gqg7s major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volumes/kubernetes.io~projected/tls-assets major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volumes/kubernetes.io~secret/config major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volumes/kubernetes.io~secret/web-config major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a90468d5-e140-4ae4-a1cb-c8f9dec579ff/volumes/kubernetes.io~projected/kube-api-access-9jqsx:{mountpoint:/var/lib/kubelet/pods/a90468d5-e140-4ae4-a1cb-c8f9dec579ff/volumes/kubernetes.io~projected/kube-api-access-9jqsx major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0ba09d3-25f2-422b-b5e2-357aabcaf81c/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/b0ba09d3-25f2-422b-b5e2-357aabcaf81c/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5730579-cca5-46be-9fbb-da9f7ae58d14/volumes/kubernetes.io~projected/kube-api-access-q5vnj:{mountpoint:/var/lib/kubelet/pods/b5730579-cca5-46be-9fbb-da9f7ae58d14/volumes/kubernetes.io~projected/kube-api-access-q5vnj major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5730579-cca5-46be-9fbb-da9f7ae58d14/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/b5730579-cca5-46be-9fbb-da9f7ae58d14/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6154c5a-4925-4c74-8db4-0acdb525b743/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/b6154c5a-4925-4c74-8db4-0acdb525b743/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf4a34cf-5e64-407a-aaf7-0d211a6be6f6/volumes/kubernetes.io~projected/kube-api-access-4fhh6:{mountpoint:/var/lib/kubelet/pods/bf4a34cf-5e64-407a-aaf7-0d211a6be6f6/volumes/kubernetes.io~projected/kube-api-access-4fhh6 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf4a34cf-5e64-407a-aaf7-0d211a6be6f6/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/bf4a34cf-5e64-407a-aaf7-0d211a6be6f6/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c027d694-64cc-46a2-b4e7-49b117f696d3/volumes/kubernetes.io~projected/kube-api-access-sqdbp:{mountpoint:/var/lib/kubelet/pods/c027d694-64cc-46a2-b4e7-49b117f696d3/volumes/kubernetes.io~projected/kube-api-access-sqdbp major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c027d694-64cc-46a2-b4e7-49b117f696d3/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/c027d694-64cc-46a2-b4e7-49b117f696d3/volumes/kubernetes.io~secret/metrics-certs major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6b58b28-fb4b-4635-a93a-0bd23222670b/volumes/kubernetes.io~projected/kube-api-access-wxz2n:{mountpoint:/var/lib/kubelet/pods/c6b58b28-fb4b-4635-a93a-0bd23222670b/volumes/kubernetes.io~projected/kube-api-access-wxz2n major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4425733-4528-4caa-99cb-e94072a00f78/volumes/kubernetes.io~projected/kube-api-access-zlg2q:{mountpoint:/var/lib/kubelet/pods/d4425733-4528-4caa-99cb-e94072a00f78/volumes/kubernetes.io~projected/kube-api-access-zlg2q major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8f1371d-471f-496f-ac95-19eeb163c2c7/volumes/kubernetes.io~projected/kube-api-access-mbjsn:{mountpoint:/var/lib/kubelet/pods/d8f1371d-471f-496f-ac95-19eeb163c2c7/volumes/kubernetes.io~projected/kube-api-access-mbjsn major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e6690a9b-44fa-4b1f-9523-173d8481dbea/volumes/kubernetes.io~projected/kube-api-access-cst2m:{mountpoint:/var/lib/kubelet/pods/e6690a9b-44fa-4b1f-9523-173d8481dbea/volumes/kubernetes.io~projected/kube-api-access-cst2m major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e6690a9b-44fa-4b1f-9523-173d8481dbea/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/e6690a9b-44fa-4b1f-9523-173d8481dbea/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e6690a9b-44fa-4b1f-9523-173d8481dbea/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/e6690a9b-44fa-4b1f-9523-173d8481dbea/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:236 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/857fc8841a4bc1a4e7733a6e7904a91fbf9cb3115164b9d9fecd9a89106eb14d/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/c247aea3ac9b0fe9af5f7a2ec20c3badce5923e32690511142c452eb31b24f91/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/b1b9fc6fde29db9f43772b41e7b944e336179d97555c2330f66a70ede3ff750a/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/64f9ed694aefad02008d79095fcf963ead5115c479bc12d7823e96a56a24d2e9/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/420e8a4cd98f0b30a3e4666d83db91e1527a645fcbbef004a41ac99f52e5a0a1/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/8c02e8c82e97d31077556e260991ef247f9ce4e94280f5f350f38a7421366ff1/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-122:{mountpoint:/var/lib/containers/storage/overlay/8bca7cda6d4960e149cc37bf09634ee2ab2370a14e777f4c04103ab49ad9d452/merged major:0 minor:122 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/85b1894b0b6b859d039048f92293d762b128663dcab65d1a44ab4ba0a9545461/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/32dfe3a2725b0a7ed50c8d92ac2828da7d782465abb81131a9e3d15966f9b276/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/cc3d20eb999141901f97af5c0412fc5ba12dcdd30450aaab8395de3a68738ed3/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-145:{mountpoint:/var/lib/containers/storage/overlay/605f0000b12781533fe966165f3363776719e93fb5278998d799e8567ebc14f3/merged major:0 minor:145 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/c495d075e6579a300ebab137864bf8ada50ef620633a1a80693b8ccf21edb6a0/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/7c4308f51b051919a4b8168d034d8a341a56bc3629605881408da69999f958a3/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/2e08ab416d5ed75f0684dd21d5b0226ef4f5446e4f68bf4beef4452f84f6e430/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/0845b1239cc45a874c4ca1a1d355ffe864ee8dec21240e577287a553fca45d64/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-164:{mountpoint:/var/lib/containers/storage/overlay/e06144dc6724cd4fb40606ff39bdb4f8a2806ecc9be2360584bde3ba16ed73c1/merged major:0 minor:164 fsType:overlay blockSize:0} overlay_0-168:{mountpoint:/var/lib/containers/storage/overlay/e25369c916bdfdc6a68a1ea978b06141c301c9d0c068ea6b701ef59b8d5fd916/merged major:0 minor:168 fsType:overlay blockSize:0} overlay_0-171:{mountpoint:/var/lib/containers/storage/overlay/22e807e669e27c3b095c1c29f7258b7e529352d10e1f0860e3370be5c325d03e/merged major:0 minor:171 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/f427a9fbd4dd40e82cacb62f9d07608bab25a97347ec4d226e18fc5f84eac0d2/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/ee81bab76ff36c46ddae2908087c83f598e6518303589c30d0a21a08943d9bb6/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/bdeb2528bb22740687bd1be3961da731a22c6b5ea0b0e91bf228c82efe715168/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/c7dd247f2615dd357eef3b108a7f60214ff1a33236bb196f54d57a6be986d7db/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/ad0a470b360d33d27ff4155003ea380339b39ab8c09d682539eab9688079908f/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/d400217d8c7e61cc1ef620410affdfef955bea8f640ee43c472007dd2a0d03f2/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/7935dc7b8ef23bfba69725290b0b15c1f0d3c3826c979fe287b99ba72aa3a736/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/c7bf75c03446fa4bf625b982f856a82382507856703e45a502357ee8d57130fd/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/c8f0eda0544f7786c44bb61fbe5c1cc0f317ac20e214190b407d0b29b1fe6853/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/cc338f2a6b1d6ea30eee77cdbcb6c98209904910f09f60957f8f3190ae0bee5d/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/1a2a737a90e359aaade004996634ef204a125050a2d595ffb820f0146eb8e773/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-244:{mountpoint:/var/lib/containers/storage/overlay/bfe41382b9b274f114719680f65c4a71dc0c06c93fab5ea13736d1e9dc227edc/merged major:0 minor:244 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/7a2c933e417b963e578ceff9397791514cb05b5f4aa958d6e153fef534cce869/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/b9ed7b2d8ee70b543d21cddff74a331d531b8bbbe6fc79485891d144d117ccaa/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/52bbec2b80c2f6674f7eb8cf1a908a3805488c0a523e1e75de0e528fe1380a98/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/82a1a85434ca3145f93e118a45e81b405f2a13b158a9352bc14e31172b21f0c8/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/63568983457a2f48bee3631a9ebba4d6b2c3068eaef5214ebf270efe53319732/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/014cd7b65b02435058c0a5a0a85553136bd0a601833584aba7835377078a5550/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/6f77c1f973142e704ad0bf9898e73833fe5a1f5b781aff3fcee9a12b32b94362/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/0045a07b4cd29d55aedd0fd05fae4cd7452b22f2d9d6e3c710da3bd5d46f15ea/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/d0ec5d0b6e9ca0ed6fd622aac61a05f1c50ffeaa0be47a6e65060ffd2f60b605/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/657697da3355a2f4ad0d5ce3463eb36ff274d99295853be977f5be9fa9f66d22/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/ced2d8227b710a304b7bcd003ab23c066d277c5481016a8d8da70086001124e4/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/d5c3e63073eecbb07045426f5ac9238107c263a730cef929948b4a972d8fa367/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/f9adedf260c88323f4ccd4385f4265434f8404371f2d0b4dafc81bc7d5fa271f/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/e44f703c6251218864f9d61d75f657eb96121dab440d5da531dfdb4a0fd7ca81/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/732841c9dbbbd550458ac92ff4e7b086a06ea1d30a4c4241d3b6d5135c73aa90/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/4cfe6105b9a64bf0fc867080af9e372a77ea8309f0382b99e194fc2bd98635c9/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/3dbc3d82cce94b6aa5d8e264572103c1ecdc6b93524f567e8de5b0f4d71a33d8/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/562d78d78b747d89a35d19b4c900fac66b0374c5fb5839a69d083c98097c001a/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/9ebedd7206827c1b10fed169d14f5caf3e80fb62f8af1284d55c2ad9787f4b68/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/4827a3675305a6f4d1d36a79986299d3e0443047c4528c41ec8fe5e20065458f/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/851b61073308754615914e5f098d94d4406ccd6684f82f687e3f09467f125720/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/af6da638921520da23321606e532f885bcac3f595c3e0b8c06ea4a48f2afaa19/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/708f878dda9a2a37fd0d4a7ad3bc2a201649ebf9fdda7add927d6b06a34c60db/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/98de6ddd1d08d7fc3711f8e7d507e32660e056ba8791b4f80c2caf98f25bc97b/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/c7e8ac7e65ad0d61b47f2e0f11fe303253656d38a0567418c4eb01ec2a864fb5/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/e3a0e46a8d361ef8d637d11ab3a8cbde44f69ba1913ee540461726cdf2810127/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/a6630cfcd71cca86c948d41e540dc1025ff69d9b2c2838c881829c9bc3cdcda0/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 22:22:50.752515 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.751625 5196 manager.go:217] Machine: {Timestamp:2026-11-03 22:22:50.750968687 +0000 UTC m=+0.060527971 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099528 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2feb0695c7105c636a56e85bfc4670 SystemUUID:ec2feb06-95c7-105c-636a-56e85bfc4670 BootID:b8b014e0-7ed1-4ab1-a7b7-0ffee440ad2d Filesystems:[{Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e6690a9b-44fa-4b1f-9523-173d8481dbea/volumes/kubernetes.io~projected/kube-api-access-cst2m DeviceMajor:0 DeviceMinor:241 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a90468d5-e140-4ae4-a1cb-c8f9dec579ff/volumes/kubernetes.io~projected/kube-api-access-9jqsx DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/73a265ef5f45b3ac3d94f657dfcb36bea82a6fc504e1212b5f7b2134f096ce40/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/73a5922cdfee41d93928be7c593ec6ee40292e4888ec656fb0de1ea3834c8062/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:325 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volumes/kubernetes.io~projected/kube-api-access-gqg7s DeviceMajor:0 DeviceMinor:339 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-171 DeviceMajor:0 DeviceMinor:171 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c8c68876c533e2f9e7aa85688046333b962cf0f2909987e8eb0a3e99c7afe8ab/userdata/shm DeviceMajor:0 DeviceMinor:271 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2cd7eb9d637d8de82bc72f1c4d6cf376f2e35ba9e47be08ddd28756f0239bc7f/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:335 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c027d694-64cc-46a2-b4e7-49b117f696d3/volumes/kubernetes.io~projected/kube-api-access-sqdbp DeviceMajor:0 DeviceMinor:52 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c6b58b28-fb4b-4635-a93a-0bd23222670b/volumes/kubernetes.io~projected/kube-api-access-wxz2n DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/34f8c7a6a612c6b18d7e00d44eaeccd5bf769ade6933c792f4c3565d04ab8dde/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b40d10d-8dd1-46f6-b897-82780c392c84/volumes/kubernetes.io~projected/kube-api-access-6lrhw DeviceMajor:0 DeviceMinor:286 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-244 DeviceMajor:0 DeviceMinor:244 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:330 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2b4d8589545194c7a73b28d8b87d606ffcb4749772644f101dd0d5956cf6cb4f/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:330 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d8f1371d-471f-496f-ac95-19eeb163c2c7/volumes/kubernetes.io~projected/kube-api-access-mbjsn DeviceMajor:0 DeviceMinor:291 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-168 DeviceMajor:0 DeviceMinor:168 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b40d10d-8dd1-46f6-b897-82780c392c84/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:282 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6fcda1ab02615ec2ec6ae197eda91031ad640509ee6d23b4425b6ade0757214a/userdata/shm DeviceMajor:0 DeviceMinor:303 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/254d9855-203d-43f2-9c75-ed2eb84c4471/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bf4a34cf-5e64-407a-aaf7-0d211a6be6f6/volumes/kubernetes.io~projected/kube-api-access-4fhh6 DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf4a34cf-5e64-407a-aaf7-0d211a6be6f6/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:329 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:338 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e6690a9b-44fa-4b1f-9523-173d8481dbea/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:240 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0e68894d-ed4f-4bc5-b36e-691e557883bd/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:265 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b0ba09d3-25f2-422b-b5e2-357aabcaf81c/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1618fd8a-9a1b-4437-8559-86690d19cc9d/volumes/kubernetes.io~projected/kube-api-access-9988v DeviceMajor:0 DeviceMinor:54 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/36442bbfb7f8ef0153c97d3e49504fbf236257d4a7b120ce8403585766bb4b69/userdata/shm DeviceMajor:0 DeviceMinor:249 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e8ff8210cf78b240869c8813a6efa43f0cc8e97887c0161838e754a0aabb833/userdata/shm DeviceMajor:0 DeviceMinor:299 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0d307ea1d74a23a33ed4b95e70c747f17dfebf86248ec7bbaf3c0d67eb8b95e9/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fd424519b7120717bc3041a2154a7338c2064d0aa6855d0a355f48237626b29a/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/20100914eddea7d757ce5f2aa3d713348c92743c7a31753896979a9a188e758c/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:337 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1618fd8a-9a1b-4437-8559-86690d19cc9d/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-145 DeviceMajor:0 DeviceMinor:145 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2127e840e13547e934d97137d91178a5271c7d343a979272861e66faf1fd80f6/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1b2df441-74d4-435f-a56a-5eff3ec4575d/volumes/kubernetes.io~projected/kube-api-access-s4n2l DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8f4e96c3a6e48b918986d2ecd8b46c7fc4a7de6ec45a32278cc33c191b90d10d/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c027d694-64cc-46a2-b4e7-49b117f696d3/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:295 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/31e115bc65dc1b6ad686c8a800c4e28b7daa4c4ce1e67ab05d1218ce18aef29b/userdata/shm DeviceMajor:0 DeviceMinor:297 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-122 DeviceMajor:0 DeviceMinor:122 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5730579-cca5-46be-9fbb-da9f7ae58d14/volumes/kubernetes.io~projected/kube-api-access-q5vnj DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/29b82faa-1aa1-4760-a9f6-1cbeb9427094/volumes/kubernetes.io~projected/kube-api-access-g5zc6 DeviceMajor:0 DeviceMinor:296 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:332 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/4e68337a-9274-466d-a238-082fdf86f4cb/volumes/kubernetes.io~projected/kube-api-access-p9swx DeviceMajor:0 DeviceMinor:51 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d4425733-4528-4caa-99cb-e94072a00f78/volumes/kubernetes.io~projected/kube-api-access-zlg2q DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/254d9855-203d-43f2-9c75-ed2eb84c4471/volumes/kubernetes.io~projected/kube-api-access-kfsqx DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cde676372f389b72eb7b34513e4f61f2ab6261fbf9e9aad50e5348722e7d51ba/userdata/shm DeviceMajor:0 DeviceMinor:292 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b6154c5a-4925-4c74-8db4-0acdb525b743/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:294 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:333 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3f63f886-379d-4636-8fb3-21b932a67831/volumes/kubernetes.io~projected/kube-api-access-69c5d DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5730579-cca5-46be-9fbb-da9f7ae58d14/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:331 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/11620f2ef1ce4776d1fe1a60b27ba56db85c0fb6d4c806344ac114bb54456f59/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0e68894d-ed4f-4bc5-b36e-691e557883bd/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:269 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:336 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1618fd8a-9a1b-4437-8559-86690d19cc9d/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cc5da91da75f5484b0e4a722d92d72b7058291a8db65f386076b30594d68ade0/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1b2df441-74d4-435f-a56a-5eff3ec4575d/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:334 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0e68894d-ed4f-4bc5-b36e-691e557883bd/volumes/kubernetes.io~projected/kube-api-access-z7qg9 DeviceMajor:0 DeviceMinor:270 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3ccc68b5b925803a4907d8bb570385c4da297fe156d3cccbc13f5b5393bfe0cb/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2f31c2f531cff99e34c0b986817d4291f63faa877f61fb1abb177d3a325faeb5/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1b2df441-74d4-435f-a56a-5eff3ec4575d/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/514a860a-f1e3-4714-8493-367b6b8b32a2/volumes/kubernetes.io~projected/kube-api-access-mmjhw DeviceMajor:0 DeviceMinor:53 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-164 DeviceMajor:0 DeviceMinor:164 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e6690a9b-44fa-4b1f-9523-173d8481dbea/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:236 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/843ca26127fa43d9fd34ef88d0fae215a9fb7bcb3e65d1925511dc7d0fa9e843/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:2b4d8589545194c MacAddress:22:86:6f:15:da:17 Speed:10000 Mtu:8901} {Name:2cd7eb9d637d8de MacAddress:8e:25:4b:ca:c9:c6 Speed:10000 Mtu:8901} {Name:2e8ff8210cf78b2 MacAddress:6e:97:fb:4a:e9:ee Speed:10000 Mtu:8901} {Name:31e115bc65dc1b6 MacAddress:22:bd:8d:6b:fe:0b Speed:10000 Mtu:8901} {Name:34f8c7a6a612c6b MacAddress:d6:9f:99:dd:3e:09 Speed:10000 Mtu:8901} {Name:36442bbfb7f8ef0 MacAddress:36:13:31:4a:19:71 Speed:10000 Mtu:8901} {Name:3ccc68b5b925803 MacAddress:b2:d7:92:8f:c0:9f Speed:10000 Mtu:8901} {Name:6fcda1ab02615ec MacAddress:d6:09:73:a4:80:2c Speed:10000 Mtu:8901} {Name:843ca26127fa43d MacAddress:8a:99:29:37:26:6e Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:79:df:2e:8e:41 Speed:0 Mtu:9001} {Name:br-int MacAddress:d2:c0:ac:83:6b:6c Speed:0 Mtu:8901} {Name:cde676372f389b7 MacAddress:66:15:68:96:56:82 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:79:df:2e:8e:41 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:da:d3:d2:dc:9f:19 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:2e:67:7f:b0:07:f6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 22:22:50.752799 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.752522 5196 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 22:22:50.752799 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.752604 5196 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 22:22:50.753152 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.753130 5196 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 22:22:50.753324 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.753155 5196 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-132.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 22:22:50.753372 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.753328 5196 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 22:22:50.753372 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.753337 5196 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 22:22:50.753372 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.753348 5196 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 22:22:50.753372 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.753366 5196 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 22:22:50.753475 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.753432 5196 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 22:22:50.753583 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.753573 5196 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 22:22:50.753674 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.753666 5196 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 22:22:50.753711 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.753701 5196 kubelet.go:493] "Attempting to sync node with API server" Nov 03 22:22:50.753740 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.753714 5196 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 22:22:50.753740 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.753723 5196 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 22:22:50.753740 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.753732 5196 kubelet.go:405] "Adding apiserver pod source" Nov 03 22:22:50.753740 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.753740 5196 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 22:22:50.754452 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.754436 5196 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 22:22:50.754452 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.754449 5196 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 22:22:50.755079 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.755062 5196 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 22:22:50.764897 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.764878 5196 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 22:22:50.764897 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.764894 5196 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 22:22:50.765029 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.765011 5196 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 22:22:50.765268 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.765251 5196 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 22:22:50.765335 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.765292 5196 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 22:22:50.765415 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.765404 5196 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 22:22:50.765469 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.765423 5196 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 22:22:50.765469 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.765434 5196 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 22:22:50.765469 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.765444 5196 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 22:22:50.765469 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.765452 5196 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 22:22:50.765469 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.765462 5196 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 22:22:50.765696 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.765471 5196 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 22:22:50.765696 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.765481 5196 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 22:22:50.765696 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.765492 5196 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 22:22:50.765696 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.765502 5196 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 22:22:50.765696 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.765516 5196 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 22:22:50.765696 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.765531 5196 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 22:22:50.765696 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.765565 5196 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 22:22:50.765696 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.765575 5196 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 22:22:50.769111 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.769096 5196 server.go:1263] "Started kubelet" Nov 03 22:22:50.769239 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.769183 5196 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 22:22:50.769289 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.769247 5196 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 22:22:50.769329 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.769295 5196 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 22:22:50.769637 ip-10-0-143-132 systemd[1]: Started Kubernetes Kubelet. Nov 03 22:22:50.770010 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.769807 5196 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 22:22:50.770798 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.770781 5196 server.go:317] "Adding debug handlers to kubelet server" Nov 03 22:22:50.775504 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.775424 5196 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 22:22:50.775504 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.775446 5196 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 22:22:50.775504 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.775497 5196 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 22:16:38 +0000 UTC" deadline="2028-04-15 22:00:55.4833944 +0000 UTC" Nov 03 22:22:50.775774 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.775515 5196 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12695h38m4.707881853s" Nov 03 22:22:50.775774 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.775548 5196 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 22:22:50.775774 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.775555 5196 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 22:22:50.775774 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.775648 5196 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 22:22:50.776107 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:22:50.775940 5196 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-132.us-west-1.compute.internal\" not found" Nov 03 22:22:50.776720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.776698 5196 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 22:22:50.776845 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.776724 5196 factory.go:55] Registering systemd factory Nov 03 22:22:50.776845 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.776737 5196 factory.go:223] Registration of the systemd container factory successfully Nov 03 22:22:50.777330 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.777313 5196 factory.go:153] Registering CRI-O factory Nov 03 22:22:50.777330 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.777332 5196 factory.go:223] Registration of the crio container factory successfully Nov 03 22:22:50.777452 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.777349 5196 factory.go:103] Registering Raw factory Nov 03 22:22:50.777452 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.777362 5196 manager.go:1199] Started watching for new ooms in manager Nov 03 22:22:50.777712 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.777697 5196 manager.go:319] Starting recovery of all containers Nov 03 22:22:50.778165 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778126 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4425733-4528-4caa-99cb-e94072a00f78" volumeName="kubernetes.io/configmap/d4425733-4528-4caa-99cb-e94072a00f78-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 22:22:50.778257 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778174 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e68894d-ed4f-4bc5-b36e-691e557883bd" volumeName="kubernetes.io/configmap/0e68894d-ed4f-4bc5-b36e-691e557883bd-metrics-client-ca" seLinuxMountContext="" Nov 03 22:22:50.778257 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778190 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b2df441-74d4-435f-a56a-5eff3ec4575d" volumeName="kubernetes.io/configmap/1b2df441-74d4-435f-a56a-5eff3ec4575d-ovnkube-config" seLinuxMountContext="" Nov 03 22:22:50.778257 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778204 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f63f886-379d-4636-8fb3-21b932a67831" volumeName="kubernetes.io/empty-dir/3f63f886-379d-4636-8fb3-21b932a67831-tmp-dir" seLinuxMountContext="" Nov 03 22:22:50.778257 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778218 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a90468d5-e140-4ae4-a1cb-c8f9dec579ff" volumeName="kubernetes.io/projected/a90468d5-e140-4ae4-a1cb-c8f9dec579ff-kube-api-access-9jqsx" seLinuxMountContext="" Nov 03 22:22:50.778257 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778230 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5730579-cca5-46be-9fbb-da9f7ae58d14" volumeName="kubernetes.io/configmap/b5730579-cca5-46be-9fbb-da9f7ae58d14-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 03 22:22:50.778257 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778243 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6154c5a-4925-4c74-8db4-0acdb525b743" volumeName="kubernetes.io/secret/b6154c5a-4925-4c74-8db4-0acdb525b743-original-pull-secret" seLinuxMountContext="" Nov 03 22:22:50.778257 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778257 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4425733-4528-4caa-99cb-e94072a00f78" volumeName="kubernetes.io/configmap/d4425733-4528-4caa-99cb-e94072a00f78-cni-binary-copy" seLinuxMountContext="" Nov 03 22:22:50.778457 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778273 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e68894d-ed4f-4bc5-b36e-691e557883bd" volumeName="kubernetes.io/secret/0e68894d-ed4f-4bc5-b36e-691e557883bd-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 22:22:50.778457 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778285 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e68894d-ed4f-4bc5-b36e-691e557883bd" volumeName="kubernetes.io/projected/0e68894d-ed4f-4bc5-b36e-691e557883bd-kube-api-access-z7qg9" seLinuxMountContext="" Nov 03 22:22:50.778457 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778313 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1618fd8a-9a1b-4437-8559-86690d19cc9d" volumeName="kubernetes.io/empty-dir/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-tuned" seLinuxMountContext="" Nov 03 22:22:50.778457 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778325 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="254d9855-203d-43f2-9c75-ed2eb84c4471" volumeName="kubernetes.io/empty-dir/254d9855-203d-43f2-9c75-ed2eb84c4471-tmp-dir" seLinuxMountContext="" Nov 03 22:22:50.778457 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778340 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" volumeName="kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Nov 03 22:22:50.778457 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778355 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6690a9b-44fa-4b1f-9523-173d8481dbea" volumeName="kubernetes.io/projected/e6690a9b-44fa-4b1f-9523-173d8481dbea-kube-api-access-cst2m" seLinuxMountContext="" Nov 03 22:22:50.778457 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778370 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5730579-cca5-46be-9fbb-da9f7ae58d14" volumeName="kubernetes.io/projected/b5730579-cca5-46be-9fbb-da9f7ae58d14-kube-api-access-q5vnj" seLinuxMountContext="" Nov 03 22:22:50.778457 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778382 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1618fd8a-9a1b-4437-8559-86690d19cc9d" volumeName="kubernetes.io/empty-dir/1618fd8a-9a1b-4437-8559-86690d19cc9d-tmp" seLinuxMountContext="" Nov 03 22:22:50.778457 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778394 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29b82faa-1aa1-4760-a9f6-1cbeb9427094" volumeName="kubernetes.io/projected/29b82faa-1aa1-4760-a9f6-1cbeb9427094-kube-api-access-g5zc6" seLinuxMountContext="" Nov 03 22:22:50.778457 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778405 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b40d10d-8dd1-46f6-b897-82780c392c84" volumeName="kubernetes.io/projected/5b40d10d-8dd1-46f6-b897-82780c392c84-kube-api-access-6lrhw" seLinuxMountContext="" Nov 03 22:22:50.778457 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778418 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" volumeName="kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Nov 03 22:22:50.778457 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778432 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6690a9b-44fa-4b1f-9523-173d8481dbea" volumeName="kubernetes.io/configmap/e6690a9b-44fa-4b1f-9523-173d8481dbea-metrics-client-ca" seLinuxMountContext="" Nov 03 22:22:50.778457 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778448 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1618fd8a-9a1b-4437-8559-86690d19cc9d" volumeName="kubernetes.io/projected/1618fd8a-9a1b-4437-8559-86690d19cc9d-kube-api-access-9988v" seLinuxMountContext="" Nov 03 22:22:50.778793 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778460 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="254d9855-203d-43f2-9c75-ed2eb84c4471" volumeName="kubernetes.io/secret/254d9855-203d-43f2-9c75-ed2eb84c4471-metrics-tls" seLinuxMountContext="" Nov 03 22:22:50.778793 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778473 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e68337a-9274-466d-a238-082fdf86f4cb" volumeName="kubernetes.io/configmap/4e68337a-9274-466d-a238-082fdf86f4cb-cni-binary-copy" seLinuxMountContext="" Nov 03 22:22:50.778793 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778498 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" volumeName="kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-trusted-ca-bundle" seLinuxMountContext="" Nov 03 22:22:50.778793 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778511 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" volumeName="kubernetes.io/projected/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-tls-assets" seLinuxMountContext="" Nov 03 22:22:50.778793 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778538 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b2df441-74d4-435f-a56a-5eff3ec4575d" volumeName="kubernetes.io/secret/1b2df441-74d4-435f-a56a-5eff3ec4575d-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 22:22:50.778793 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778566 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="514a860a-f1e3-4714-8493-367b6b8b32a2" volumeName="kubernetes.io/projected/514a860a-f1e3-4714-8493-367b6b8b32a2-kube-api-access-mmjhw" seLinuxMountContext="" Nov 03 22:22:50.778793 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778581 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" volumeName="kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-config" seLinuxMountContext="" Nov 03 22:22:50.778793 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778598 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5730579-cca5-46be-9fbb-da9f7ae58d14" volumeName="kubernetes.io/empty-dir/b5730579-cca5-46be-9fbb-da9f7ae58d14-data-volume" seLinuxMountContext="" Nov 03 22:22:50.778793 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778610 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c027d694-64cc-46a2-b4e7-49b117f696d3" volumeName="kubernetes.io/secret/c027d694-64cc-46a2-b4e7-49b117f696d3-metrics-certs" seLinuxMountContext="" Nov 03 22:22:50.778793 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778623 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e68894d-ed4f-4bc5-b36e-691e557883bd" volumeName="kubernetes.io/secret/0e68894d-ed4f-4bc5-b36e-691e557883bd-node-exporter-tls" seLinuxMountContext="" Nov 03 22:22:50.778793 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778636 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" volumeName="kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-grpc-tls" seLinuxMountContext="" Nov 03 22:22:50.778793 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778649 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a90468d5-e140-4ae4-a1cb-c8f9dec579ff" volumeName="kubernetes.io/configmap/a90468d5-e140-4ae4-a1cb-c8f9dec579ff-iptables-alerter-script" seLinuxMountContext="" Nov 03 22:22:50.778793 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778662 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8f1371d-471f-496f-ac95-19eeb163c2c7" volumeName="kubernetes.io/projected/d8f1371d-471f-496f-ac95-19eeb163c2c7-kube-api-access-mbjsn" seLinuxMountContext="" Nov 03 22:22:50.778793 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778674 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6690a9b-44fa-4b1f-9523-173d8481dbea" volumeName="kubernetes.io/secret/e6690a9b-44fa-4b1f-9523-173d8481dbea-prometheus-operator-tls" seLinuxMountContext="" Nov 03 22:22:50.778793 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778686 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" volumeName="kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Nov 03 22:22:50.778793 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778699 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf4a34cf-5e64-407a-aaf7-0d211a6be6f6" volumeName="kubernetes.io/secret/bf4a34cf-5e64-407a-aaf7-0d211a6be6f6-cert" seLinuxMountContext="" Nov 03 22:22:50.778793 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778713 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6b58b28-fb4b-4635-a93a-0bd23222670b" volumeName="kubernetes.io/projected/c6b58b28-fb4b-4635-a93a-0bd23222670b-kube-api-access-wxz2n" seLinuxMountContext="" Nov 03 22:22:50.778793 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778739 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" volumeName="kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-web-config" seLinuxMountContext="" Nov 03 22:22:50.778793 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778752 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" volumeName="kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-metrics-client-certs" seLinuxMountContext="" Nov 03 22:22:50.778793 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778765 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e68894d-ed4f-4bc5-b36e-691e557883bd" volumeName="kubernetes.io/configmap/0e68894d-ed4f-4bc5-b36e-691e557883bd-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 03 22:22:50.778793 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778779 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" volumeName="kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Nov 03 22:22:50.779506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778807 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" volumeName="kubernetes.io/projected/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-kube-api-access-gqg7s" seLinuxMountContext="" Nov 03 22:22:50.779506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778820 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4425733-4528-4caa-99cb-e94072a00f78" volumeName="kubernetes.io/projected/d4425733-4528-4caa-99cb-e94072a00f78-kube-api-access-zlg2q" seLinuxMountContext="" Nov 03 22:22:50.779506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778832 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8f1371d-471f-496f-ac95-19eeb163c2c7" volumeName="kubernetes.io/empty-dir/d8f1371d-471f-496f-ac95-19eeb163c2c7-tmp" seLinuxMountContext="" Nov 03 22:22:50.779506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778846 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b2df441-74d4-435f-a56a-5eff3ec4575d" volumeName="kubernetes.io/configmap/1b2df441-74d4-435f-a56a-5eff3ec4575d-env-overrides" seLinuxMountContext="" Nov 03 22:22:50.779506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778890 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" volumeName="kubernetes.io/empty-dir/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-db" seLinuxMountContext="" Nov 03 22:22:50.779506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778909 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4425733-4528-4caa-99cb-e94072a00f78" volumeName="kubernetes.io/configmap/d4425733-4528-4caa-99cb-e94072a00f78-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 22:22:50.779506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778923 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6690a9b-44fa-4b1f-9523-173d8481dbea" volumeName="kubernetes.io/secret/e6690a9b-44fa-4b1f-9523-173d8481dbea-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 22:22:50.779506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778940 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="254d9855-203d-43f2-9c75-ed2eb84c4471" volumeName="kubernetes.io/configmap/254d9855-203d-43f2-9c75-ed2eb84c4471-config-volume" seLinuxMountContext="" Nov 03 22:22:50.779506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778953 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="254d9855-203d-43f2-9c75-ed2eb84c4471" volumeName="kubernetes.io/projected/254d9855-203d-43f2-9c75-ed2eb84c4471-kube-api-access-kfsqx" seLinuxMountContext="" Nov 03 22:22:50.779506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778966 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" volumeName="kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Nov 03 22:22:50.779506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778978 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" volumeName="kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 03 22:22:50.779506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.778992 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" volumeName="kubernetes.io/empty-dir/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-config-out" seLinuxMountContext="" Nov 03 22:22:50.779506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.779004 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" volumeName="kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-kube-rbac-proxy" seLinuxMountContext="" Nov 03 22:22:50.779506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.779034 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" volumeName="kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Nov 03 22:22:50.779506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.779059 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0ba09d3-25f2-422b-b5e2-357aabcaf81c" volumeName="kubernetes.io/configmap/b0ba09d3-25f2-422b-b5e2-357aabcaf81c-konnectivity-ca" seLinuxMountContext="" Nov 03 22:22:50.779506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.779075 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0ba09d3-25f2-422b-b5e2-357aabcaf81c" volumeName="kubernetes.io/secret/b0ba09d3-25f2-422b-b5e2-357aabcaf81c-agent-certs" seLinuxMountContext="" Nov 03 22:22:50.779506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.779087 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e68894d-ed4f-4bc5-b36e-691e557883bd" volumeName="kubernetes.io/empty-dir/0e68894d-ed4f-4bc5-b36e-691e557883bd-node-exporter-textfile" seLinuxMountContext="" Nov 03 22:22:50.779506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.779099 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b2df441-74d4-435f-a56a-5eff3ec4575d" volumeName="kubernetes.io/projected/1b2df441-74d4-435f-a56a-5eff3ec4575d-kube-api-access-s4n2l" seLinuxMountContext="" Nov 03 22:22:50.779506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.779111 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e68337a-9274-466d-a238-082fdf86f4cb" volumeName="kubernetes.io/projected/4e68337a-9274-466d-a238-082fdf86f4cb-kube-api-access-p9swx" seLinuxMountContext="" Nov 03 22:22:50.779506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.779125 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b2df441-74d4-435f-a56a-5eff3ec4575d" volumeName="kubernetes.io/configmap/1b2df441-74d4-435f-a56a-5eff3ec4575d-ovnkube-script-lib" seLinuxMountContext="" Nov 03 22:22:50.779506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.779137 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f63f886-379d-4636-8fb3-21b932a67831" volumeName="kubernetes.io/projected/3f63f886-379d-4636-8fb3-21b932a67831-kube-api-access-69c5d" seLinuxMountContext="" Nov 03 22:22:50.779506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.779149 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e68337a-9274-466d-a238-082fdf86f4cb" volumeName="kubernetes.io/configmap/4e68337a-9274-466d-a238-082fdf86f4cb-multus-daemon-config" seLinuxMountContext="" Nov 03 22:22:50.779506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.779162 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b40d10d-8dd1-46f6-b897-82780c392c84" volumeName="kubernetes.io/secret/5b40d10d-8dd1-46f6-b897-82780c392c84-monitoring-plugin-cert" seLinuxMountContext="" Nov 03 22:22:50.779506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.779186 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" volumeName="kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-thanos-prometheus-http-client-file" seLinuxMountContext="" Nov 03 22:22:50.779506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.779197 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf4a34cf-5e64-407a-aaf7-0d211a6be6f6" volumeName="kubernetes.io/projected/bf4a34cf-5e64-407a-aaf7-0d211a6be6f6-kube-api-access-4fhh6" seLinuxMountContext="" Nov 03 22:22:50.779506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.779209 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5730579-cca5-46be-9fbb-da9f7ae58d14" volumeName="kubernetes.io/secret/b5730579-cca5-46be-9fbb-da9f7ae58d14-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 03 22:22:50.779506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.779220 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="514a860a-f1e3-4714-8493-367b6b8b32a2" volumeName="kubernetes.io/configmap/514a860a-f1e3-4714-8493-367b6b8b32a2-serviceca" seLinuxMountContext="" Nov 03 22:22:50.779506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.779231 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" volumeName="kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-configmap-metrics-client-ca" seLinuxMountContext="" Nov 03 22:22:50.779506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.779241 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" volumeName="kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-prometheus-k8s-tls" seLinuxMountContext="" Nov 03 22:22:50.779506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.779252 5196 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c027d694-64cc-46a2-b4e7-49b117f696d3" volumeName="kubernetes.io/projected/c027d694-64cc-46a2-b4e7-49b117f696d3-kube-api-access-sqdbp" seLinuxMountContext="" Nov 03 22:22:50.779506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.779263 5196 reconstruct.go:97] "Volume reconstruction finished" Nov 03 22:22:50.779506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.779270 5196 reconciler.go:29] "Reconciler: start to sync state" Nov 03 22:22:50.805891 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.805834 5196 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 22:22:50.807365 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.807342 5196 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 22:22:50.807365 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.807365 5196 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 22:22:50.807528 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.807388 5196 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 22:22:50.807528 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:22:50.807427 5196 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 22:22:50.837505 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.837472 5196 generic.go:356] "Generic (PLEG): container finished" podID="9bb86a6441b6931e82229a352ca0e493" containerID="fca4dce96915bea16e8548eda8b5fbecb51a93451c1e8ae54245b67c3bb88da2" exitCode=0 Nov 03 22:22:50.840113 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.840079 5196 generic.go:356] "Generic (PLEG): container finished" podID="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" containerID="1bbcf6f7eeeec5392123a3b9318980d2d5b05a92d70b2c8aebf5d0ad0780c635" exitCode=0 Nov 03 22:22:50.846372 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.846340 5196 generic.go:356] "Generic (PLEG): container finished" podID="0e68894d-ed4f-4bc5-b36e-691e557883bd" containerID="43a9341e30c80bc69e6aadaeb0894b714daec9c722a189d10ebe8ed6f0cedd12" exitCode=0 Nov 03 22:22:50.860380 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.860359 5196 generic.go:356] "Generic (PLEG): container finished" podID="d4425733-4528-4caa-99cb-e94072a00f78" containerID="c92df619635d1a29a4909fe0ed9549ca9a6a64d8db7bbbfeff74404ae7ae1078" exitCode=0 Nov 03 22:22:50.860380 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.860378 5196 generic.go:356] "Generic (PLEG): container finished" podID="d4425733-4528-4caa-99cb-e94072a00f78" containerID="79927d94fab030ee9dba1fa91448445f8199385057ce029c1b0d8ed2b8818b2c" exitCode=0 Nov 03 22:22:50.860483 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.860387 5196 generic.go:356] "Generic (PLEG): container finished" podID="d4425733-4528-4caa-99cb-e94072a00f78" containerID="46697433e9e4a240671e4b7536319819b42ebf66560f3389b56c455073f17eaf" exitCode=0 Nov 03 22:22:50.860483 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.860392 5196 generic.go:356] "Generic (PLEG): container finished" podID="d4425733-4528-4caa-99cb-e94072a00f78" containerID="087a1ae1427d3f7988ab424a8b7cbfc16d90a36dd835fa2c14b2f31e87d20790" exitCode=0 Nov 03 22:22:50.860483 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.860397 5196 generic.go:356] "Generic (PLEG): container finished" podID="d4425733-4528-4caa-99cb-e94072a00f78" containerID="d0cb86b6ed4b868e4aec1dd10ee10b30b70bafe8d8e21a083a8bb80dcc39197f" exitCode=0 Nov 03 22:22:50.860483 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.860402 5196 generic.go:356] "Generic (PLEG): container finished" podID="d4425733-4528-4caa-99cb-e94072a00f78" containerID="03b241e13e64283f332bf3a4364cc62616f9f35012899a92dcf12f3144b16665" exitCode=0 Nov 03 22:22:50.863478 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.863462 5196 manager.go:324] Recovery completed Nov 03 22:22:50.876407 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:22:50.876359 5196 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-132.us-west-1.compute.internal\" not found" Nov 03 22:22:50.880510 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.880495 5196 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:22:50.882000 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.881983 5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:22:50.882061 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.882012 5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:22:50.882061 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.882022 5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 22:22:50.883970 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.883953 5196 cpu_manager.go:226] "Starting" policy="none" Nov 03 22:22:50.883970 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.883967 5196 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 22:22:50.884094 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.883972 5196 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:22:50.884094 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.883985 5196 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 22:22:50.884200 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.884124 5196 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 22:22:50.884200 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.884139 5196 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 22:22:50.884200 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.884162 5196 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 22:22:50.884200 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.884170 5196 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 22:22:50.884200 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.884177 5196 policy_none.go:50] "Start" Nov 03 22:22:50.884200 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.884189 5196 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 22:22:50.884200 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.884201 5196 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 22:22:50.884511 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.884372 5196 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 22:22:50.884511 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.884382 5196 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 22:22:50.884511 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.884391 5196 policy_none.go:44] "Start" Nov 03 22:22:50.884874 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.884829 5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:22:50.884941 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.884883 5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:22:50.884941 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.884898 5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 22:22:50.889164 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.889139 5196 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 22:22:50.892385 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.892365 5196 manager.go:341] "Starting Device Plugin manager" Nov 03 22:22:50.892467 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:22:50.892397 5196 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 22:22:50.892467 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.892407 5196 server.go:87] "Starting device plugin registration server" Nov 03 22:22:50.892726 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.892711 5196 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 22:22:50.892778 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.892728 5196 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 22:22:50.892825 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.892811 5196 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 22:22:50.892992 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.892935 5196 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 22:22:50.893051 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.893015 5196 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 22:22:50.893051 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.893022 5196 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 22:22:50.893165 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.893094 5196 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T22:22:50.892965624Z","UUID":"48ca9afc-9415-44a9-8303-6979fd1f9d84","Handler":null,"Name":"","Endpoint":""} Nov 03 22:22:50.894139 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.894122 5196 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 22:22:50.894209 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.894150 5196 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 22:22:50.894963 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:22:50.894607 5196 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 22:22:50.895041 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:22:50.894997 5196 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-132.us-west-1.compute.internal\" not found" Nov 03 22:22:50.908338 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.908300 5196 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-132.us-west-1.compute.internal"] Nov 03 22:22:50.908425 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.908415 5196 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:22:50.909154 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.909140 5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:22:50.909217 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.909166 5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:22:50.909217 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.909176 5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 22:22:50.909426 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.909416 5196 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:22:50.909569 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.909546 5196 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:22:50.910101 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.910088 5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:22:50.910160 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.910111 5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:22:50.910160 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.910120 5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 22:22:50.910160 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.910145 5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:22:50.910250 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.910167 5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:22:50.910250 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.910177 5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 22:22:50.910250 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:22:50.910231 5196 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-132.us-west-1.compute.internal\" not found" node="ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:22:50.910445 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.910420 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal" event={"ID":"9bb86a6441b6931e82229a352ca0e493","Type":"ContainerStarted","Data":"1b04557f1b6393af227bb855f4b3208bff8e8433af562489183ee6bbc4262199"} Nov 03 22:22:50.910476 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.910454 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal" event={"ID":"9bb86a6441b6931e82229a352ca0e493","Type":"ContainerDied","Data":"fca4dce96915bea16e8548eda8b5fbecb51a93451c1e8ae54245b67c3bb88da2"} Nov 03 22:22:50.910476 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.910469 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal" event={"ID":"9bb86a6441b6931e82229a352ca0e493","Type":"ContainerStarted","Data":"fd424519b7120717bc3041a2154a7338c2064d0aa6855d0a355f48237626b29a"} Nov 03 22:22:50.910538 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.910488 5196 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:22:50.910538 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.910499 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-132.us-west-1.compute.internal" event={"ID":"bd09a9d6123011e89fe0e21d9dfdc2d8","Type":"ContainerStarted","Data":"e510afdbebdf9514515f2a59d6d3c10d55632221b86784f23141b6582e4a1dcc"} Nov 03 22:22:50.910538 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.910511 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-132.us-west-1.compute.internal" event={"ID":"bd09a9d6123011e89fe0e21d9dfdc2d8","Type":"ContainerStarted","Data":"0d307ea1d74a23a33ed4b95e70c747f17dfebf86248ec7bbaf3c0d67eb8b95e9"} Nov 03 22:22:50.911186 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.911158 5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:22:50.911225 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.911200 5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:22:50.911225 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.911213 5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 22:22:50.911337 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:22:50.911324 5196 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-132.us-west-1.compute.internal\" not found" node="ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:22:50.923453 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.923438 5196 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:22:50.923575 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.923559 5196 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:22:50.970488 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.970462 5196 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:22:50.977714 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.977696 5196 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:22:50.992806 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.992780 5196 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:22:50.993814 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.993789 5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:22:50.993936 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.993825 5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:22:50.993936 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.993838 5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 22:22:50.993936 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:50.993886 5196 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:22:51.020220 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.020198 5196 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 22:22:51.071826 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.071796 5196 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:22:51.071993 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.071974 5196 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:22:51.072051 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:22:51.072014 5196 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:22:51.072051 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.072027 5196 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:22:51.121230 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.121205 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd09a9d6123011e89fe0e21d9dfdc2d8-config\") pod \"kube-apiserver-proxy-ip-10-0-143-132.us-west-1.compute.internal\" (UID: \"bd09a9d6123011e89fe0e21d9dfdc2d8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:22:51.121319 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.121243 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9bb86a6441b6931e82229a352ca0e493-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal\" (UID: \"9bb86a6441b6931e82229a352ca0e493\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:22:51.121319 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.121262 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bb86a6441b6931e82229a352ca0e493-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal\" (UID: \"9bb86a6441b6931e82229a352ca0e493\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:22:51.131112 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.131050 5196 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:22:51.131200 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.131117 5196 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:22:51.131734 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.131716 5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeNotReady" Nov 03 22:22:51.131819 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.131748 5196 setters.go:546] "Node became not ready" node="ip-10-0-143-132.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-03T22:22:51Z","lastTransitionTime":"2026-11-03T22:22:51Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 03 22:22:51.146819 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.146798 5196 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:22:51.146893 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:22:51.146835 5196 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-132.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:22:51.186007 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.185984 5196 nodeinfomanager.go:471] existing CSINode "ip-10-0-143-132.us-west-1.compute.internal" is owned by different node (oldNodeID="5a936c08-cb36-45bd-9a0e-890fcbad0a9a", newNodeID=""), cleaning up... Nov 03 22:22:51.221938 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.221920 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd09a9d6123011e89fe0e21d9dfdc2d8-config\") pod \"kube-apiserver-proxy-ip-10-0-143-132.us-west-1.compute.internal\" (UID: \"bd09a9d6123011e89fe0e21d9dfdc2d8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:22:51.222023 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.221979 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd09a9d6123011e89fe0e21d9dfdc2d8-config\") pod \"kube-apiserver-proxy-ip-10-0-143-132.us-west-1.compute.internal\" (UID: \"bd09a9d6123011e89fe0e21d9dfdc2d8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:22:51.222023 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.222009 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9bb86a6441b6931e82229a352ca0e493-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal\" (UID: \"9bb86a6441b6931e82229a352ca0e493\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:22:51.222132 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.222040 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bb86a6441b6931e82229a352ca0e493-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal\" (UID: \"9bb86a6441b6931e82229a352ca0e493\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:22:51.222132 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.222080 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bb86a6441b6931e82229a352ca0e493-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal\" (UID: \"9bb86a6441b6931e82229a352ca0e493\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:22:51.222132 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.222110 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9bb86a6441b6931e82229a352ca0e493-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal\" (UID: \"9bb86a6441b6931e82229a352ca0e493\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:22:51.754417 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.754389 5196 apiserver.go:52] "Watching apiserver" Nov 03 22:22:51.862589 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.862558 5196 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 22:22:51.863160 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.863138 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-dwf9b","openshift-dns/dns-default-6d49h","openshift-insights/insights-runtime-extractor-jwm6k","openshift-monitoring/node-exporter-zt4dd","openshift-console/downloads-6dd789c8b7-lqwmt","openshift-network-operator/iptables-alerter-p55z4","openshift-ingress-canary/ingress-canary-gw5gm","openshift-image-registry/node-ca-t5l5t","openshift-monitoring/prometheus-k8s-0","openshift-multus/multus-additional-cni-plugins-sq9mk","openshift-multus/multus-4gfds","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng","openshift-dns/node-resolver-k4h5g","openshift-multus/network-metrics-daemon-vqppk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal","openshift-network-diagnostics/network-check-target-jkjnj","kube-system/global-pull-secret-syncer-h8jxb","kube-system/konnectivity-agent-g6qjz","kube-system/kube-apiserver-proxy-ip-10-0-143-132.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-j9x58","openshift-monitoring/prometheus-operator-9cd89b74c-qb4qm","openshift-monitoring/monitoring-plugin-77f4f76594-hmr98"] Nov 03 22:22:51.865320 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.865292 5196 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:22:51.865963 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.865943 5196 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:22:51.878125 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.878102 5196 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 22:22:51.919424 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.919392 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 22:22:51.919608 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.919590 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:22:51.920234 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.920220 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 22:22:51.920751 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.920734 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 22:22:51.920837 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.920794 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 22:22:51.921016 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.921001 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 22:22:51.922075 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.922058 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dnsfs\"" Nov 03 22:22:51.922075 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.922073 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-dx5sz\"" Nov 03 22:22:51.922191 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.922081 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 22:22:51.922191 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.922089 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-px8pz\"" Nov 03 22:22:51.922191 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.922110 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 22:22:51.922191 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.922142 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xglhz\"" Nov 03 22:22:51.922191 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.922194 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dbbtb\"" Nov 03 22:22:51.924678 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.924665 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:22:51.926953 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.926934 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:51.927000 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.926962 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6690a9b-44fa-4b1f-9523-173d8481dbea-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qb4qm\" (UID: \"e6690a9b-44fa-4b1f-9523-173d8481dbea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qb4qm" Nov 03 22:22:51.927000 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.926982 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8f1371d-471f-496f-ac95-19eeb163c2c7-tmp\") pod \"downloads-6dd789c8b7-lqwmt\" (UID: \"d8f1371d-471f-496f-ac95-19eeb163c2c7\") " pod="openshift-console/downloads-6dd789c8b7-lqwmt" Nov 03 22:22:51.927078 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.927010 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-host-run-multus-certs\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:51.927078 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.927065 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-node-log\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:51.927155 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.927098 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:51.927191 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.927180 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8f1371d-471f-496f-ac95-19eeb163c2c7-tmp\") pod \"downloads-6dd789c8b7-lqwmt\" (UID: \"d8f1371d-471f-496f-ac95-19eeb163c2c7\") " pod="openshift-console/downloads-6dd789c8b7-lqwmt" Nov 03 22:22:51.927232 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.927177 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5vnj\" (UniqueName: \"kubernetes.io/projected/b5730579-cca5-46be-9fbb-da9f7ae58d14-kube-api-access-q5vnj\") pod \"insights-runtime-extractor-jwm6k\" (UID: \"b5730579-cca5-46be-9fbb-da9f7ae58d14\") " pod="openshift-insights/insights-runtime-extractor-jwm6k" Nov 03 22:22:51.927273 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.927234 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-tuned\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:51.927273 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.927259 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:51.927373 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.927285 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a90468d5-e140-4ae4-a1cb-c8f9dec579ff-iptables-alerter-script\") pod \"iptables-alerter-p55z4\" (UID: \"a90468d5-e140-4ae4-a1cb-c8f9dec579ff\") " pod="openshift-network-operator/iptables-alerter-p55z4" Nov 03 22:22:51.927373 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.927311 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-sys\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:51.927373 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.927330 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-tuned\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:51.927373 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.927337 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lrhw\" (UniqueName: \"kubernetes.io/projected/5b40d10d-8dd1-46f6-b897-82780c392c84-kube-api-access-6lrhw\") pod \"monitoring-plugin-77f4f76594-hmr98\" (UID: \"5b40d10d-8dd1-46f6-b897-82780c392c84\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hmr98" Nov 03 22:22:51.927558 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.927378 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b6154c5a-4925-4c74-8db4-0acdb525b743-original-pull-secret\") pod \"global-pull-secret-syncer-h8jxb\" (UID: \"b6154c5a-4925-4c74-8db4-0acdb525b743\") " pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:22:51.927558 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.927428 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/254d9855-203d-43f2-9c75-ed2eb84c4471-tmp-dir\") pod \"dns-default-6d49h\" (UID: \"254d9855-203d-43f2-9c75-ed2eb84c4471\") " pod="openshift-dns/dns-default-6d49h" Nov 03 22:22:51.927558 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.927463 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6690a9b-44fa-4b1f-9523-173d8481dbea-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qb4qm\" (UID: \"e6690a9b-44fa-4b1f-9523-173d8481dbea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qb4qm" Nov 03 22:22:51.927558 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.927495 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/514a860a-f1e3-4714-8493-367b6b8b32a2-serviceca\") pod \"node-ca-t5l5t\" (UID: \"514a860a-f1e3-4714-8493-367b6b8b32a2\") " pod="openshift-image-registry/node-ca-t5l5t" Nov 03 22:22:51.927558 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.927526 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b5730579-cca5-46be-9fbb-da9f7ae58d14-crio-socket\") pod \"insights-runtime-extractor-jwm6k\" (UID: \"b5730579-cca5-46be-9fbb-da9f7ae58d14\") " pod="openshift-insights/insights-runtime-extractor-jwm6k" Nov 03 22:22:51.927558 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.927542 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a90468d5-e140-4ae4-a1cb-c8f9dec579ff-iptables-alerter-script\") pod \"iptables-alerter-p55z4\" (UID: \"a90468d5-e140-4ae4-a1cb-c8f9dec579ff\") " pod="openshift-network-operator/iptables-alerter-p55z4" Nov 03 22:22:51.927558 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.927554 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:51.927810 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.927548 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/254d9855-203d-43f2-9c75-ed2eb84c4471-tmp-dir\") pod \"dns-default-6d49h\" (UID: \"254d9855-203d-43f2-9c75-ed2eb84c4471\") " pod="openshift-dns/dns-default-6d49h" Nov 03 22:22:51.927810 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.927632 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:51.927810 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.927662 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:51.927810 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.927704 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxz2n\" (UniqueName: \"kubernetes.io/projected/c6b58b28-fb4b-4635-a93a-0bd23222670b-kube-api-access-wxz2n\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:22:51.927810 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.927728 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3f63f886-379d-4636-8fb3-21b932a67831-hosts-file\") pod \"node-resolver-k4h5g\" (UID: \"3f63f886-379d-4636-8fb3-21b932a67831\") " pod="openshift-dns/node-resolver-k4h5g" Nov 03 22:22:51.927810 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.927750 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-sysconfig\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:51.927810 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.927777 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b6154c5a-4925-4c74-8db4-0acdb525b743-original-pull-secret\") pod \"global-pull-secret-syncer-h8jxb\" (UID: \"b6154c5a-4925-4c74-8db4-0acdb525b743\") " pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:22:51.927810 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.927801 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d4425733-4528-4caa-99cb-e94072a00f78-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sq9mk\" (UID: \"d4425733-4528-4caa-99cb-e94072a00f78\") " pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:22:51.928290 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.927903 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b2df441-74d4-435f-a56a-5eff3ec4575d-ovn-node-metrics-cert\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:51.928290 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.927935 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/514a860a-f1e3-4714-8493-367b6b8b32a2-host\") pod \"node-ca-t5l5t\" (UID: \"514a860a-f1e3-4714-8493-367b6b8b32a2\") " pod="openshift-image-registry/node-ca-t5l5t" Nov 03 22:22:51.928290 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.927962 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:51.928290 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.927993 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6b58b28-fb4b-4635-a93a-0bd23222670b-device-dir\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:22:51.928290 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.928022 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6b58b28-fb4b-4635-a93a-0bd23222670b-etc-selinux\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:22:51.928290 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.928059 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jqsx\" (UniqueName: \"kubernetes.io/projected/a90468d5-e140-4ae4-a1cb-c8f9dec579ff-kube-api-access-9jqsx\") pod \"iptables-alerter-p55z4\" (UID: \"a90468d5-e140-4ae4-a1cb-c8f9dec579ff\") " pod="openshift-network-operator/iptables-alerter-p55z4" Nov 03 22:22:51.928290 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.928088 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-run\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:51.928290 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.928117 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b5730579-cca5-46be-9fbb-da9f7ae58d14-data-volume\") pod \"insights-runtime-extractor-jwm6k\" (UID: \"b5730579-cca5-46be-9fbb-da9f7ae58d14\") " pod="openshift-insights/insights-runtime-extractor-jwm6k" Nov 03 22:22:51.928290 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.928221 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6b58b28-fb4b-4635-a93a-0bd23222670b-registration-dir\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:22:51.928290 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.928244 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-multus-conf-dir\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:51.928749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.928285 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b5730579-cca5-46be-9fbb-da9f7ae58d14-data-volume\") pod \"insights-runtime-extractor-jwm6k\" (UID: \"b5730579-cca5-46be-9fbb-da9f7ae58d14\") " pod="openshift-insights/insights-runtime-extractor-jwm6k" Nov 03 22:22:51.928749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.928277 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-run-openvswitch\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:51.928749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.928368 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:51.928749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.928420 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 22:22:51.928749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.928427 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-slash\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:51.928749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.928476 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-run-systemd\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:51.928749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.928513 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0e68894d-ed4f-4bc5-b36e-691e557883bd-root\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:51.928749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.928597 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0e68894d-ed4f-4bc5-b36e-691e557883bd-node-exporter-wtmp\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:51.928749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.928628 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-web-config\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:51.928749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.928652 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 22:22:51.928749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.928728 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 22:22:51.928749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.928737 5196 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:22:51.929182 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:22:51.928791 5196 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-132.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:22:51.929182 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.928801 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 22:22:51.929182 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.928653 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:51.929182 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.928845 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-config\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:51.929182 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.928732 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 22:22:51.929182 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.928935 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-var-lib-kubelet\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:51.929182 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.928965 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4e68337a-9274-466d-a238-082fdf86f4cb-multus-daemon-config\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:51.929182 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.928983 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-systemd-units\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:51.929182 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.929000 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-cni-bin\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:51.929182 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.929019 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5b40d10d-8dd1-46f6-b897-82780c392c84-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hmr98\" (UID: \"5b40d10d-8dd1-46f6-b897-82780c392c84\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hmr98" Nov 03 22:22:51.929182 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.929053 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:51.929182 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.929081 5196 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:22:51.929182 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:22:51.929105 5196 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-132.us-west-1.compute.internal" Nov 03 22:22:51.929182 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.929076 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cst2m\" (UniqueName: \"kubernetes.io/projected/e6690a9b-44fa-4b1f-9523-173d8481dbea-kube-api-access-cst2m\") pod \"prometheus-operator-9cd89b74c-qb4qm\" (UID: \"e6690a9b-44fa-4b1f-9523-173d8481dbea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qb4qm" Nov 03 22:22:51.929712 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.929199 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1b2df441-74d4-435f-a56a-5eff3ec4575d-env-overrides\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:51.929712 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.929157 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:51.929712 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.929232 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0e68894d-ed4f-4bc5-b36e-691e557883bd-node-exporter-accelerators-collector-config\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:51.929712 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.929264 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-host-var-lib-cni-bin\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:51.929712 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.929271 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4e68337a-9274-466d-a238-082fdf86f4cb-multus-daemon-config\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:51.929712 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.929322 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69c5d\" (UniqueName: \"kubernetes.io/projected/3f63f886-379d-4636-8fb3-21b932a67831-kube-api-access-69c5d\") pod \"node-resolver-k4h5g\" (UID: \"3f63f886-379d-4636-8fb3-21b932a67831\") " pod="openshift-dns/node-resolver-k4h5g" Nov 03 22:22:51.929712 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.929360 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5zc6\" (UniqueName: \"kubernetes.io/projected/29b82faa-1aa1-4760-a9f6-1cbeb9427094-kube-api-access-g5zc6\") pod \"network-check-target-jkjnj\" (UID: \"29b82faa-1aa1-4760-a9f6-1cbeb9427094\") " pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:22:51.929712 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.929388 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-kubelet\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:51.929712 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.929444 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-var-lib-openvswitch\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:51.929712 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.929472 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e68894d-ed4f-4bc5-b36e-691e557883bd-metrics-client-ca\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:51.929712 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.929524 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7qg9\" (UniqueName: \"kubernetes.io/projected/0e68894d-ed4f-4bc5-b36e-691e557883bd-kube-api-access-z7qg9\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:51.929712 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.929636 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-system-cni-dir\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:51.929712 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.929660 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:51.929712 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.929684 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:51.930108 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.929722 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/254d9855-203d-43f2-9c75-ed2eb84c4471-metrics-tls\") pod \"dns-default-6d49h\" (UID: \"254d9855-203d-43f2-9c75-ed2eb84c4471\") " pod="openshift-dns/dns-default-6d49h" Nov 03 22:22:51.930108 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.929749 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-log-socket\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:51.930108 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.929804 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0e68894d-ed4f-4bc5-b36e-691e557883bd-node-exporter-textfile\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:51.930108 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.929825 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-systemd\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:51.930108 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.929843 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e68337a-9274-466d-a238-082fdf86f4cb-cni-binary-copy\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:51.930108 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.929877 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d4425733-4528-4caa-99cb-e94072a00f78-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sq9mk\" (UID: \"d4425733-4528-4caa-99cb-e94072a00f78\") " pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:22:51.930108 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.929906 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-multus-cni-dir\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:51.930108 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.929908 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0e68894d-ed4f-4bc5-b36e-691e557883bd-node-exporter-textfile\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:51.930108 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.929954 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-kubernetes\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:51.930108 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.929976 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9swx\" (UniqueName: \"kubernetes.io/projected/4e68337a-9274-466d-a238-082fdf86f4cb-kube-api-access-p9swx\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:51.930108 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.930013 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0ba09d3-25f2-422b-b5e2-357aabcaf81c-konnectivity-ca\") pod \"konnectivity-agent-g6qjz\" (UID: \"b0ba09d3-25f2-422b-b5e2-357aabcaf81c\") " pod="kube-system/konnectivity-agent-g6qjz" Nov 03 22:22:51.930108 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.930035 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmjhw\" (UniqueName: \"kubernetes.io/projected/514a860a-f1e3-4714-8493-367b6b8b32a2-kube-api-access-mmjhw\") pod \"node-ca-t5l5t\" (UID: \"514a860a-f1e3-4714-8493-367b6b8b32a2\") " pod="openshift-image-registry/node-ca-t5l5t" Nov 03 22:22:51.930108 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.930053 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fhh6\" (UniqueName: \"kubernetes.io/projected/bf4a34cf-5e64-407a-aaf7-0d211a6be6f6-kube-api-access-4fhh6\") pod \"ingress-canary-gw5gm\" (UID: \"bf4a34cf-5e64-407a-aaf7-0d211a6be6f6\") " pod="openshift-ingress-canary/ingress-canary-gw5gm" Nov 03 22:22:51.930108 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.930067 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-sysctl-conf\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:51.930108 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.930094 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlg2q\" (UniqueName: \"kubernetes.io/projected/d4425733-4528-4caa-99cb-e94072a00f78-kube-api-access-zlg2q\") pod \"multus-additional-cni-plugins-sq9mk\" (UID: \"d4425733-4528-4caa-99cb-e94072a00f78\") " pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:22:51.930108 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.930114 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-cni-netd\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:51.930680 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.930146 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4n2l\" (UniqueName: \"kubernetes.io/projected/1b2df441-74d4-435f-a56a-5eff3ec4575d-kube-api-access-s4n2l\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:51.930680 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.930181 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-config-out\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:51.930680 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.930213 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0ba09d3-25f2-422b-b5e2-357aabcaf81c-konnectivity-ca\") pod \"konnectivity-agent-g6qjz\" (UID: \"b0ba09d3-25f2-422b-b5e2-357aabcaf81c\") " pod="kube-system/konnectivity-agent-g6qjz" Nov 03 22:22:51.930680 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.930227 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-sysctl-d\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:51.930680 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.930268 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-config-out\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:51.930680 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.930347 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-host\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:51.930680 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.930389 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d4425733-4528-4caa-99cb-e94072a00f78-cni-binary-copy\") pod \"multus-additional-cni-plugins-sq9mk\" (UID: \"d4425733-4528-4caa-99cb-e94072a00f78\") " pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:22:51.930680 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.930417 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b6154c5a-4925-4c74-8db4-0acdb525b743-dbus\") pod \"global-pull-secret-syncer-h8jxb\" (UID: \"b6154c5a-4925-4c74-8db4-0acdb525b743\") " pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:22:51.930680 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.930440 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a90468d5-e140-4ae4-a1cb-c8f9dec579ff-host-slash\") pod \"iptables-alerter-p55z4\" (UID: \"a90468d5-e140-4ae4-a1cb-c8f9dec579ff\") " pod="openshift-network-operator/iptables-alerter-p55z4" Nov 03 22:22:51.930680 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.930476 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf4a34cf-5e64-407a-aaf7-0d211a6be6f6-cert\") pod \"ingress-canary-gw5gm\" (UID: \"bf4a34cf-5e64-407a-aaf7-0d211a6be6f6\") " pod="openshift-ingress-canary/ingress-canary-gw5gm" Nov 03 22:22:51.930680 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.930522 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d4425733-4528-4caa-99cb-e94072a00f78-cnibin\") pod \"multus-additional-cni-plugins-sq9mk\" (UID: \"d4425733-4528-4caa-99cb-e94072a00f78\") " pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:22:51.930680 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.930591 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqdbp\" (UniqueName: \"kubernetes.io/projected/c027d694-64cc-46a2-b4e7-49b117f696d3-kube-api-access-sqdbp\") pod \"network-metrics-daemon-vqppk\" (UID: \"c027d694-64cc-46a2-b4e7-49b117f696d3\") " pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:22:51.931135 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.930637 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/254d9855-203d-43f2-9c75-ed2eb84c4471-config-volume\") pod \"dns-default-6d49h\" (UID: \"254d9855-203d-43f2-9c75-ed2eb84c4471\") " pod="openshift-dns/dns-default-6d49h" Nov 03 22:22:51.931135 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.931059 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqg7s\" (UniqueName: \"kubernetes.io/projected/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-kube-api-access-gqg7s\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:51.931135 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.931075 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b6154c5a-4925-4c74-8db4-0acdb525b743-kubelet-config\") pod \"global-pull-secret-syncer-h8jxb\" (UID: \"b6154c5a-4925-4c74-8db4-0acdb525b743\") " pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:22:51.931135 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.931091 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-lib-modules\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:51.931135 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.931107 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9988v\" (UniqueName: \"kubernetes.io/projected/1618fd8a-9a1b-4437-8559-86690d19cc9d-kube-api-access-9988v\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:51.931135 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.931123 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbjsn\" (UniqueName: \"kubernetes.io/projected/d8f1371d-471f-496f-ac95-19eeb163c2c7-kube-api-access-mbjsn\") pod \"downloads-6dd789c8b7-lqwmt\" (UID: \"d8f1371d-471f-496f-ac95-19eeb163c2c7\") " pod="openshift-console/downloads-6dd789c8b7-lqwmt" Nov 03 22:22:51.931407 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.931139 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-host-var-lib-kubelet\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:51.931407 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.931195 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-run-netns\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:51.931407 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.931281 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:51.931407 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.931311 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5730579-cca5-46be-9fbb-da9f7ae58d14-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jwm6k\" (UID: \"b5730579-cca5-46be-9fbb-da9f7ae58d14\") " pod="openshift-insights/insights-runtime-extractor-jwm6k" Nov 03 22:22:51.931407 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.931344 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0e68894d-ed4f-4bc5-b36e-691e557883bd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:51.931615 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.931409 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6b58b28-fb4b-4635-a93a-0bd23222670b-socket-dir\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:22:51.931615 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.931441 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f63f886-379d-4636-8fb3-21b932a67831-tmp-dir\") pod \"node-resolver-k4h5g\" (UID: \"3f63f886-379d-4636-8fb3-21b932a67831\") " pod="openshift-dns/node-resolver-k4h5g" Nov 03 22:22:51.931615 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.931494 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d4425733-4528-4caa-99cb-e94072a00f78-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sq9mk\" (UID: \"d4425733-4528-4caa-99cb-e94072a00f78\") " pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:22:51.931615 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.931537 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-host-run-k8s-cni-cncf-io\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:51.931615 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.931555 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:51.931615 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.931571 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6b58b28-fb4b-4635-a93a-0bd23222670b-sys-fs\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:22:51.931615 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.931587 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c027d694-64cc-46a2-b4e7-49b117f696d3-metrics-certs\") pod \"network-metrics-daemon-vqppk\" (UID: \"c027d694-64cc-46a2-b4e7-49b117f696d3\") " pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:22:51.931615 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.931589 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f63f886-379d-4636-8fb3-21b932a67831-tmp-dir\") pod \"node-resolver-k4h5g\" (UID: \"3f63f886-379d-4636-8fb3-21b932a67831\") " pod="openshift-dns/node-resolver-k4h5g" Nov 03 22:22:51.931960 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.931627 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-host-var-lib-cni-multus\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:51.931960 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.931656 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-etc-kubernetes\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:51.931960 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.931693 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:51.931960 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.931711 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6b58b28-fb4b-4635-a93a-0bd23222670b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:22:51.931960 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.931727 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-modprobe-d\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:51.931960 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.931741 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d4425733-4528-4caa-99cb-e94072a00f78-system-cni-dir\") pod \"multus-additional-cni-plugins-sq9mk\" (UID: \"d4425733-4528-4caa-99cb-e94072a00f78\") " pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:22:51.931960 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.931755 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-host-run-netns\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:51.931960 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.931795 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-etc-openvswitch\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:51.931960 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.931813 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-run-ovn\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:51.931960 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.931830 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1b2df441-74d4-435f-a56a-5eff3ec4575d-ovnkube-config\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:51.931960 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.931849 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b5730579-cca5-46be-9fbb-da9f7ae58d14-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jwm6k\" (UID: \"b5730579-cca5-46be-9fbb-da9f7ae58d14\") " pod="openshift-insights/insights-runtime-extractor-jwm6k" Nov 03 22:22:51.931960 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.931906 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0e68894d-ed4f-4bc5-b36e-691e557883bd-node-exporter-tls\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:51.931960 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.931945 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-hostroot\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:51.932435 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.931972 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-run-ovn-kubernetes\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:51.932435 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.931999 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e68894d-ed4f-4bc5-b36e-691e557883bd-sys\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:51.932435 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.932031 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:51.932435 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.932067 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e6690a9b-44fa-4b1f-9523-173d8481dbea-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qb4qm\" (UID: \"e6690a9b-44fa-4b1f-9523-173d8481dbea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qb4qm" Nov 03 22:22:51.932435 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.932099 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-cnibin\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:51.932435 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.932136 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-multus-socket-dir-parent\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:51.932435 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.932162 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0ba09d3-25f2-422b-b5e2-357aabcaf81c-agent-certs\") pod \"konnectivity-agent-g6qjz\" (UID: \"b0ba09d3-25f2-422b-b5e2-357aabcaf81c\") " pod="kube-system/konnectivity-agent-g6qjz" Nov 03 22:22:51.932435 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.932189 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1b2df441-74d4-435f-a56a-5eff3ec4575d-ovnkube-script-lib\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:51.932435 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.932207 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfsqx\" (UniqueName: \"kubernetes.io/projected/254d9855-203d-43f2-9c75-ed2eb84c4471-kube-api-access-kfsqx\") pod \"dns-default-6d49h\" (UID: \"254d9855-203d-43f2-9c75-ed2eb84c4471\") " pod="openshift-dns/dns-default-6d49h" Nov 03 22:22:51.932435 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.932232 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1618fd8a-9a1b-4437-8559-86690d19cc9d-tmp\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:51.932435 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.932284 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1618fd8a-9a1b-4437-8559-86690d19cc9d-tmp\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:51.932435 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.932309 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:51.932435 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.932319 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0ba09d3-25f2-422b-b5e2-357aabcaf81c-agent-certs\") pod \"konnectivity-agent-g6qjz\" (UID: \"b0ba09d3-25f2-422b-b5e2-357aabcaf81c\") " pod="kube-system/konnectivity-agent-g6qjz" Nov 03 22:22:51.938869 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.938843 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/514a860a-f1e3-4714-8493-367b6b8b32a2-serviceca\") pod \"node-ca-t5l5t\" (UID: \"514a860a-f1e3-4714-8493-367b6b8b32a2\") " pod="openshift-image-registry/node-ca-t5l5t" Nov 03 22:22:51.991935 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.991904 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jqsx\" (UniqueName: \"kubernetes.io/projected/a90468d5-e140-4ae4-a1cb-c8f9dec579ff-kube-api-access-9jqsx\") pod \"iptables-alerter-p55z4\" (UID: \"a90468d5-e140-4ae4-a1cb-c8f9dec579ff\") " pod="openshift-network-operator/iptables-alerter-p55z4" Nov 03 22:22:51.995588 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.995568 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9988v\" (UniqueName: \"kubernetes.io/projected/1618fd8a-9a1b-4437-8559-86690d19cc9d-kube-api-access-9988v\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:51.997138 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:51.997117 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmjhw\" (UniqueName: \"kubernetes.io/projected/514a860a-f1e3-4714-8493-367b6b8b32a2-kube-api-access-mmjhw\") pod \"node-ca-t5l5t\" (UID: \"514a860a-f1e3-4714-8493-367b6b8b32a2\") " pod="openshift-image-registry/node-ca-t5l5t" Nov 03 22:22:52.014376 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.014321 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 22:22:52.014686 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.014663 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 22:22:52.014744 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.014695 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 22:22:52.014744 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.014694 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 22:22:52.014744 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.014706 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 22:22:52.014886 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.014735 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 22:22:52.014886 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.014780 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 22:22:52.014886 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.014885 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 22:22:52.015015 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.014931 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 22:22:52.015015 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.014996 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 22:22:52.015015 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.015001 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 22:22:52.015168 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.015057 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 22:22:52.015168 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.015084 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 22:22:52.015168 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.015117 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 22:22:52.015168 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.015125 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 22:22:52.015414 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.015394 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 22:22:52.015522 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.015457 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 22:22:52.015522 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.015459 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 22:22:52.015522 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.015459 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 22:22:52.015522 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.015457 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 22:22:52.015522 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.015517 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 22:22:52.015696 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.015546 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 22:22:52.015696 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.015590 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 22:22:52.016151 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.016128 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 22:22:52.016255 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.016239 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 22:22:52.016307 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.016295 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 22:22:52.016354 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.016343 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 22:22:52.017082 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.017063 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 22:22:52.017288 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.017271 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 22:22:52.017420 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.017406 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 22:22:52.017916 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.017896 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6690a9b-44fa-4b1f-9523-173d8481dbea-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qb4qm\" (UID: \"e6690a9b-44fa-4b1f-9523-173d8481dbea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qb4qm" Nov 03 22:22:52.017992 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.017934 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7qg9\" (UniqueName: \"kubernetes.io/projected/0e68894d-ed4f-4bc5-b36e-691e557883bd-kube-api-access-z7qg9\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:52.018096 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.018077 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:52.018557 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.018533 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlg2q\" (UniqueName: \"kubernetes.io/projected/d4425733-4528-4caa-99cb-e94072a00f78-kube-api-access-zlg2q\") pod \"multus-additional-cni-plugins-sq9mk\" (UID: \"d4425733-4528-4caa-99cb-e94072a00f78\") " pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:22:52.018739 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.018720 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:52.019211 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.019188 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fhh6\" (UniqueName: \"kubernetes.io/projected/bf4a34cf-5e64-407a-aaf7-0d211a6be6f6-kube-api-access-4fhh6\") pod \"ingress-canary-gw5gm\" (UID: \"bf4a34cf-5e64-407a-aaf7-0d211a6be6f6\") " pod="openshift-ingress-canary/ingress-canary-gw5gm" Nov 03 22:22:52.020579 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.020555 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1b2df441-74d4-435f-a56a-5eff3ec4575d-env-overrides\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:52.020689 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.020666 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d4425733-4528-4caa-99cb-e94072a00f78-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sq9mk\" (UID: \"d4425733-4528-4caa-99cb-e94072a00f78\") " pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:22:52.020769 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.020587 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e68337a-9274-466d-a238-082fdf86f4cb-cni-binary-copy\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:52.020769 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.020609 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:52.020769 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.020627 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e68894d-ed4f-4bc5-b36e-691e557883bd-metrics-client-ca\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:52.020769 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.020577 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0e68894d-ed4f-4bc5-b36e-691e557883bd-node-exporter-accelerators-collector-config\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:52.020977 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.020846 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d4425733-4528-4caa-99cb-e94072a00f78-cni-binary-copy\") pod \"multus-additional-cni-plugins-sq9mk\" (UID: \"d4425733-4528-4caa-99cb-e94072a00f78\") " pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:22:52.020977 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.020905 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/254d9855-203d-43f2-9c75-ed2eb84c4471-config-volume\") pod \"dns-default-6d49h\" (UID: \"254d9855-203d-43f2-9c75-ed2eb84c4471\") " pod="openshift-dns/dns-default-6d49h" Nov 03 22:22:52.021098 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.021074 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4n2l\" (UniqueName: \"kubernetes.io/projected/1b2df441-74d4-435f-a56a-5eff3ec4575d-kube-api-access-s4n2l\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:52.021166 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.021139 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqg7s\" (UniqueName: \"kubernetes.io/projected/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-kube-api-access-gqg7s\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:52.021228 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.021212 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cst2m\" (UniqueName: \"kubernetes.io/projected/e6690a9b-44fa-4b1f-9523-173d8481dbea-kube-api-access-cst2m\") pod \"prometheus-operator-9cd89b74c-qb4qm\" (UID: \"e6690a9b-44fa-4b1f-9523-173d8481dbea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qb4qm" Nov 03 22:22:52.021261 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.021217 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5zc6\" (UniqueName: \"kubernetes.io/projected/29b82faa-1aa1-4760-a9f6-1cbeb9427094-kube-api-access-g5zc6\") pod \"network-check-target-jkjnj\" (UID: \"29b82faa-1aa1-4760-a9f6-1cbeb9427094\") " pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:22:52.021687 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.021671 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxz2n\" (UniqueName: \"kubernetes.io/projected/c6b58b28-fb4b-4635-a93a-0bd23222670b-kube-api-access-wxz2n\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:22:52.021849 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.021834 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d4425733-4528-4caa-99cb-e94072a00f78-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sq9mk\" (UID: \"d4425733-4528-4caa-99cb-e94072a00f78\") " pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:22:52.022041 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.021994 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:52.022171 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.022152 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5vnj\" (UniqueName: \"kubernetes.io/projected/b5730579-cca5-46be-9fbb-da9f7ae58d14-kube-api-access-q5vnj\") pod \"insights-runtime-extractor-jwm6k\" (UID: \"b5730579-cca5-46be-9fbb-da9f7ae58d14\") " pod="openshift-insights/insights-runtime-extractor-jwm6k" Nov 03 22:22:52.022206 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.022195 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b5730579-cca5-46be-9fbb-da9f7ae58d14-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jwm6k\" (UID: \"b5730579-cca5-46be-9fbb-da9f7ae58d14\") " pod="openshift-insights/insights-runtime-extractor-jwm6k" Nov 03 22:22:52.022254 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.022236 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lrhw\" (UniqueName: \"kubernetes.io/projected/5b40d10d-8dd1-46f6-b897-82780c392c84-kube-api-access-6lrhw\") pod \"monitoring-plugin-77f4f76594-hmr98\" (UID: \"5b40d10d-8dd1-46f6-b897-82780c392c84\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hmr98" Nov 03 22:22:52.022291 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.022250 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1b2df441-74d4-435f-a56a-5eff3ec4575d-ovnkube-config\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:52.023359 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.023344 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1b2df441-74d4-435f-a56a-5eff3ec4575d-ovnkube-script-lib\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:52.023392 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.023348 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:52.025816 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.025797 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqdbp\" (UniqueName: \"kubernetes.io/projected/c027d694-64cc-46a2-b4e7-49b117f696d3-kube-api-access-sqdbp\") pod \"network-metrics-daemon-vqppk\" (UID: \"c027d694-64cc-46a2-b4e7-49b117f696d3\") " pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:22:52.025851 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.025832 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfsqx\" (UniqueName: \"kubernetes.io/projected/254d9855-203d-43f2-9c75-ed2eb84c4471-kube-api-access-kfsqx\") pod \"dns-default-6d49h\" (UID: \"254d9855-203d-43f2-9c75-ed2eb84c4471\") " pod="openshift-dns/dns-default-6d49h" Nov 03 22:22:52.025967 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.025953 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9swx\" (UniqueName: \"kubernetes.io/projected/4e68337a-9274-466d-a238-082fdf86f4cb-kube-api-access-p9swx\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:52.026007 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.025989 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69c5d\" (UniqueName: \"kubernetes.io/projected/3f63f886-379d-4636-8fb3-21b932a67831-kube-api-access-69c5d\") pod \"node-resolver-k4h5g\" (UID: \"3f63f886-379d-4636-8fb3-21b932a67831\") " pod="openshift-dns/node-resolver-k4h5g" Nov 03 22:22:52.026064 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.026050 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbjsn\" (UniqueName: \"kubernetes.io/projected/d8f1371d-471f-496f-ac95-19eeb163c2c7-kube-api-access-mbjsn\") pod \"downloads-6dd789c8b7-lqwmt\" (UID: \"d8f1371d-471f-496f-ac95-19eeb163c2c7\") " pod="openshift-console/downloads-6dd789c8b7-lqwmt" Nov 03 22:22:52.033324 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033309 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-sysctl-conf\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:52.033369 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033331 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-cni-netd\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:52.033369 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033347 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-sysctl-d\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:52.033369 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033360 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-host\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:52.033488 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033374 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b6154c5a-4925-4c74-8db4-0acdb525b743-dbus\") pod \"global-pull-secret-syncer-h8jxb\" (UID: \"b6154c5a-4925-4c74-8db4-0acdb525b743\") " pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:22:52.033488 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033397 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a90468d5-e140-4ae4-a1cb-c8f9dec579ff-host-slash\") pod \"iptables-alerter-p55z4\" (UID: \"a90468d5-e140-4ae4-a1cb-c8f9dec579ff\") " pod="openshift-network-operator/iptables-alerter-p55z4" Nov 03 22:22:52.033488 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033422 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d4425733-4528-4caa-99cb-e94072a00f78-cnibin\") pod \"multus-additional-cni-plugins-sq9mk\" (UID: \"d4425733-4528-4caa-99cb-e94072a00f78\") " pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:22:52.033488 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033445 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-host\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:52.033488 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033461 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d4425733-4528-4caa-99cb-e94072a00f78-cnibin\") pod \"multus-additional-cni-plugins-sq9mk\" (UID: \"d4425733-4528-4caa-99cb-e94072a00f78\") " pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:22:52.033488 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033421 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-cni-netd\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:52.033488 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033469 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b6154c5a-4925-4c74-8db4-0acdb525b743-dbus\") pod \"global-pull-secret-syncer-h8jxb\" (UID: \"b6154c5a-4925-4c74-8db4-0acdb525b743\") " pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:22:52.033488 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033468 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-sysctl-conf\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:52.033488 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033450 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b6154c5a-4925-4c74-8db4-0acdb525b743-kubelet-config\") pod \"global-pull-secret-syncer-h8jxb\" (UID: \"b6154c5a-4925-4c74-8db4-0acdb525b743\") " pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:22:52.033843 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033499 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-sysctl-d\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:52.033843 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033515 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-lib-modules\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:52.033843 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033495 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b6154c5a-4925-4c74-8db4-0acdb525b743-kubelet-config\") pod \"global-pull-secret-syncer-h8jxb\" (UID: \"b6154c5a-4925-4c74-8db4-0acdb525b743\") " pod="kube-system/global-pull-secret-syncer-h8jxb" Nov 03 22:22:52.033843 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033486 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a90468d5-e140-4ae4-a1cb-c8f9dec579ff-host-slash\") pod \"iptables-alerter-p55z4\" (UID: \"a90468d5-e140-4ae4-a1cb-c8f9dec579ff\") " pod="openshift-network-operator/iptables-alerter-p55z4" Nov 03 22:22:52.033843 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033542 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-host-var-lib-kubelet\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:52.033843 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033572 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-run-netns\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:52.033843 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033589 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6b58b28-fb4b-4635-a93a-0bd23222670b-socket-dir\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:22:52.033843 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033605 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-host-run-k8s-cni-cncf-io\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:52.033843 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033621 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6b58b28-fb4b-4635-a93a-0bd23222670b-sys-fs\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:22:52.033843 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033589 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-host-var-lib-kubelet\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:52.033843 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033634 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-run-netns\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:52.033843 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033655 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-host-run-k8s-cni-cncf-io\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:52.033843 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033662 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-lib-modules\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:52.033843 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033672 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6b58b28-fb4b-4635-a93a-0bd23222670b-sys-fs\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:22:52.033843 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033668 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-host-var-lib-cni-multus\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:52.033843 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033655 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6b58b28-fb4b-4635-a93a-0bd23222670b-socket-dir\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:22:52.033843 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033692 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-host-var-lib-cni-multus\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:52.033843 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033698 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-etc-kubernetes\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:52.033843 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033715 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-etc-kubernetes\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:52.033843 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033728 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6b58b28-fb4b-4635-a93a-0bd23222670b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:22:52.033843 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033761 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-modprobe-d\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:52.033843 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033775 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d4425733-4528-4caa-99cb-e94072a00f78-system-cni-dir\") pod \"multus-additional-cni-plugins-sq9mk\" (UID: \"d4425733-4528-4caa-99cb-e94072a00f78\") " pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:22:52.033843 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033782 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6b58b28-fb4b-4635-a93a-0bd23222670b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:22:52.033843 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033795 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-host-run-netns\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:52.033843 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033818 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-etc-openvswitch\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:52.033843 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033823 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d4425733-4528-4caa-99cb-e94072a00f78-system-cni-dir\") pod \"multus-additional-cni-plugins-sq9mk\" (UID: \"d4425733-4528-4caa-99cb-e94072a00f78\") " pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:22:52.033843 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033841 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-run-ovn\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:52.033843 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033847 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-modprobe-d\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033884 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-host-run-netns\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033886 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-hostroot\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033903 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-etc-openvswitch\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033916 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-hostroot\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033916 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-run-ovn-kubernetes\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033946 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-run-ovn-kubernetes\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033954 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-run-ovn\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.033981 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e68894d-ed4f-4bc5-b36e-691e557883bd-sys\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034004 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e68894d-ed4f-4bc5-b36e-691e557883bd-sys\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034005 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-cnibin\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034024 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-multus-socket-dir-parent\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034044 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-host-run-multus-certs\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034058 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-node-log\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034073 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034078 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-multus-socket-dir-parent\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034089 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-sys\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034109 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-cnibin\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034111 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b5730579-cca5-46be-9fbb-da9f7ae58d14-crio-socket\") pod \"insights-runtime-extractor-jwm6k\" (UID: \"b5730579-cca5-46be-9fbb-da9f7ae58d14\") " pod="openshift-insights/insights-runtime-extractor-jwm6k" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034117 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-host-run-multus-certs\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034118 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-node-log\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034129 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034145 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3f63f886-379d-4636-8fb3-21b932a67831-hosts-file\") pod \"node-resolver-k4h5g\" (UID: \"3f63f886-379d-4636-8fb3-21b932a67831\") " pod="openshift-dns/node-resolver-k4h5g" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034150 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-sys\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034153 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b5730579-cca5-46be-9fbb-da9f7ae58d14-crio-socket\") pod \"insights-runtime-extractor-jwm6k\" (UID: \"b5730579-cca5-46be-9fbb-da9f7ae58d14\") " pod="openshift-insights/insights-runtime-extractor-jwm6k" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034187 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3f63f886-379d-4636-8fb3-21b932a67831-hosts-file\") pod \"node-resolver-k4h5g\" (UID: \"3f63f886-379d-4636-8fb3-21b932a67831\") " pod="openshift-dns/node-resolver-k4h5g" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034201 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-sysconfig\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034236 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d4425733-4528-4caa-99cb-e94072a00f78-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sq9mk\" (UID: \"d4425733-4528-4caa-99cb-e94072a00f78\") " pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034256 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/514a860a-f1e3-4714-8493-367b6b8b32a2-host\") pod \"node-ca-t5l5t\" (UID: \"514a860a-f1e3-4714-8493-367b6b8b32a2\") " pod="openshift-image-registry/node-ca-t5l5t" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034278 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6b58b28-fb4b-4635-a93a-0bd23222670b-device-dir\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034301 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d4425733-4528-4caa-99cb-e94072a00f78-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sq9mk\" (UID: \"d4425733-4528-4caa-99cb-e94072a00f78\") " pod="openshift-multus/multus-additional-cni-plugins-sq9mk" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034310 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/514a860a-f1e3-4714-8493-367b6b8b32a2-host\") pod \"node-ca-t5l5t\" (UID: \"514a860a-f1e3-4714-8493-367b6b8b32a2\") " pod="openshift-image-registry/node-ca-t5l5t" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034305 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6b58b28-fb4b-4635-a93a-0bd23222670b-device-dir\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034338 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6b58b28-fb4b-4635-a93a-0bd23222670b-etc-selinux\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034302 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6b58b28-fb4b-4635-a93a-0bd23222670b-etc-selinux\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034363 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-run\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034382 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6b58b28-fb4b-4635-a93a-0bd23222670b-registration-dir\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034282 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-sysconfig\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034411 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-multus-conf-dir\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034422 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-run\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034436 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-run-openvswitch\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034444 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-multus-conf-dir\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034452 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-slash\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034444 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6b58b28-fb4b-4635-a93a-0bd23222670b-registration-dir\") pod \"aws-ebs-csi-driver-node-79vng\" (UID: \"c6b58b28-fb4b-4635-a93a-0bd23222670b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-79vng" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034467 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-run-systemd\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034476 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-run-openvswitch\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034479 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-slash\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034500 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0e68894d-ed4f-4bc5-b36e-691e557883bd-root\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034500 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-run-systemd\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034482 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0e68894d-ed4f-4bc5-b36e-691e557883bd-root\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034535 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0e68894d-ed4f-4bc5-b36e-691e557883bd-node-exporter-wtmp\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034563 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-var-lib-kubelet\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034586 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0e68894d-ed4f-4bc5-b36e-691e557883bd-node-exporter-wtmp\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034591 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-systemd-units\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034614 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-var-lib-kubelet\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034617 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-cni-bin\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034647 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-host-var-lib-cni-bin\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034688 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-cni-bin\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034695 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-systemd-units\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034694 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-host-var-lib-cni-bin\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034715 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-kubelet\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034746 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-host-kubelet\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034750 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-var-lib-openvswitch\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034775 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-system-cni-dir\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034791 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-log-socket\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034805 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-systemd\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034823 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-system-cni-dir\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034825 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-multus-cni-dir\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034848 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-log-socket\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034872 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-kubernetes\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034881 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-systemd\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034892 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b2df441-74d4-435f-a56a-5eff3ec4575d-var-lib-openvswitch\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034898 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1618fd8a-9a1b-4437-8559-86690d19cc9d-etc-kubernetes\") pod \"tuned-j9x58\" (UID: \"1618fd8a-9a1b-4437-8559-86690d19cc9d\") " pod="openshift-cluster-node-tuning-operator/tuned-j9x58" Nov 03 22:22:52.034995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.034918 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e68337a-9274-466d-a238-082fdf86f4cb-multus-cni-dir\") pod \"multus-4gfds\" (UID: \"4e68337a-9274-466d-a238-082fdf86f4cb\") " pod="openshift-multus/multus-4gfds" Nov 03 22:22:52.051438 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.051412 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 22:22:52.051623 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.051604 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 22:22:52.058712 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.058693 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:52.062621 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.062604 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:52.065435 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.065417 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 22:22:52.065523 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.065478 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-9gf2r\"" Nov 03 22:22:52.065523 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.065495 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-q82l2\"" Nov 03 22:22:52.065523 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.065494 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-b2scz\"" Nov 03 22:22:52.065715 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.065701 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 03 22:22:52.065768 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.065719 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 22:22:52.065768 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.065747 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 22:22:52.065842 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.065758 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 22:22:52.065904 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.065880 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vc5c5\"" Nov 03 22:22:52.065904 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.065893 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-94lr2\"" Nov 03 22:22:52.065988 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.065913 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 22:22:52.066167 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.066157 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 22:22:52.066217 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.066204 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-5ghk8\"" Nov 03 22:22:52.066262 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.066252 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 22:22:52.066309 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.066279 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 22:22:52.066309 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.066285 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 22:22:52.066309 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.066288 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-j25tw\"" Nov 03 22:22:52.066454 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.066327 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 22:22:52.066454 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.066375 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 22:22:52.066454 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.066390 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 03 22:22:52.066454 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.066423 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-rrf2w\"" Nov 03 22:22:52.066454 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.066430 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-q5zlz\"" Nov 03 22:22:52.066664 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.066459 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 22:22:52.066664 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.066464 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 22:22:52.066664 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.066479 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zhswt\"" Nov 03 22:22:52.066664 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.066561 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-26t3ivs4b9nmd\"" Nov 03 22:22:52.066664 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.066578 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 03 22:22:52.066664 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.066609 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-m2wh6\"" Nov 03 22:22:52.066664 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.066613 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-2kpmz\"" Nov 03 22:22:52.066664 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.066630 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 22:22:52.066664 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.066650 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 22:22:52.067014 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.066727 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 22:22:52.067014 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.066760 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ft6gs\"" Nov 03 22:22:52.067014 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.066769 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-t9khm\"" Nov 03 22:22:52.067014 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.066768 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 22:22:52.067014 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.066782 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mx976\"" Nov 03 22:22:52.067424 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.067410 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:52.067881 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.067850 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:52.067915 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.067851 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6690a9b-44fa-4b1f-9523-173d8481dbea-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qb4qm\" (UID: \"e6690a9b-44fa-4b1f-9523-173d8481dbea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qb4qm" Nov 03 22:22:52.069075 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.069056 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b2df441-74d4-435f-a56a-5eff3ec4575d-ovn-node-metrics-cert\") pod \"ovnkube-node-dwf9b\" (UID: \"1b2df441-74d4-435f-a56a-5eff3ec4575d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:52.069210 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.069081 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-web-config\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:52.069210 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.069144 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:52.069210 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.069171 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:52.069210 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.069190 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:52.069379 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.069265 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5b40d10d-8dd1-46f6-b897-82780c392c84-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hmr98\" (UID: \"5b40d10d-8dd1-46f6-b897-82780c392c84\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hmr98" Nov 03 22:22:52.069379 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.069321 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-config\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:52.070312 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.070290 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:52.070378 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.070317 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/254d9855-203d-43f2-9c75-ed2eb84c4471-metrics-tls\") pod \"dns-default-6d49h\" (UID: \"254d9855-203d-43f2-9c75-ed2eb84c4471\") " pod="openshift-dns/dns-default-6d49h" Nov 03 22:22:52.071464 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.071450 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf4a34cf-5e64-407a-aaf7-0d211a6be6f6-cert\") pod \"ingress-canary-gw5gm\" (UID: \"bf4a34cf-5e64-407a-aaf7-0d211a6be6f6\") " pod="openshift-ingress-canary/ingress-canary-gw5gm" Nov 03 22:22:52.071604 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.071587 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b5730579-cca5-46be-9fbb-da9f7ae58d14-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jwm6k\" (UID: \"b5730579-cca5-46be-9fbb-da9f7ae58d14\") " pod="openshift-insights/insights-runtime-extractor-jwm6k" Nov 03 22:22:52.071671 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.071654 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0e68894d-ed4f-4bc5-b36e-691e557883bd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:52.071755 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.071743 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:52.071806 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.071793 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c027d694-64cc-46a2-b4e7-49b117f696d3-metrics-certs\") pod \"network-metrics-daemon-vqppk\" (UID: \"c027d694-64cc-46a2-b4e7-49b117f696d3\") " pod="openshift-multus/network-metrics-daemon-vqppk" Nov 03 22:22:52.072592 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.072578 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e6690a9b-44fa-4b1f-9523-173d8481dbea-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qb4qm\" (UID: \"e6690a9b-44fa-4b1f-9523-173d8481dbea\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qb4qm" Nov 03 22:22:52.072636 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.072584 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0e68894d-ed4f-4bc5-b36e-691e557883bd-node-exporter-tls\") pod \"node-exporter-zt4dd\" (UID: \"0e68894d-ed4f-4bc5-b36e-691e557883bd\") " pod="openshift-monitoring/node-exporter-zt4dd" Nov 03 22:22:52.072694 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.072680 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:22:52.175564 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.175546 5196 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:22:52.388550 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.388512 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:52.549171 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.549136 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:52.926147 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:52.926117 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6d49h" Nov 03 22:22:53.402264 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:53.402233 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:53.788756 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:53.788630 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:53.929289 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:53.929242 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:54.262745 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:54.262705 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:55.168187 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:55.166959 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:55.168187 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:55.167224 5196 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 22:22:55.665329 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:55.665293 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hmr98" Nov 03 22:22:55.670585 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:55.670559 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hmr98" Nov 03 22:22:55.921658 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:22:55.921582 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:22:55.938989 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:55.938957 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:56.102707 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:56.102676 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:56.242914 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:56.242801 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:56.887568 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:56.887534 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a83dfb48-acb6-4cbc-bfc0-61103dafa90f","Type":"ContainerStarted","Data":"88d04cbf2b22ea9f5917f15b5a4537e084b2380e5624151c4df5c28bdab98e5b"} Nov 03 22:22:56.887720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:56.887607 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a83dfb48-acb6-4cbc-bfc0-61103dafa90f","Type":"ContainerStarted","Data":"5c86517d0dcc6eb524b065e086aaed1fd2f51845d0f21e6050f636cec774f035"} Nov 03 22:22:56.927781 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:56.927733 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:57.077334 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:57.077298 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:58.155148 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:58.155106 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dwf9b" Nov 03 22:22:58.268618 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:58.268534 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:58.411320 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:58.411248 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:22:58.899561 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:58.899522 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a83dfb48-acb6-4cbc-bfc0-61103dafa90f","Type":"ContainerStarted","Data":"e0a99d3e39e5ffd83e6e7cf7f06f4523d1cd5da6801c34a306cad57614dddde6"} Nov 03 22:22:58.899684 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:58.899588 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a83dfb48-acb6-4cbc-bfc0-61103dafa90f","Type":"ContainerStarted","Data":"271eaab0ca06566c95c2df8668bfc62f7c850b3ba50d6fe8c122c8f8313d4eeb"} Nov 03 22:22:58.899684 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:58.899621 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a83dfb48-acb6-4cbc-bfc0-61103dafa90f","Type":"ContainerStarted","Data":"658e3f5c349132244c9d37d07235fdce87ee0c1108b4deb5747b830a68d5b892"} Nov 03 22:22:58.899684 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:22:58.899658 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a83dfb48-acb6-4cbc-bfc0-61103dafa90f","Type":"ContainerStarted","Data":"c87203d96b6e0268ccac49203a40e7708c346175703815be5f4130a40cb9ebe5"} Nov 03 22:23:00.470134 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:00.469898 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:00.614433 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:00.614389 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:00.926016 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:00.925986 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:23:00.954625 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:00.954596 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:01.214722 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:01.214651 5196 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-132.us-west-1.compute.internal" event="NodeReady" Nov 03 22:23:05.110374 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:05.110327 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:05.297065 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:05.297022 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:09.350400 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:09.350375 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:09.500329 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:09.500281 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:09.941548 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:09.941508 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-lqwmt" event={"ID":"d8f1371d-471f-496f-ac95-19eeb163c2c7","Type":"ContainerStarted","Data":"ca0ed3f2845c1e413f0a08a085933ce319441a8921137ee27abe412c0ad52cc2"} Nov 03 22:23:09.941894 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:09.941848 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-lqwmt" Nov 03 22:23:09.953960 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:09.953933 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-lqwmt" Nov 03 22:23:11.672850 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:11.672806 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:11.859232 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:11.859193 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:12.137401 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:12.137359 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:12.184518 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:12.184494 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:12.971236 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:12.971205 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:13.315972 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:13.315895 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-g6qjz" Nov 03 22:23:16.600572 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:16.600532 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:16.740924 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:16.740887 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:20.535537 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:20.535504 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:20.828195 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:20.828126 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:20.929038 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:20.929014 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jkjnj" Nov 03 22:23:23.259713 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:23.259680 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:24.099519 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:24.099478 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:24.341021 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:24.340985 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:24.481981 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:24.481892 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:25.917322 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:23:25.917288 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:23:28.497653 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.497620 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 22:23:28.498224 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.498071 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" containerName="kube-rbac-proxy-web" containerID="cri-o://658e3f5c349132244c9d37d07235fdce87ee0c1108b4deb5747b830a68d5b892" gracePeriod=600 Nov 03 22:23:28.498224 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.498096 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" containerName="thanos-sidecar" containerID="cri-o://c87203d96b6e0268ccac49203a40e7708c346175703815be5f4130a40cb9ebe5" gracePeriod=600 Nov 03 22:23:28.498224 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.498154 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" containerName="config-reloader" containerID="cri-o://88d04cbf2b22ea9f5917f15b5a4537e084b2380e5624151c4df5c28bdab98e5b" gracePeriod=600 Nov 03 22:23:28.498224 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.498105 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" containerName="kube-rbac-proxy-thanos" containerID="cri-o://e0a99d3e39e5ffd83e6e7cf7f06f4523d1cd5da6801c34a306cad57614dddde6" gracePeriod=600 Nov 03 22:23:28.498224 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.498066 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" containerName="prometheus" containerID="cri-o://5c86517d0dcc6eb524b065e086aaed1fd2f51845d0f21e6050f636cec774f035" gracePeriod=600 Nov 03 22:23:28.498447 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.498216 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" containerName="kube-rbac-proxy" containerID="cri-o://271eaab0ca06566c95c2df8668bfc62f7c850b3ba50d6fe8c122c8f8313d4eeb" gracePeriod=600 Nov 03 22:23:28.775835 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.775766 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:28.849377 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.849342 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-prometheus-k8s-tls\") pod \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " Nov 03 22:23:28.849541 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.849389 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-rulefiles-0\") pod \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " Nov 03 22:23:28.849541 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.849416 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-web-config\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-web-config\") pod \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " Nov 03 22:23:28.849661 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.849555 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-trusted-ca-bundle\") pod \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " Nov 03 22:23:28.849661 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.849594 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-kube-rbac-proxy\") pod \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " Nov 03 22:23:28.849760 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.849733 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-kube-api-access-gqg7s\" (UniqueName: \"kubernetes.io/projected/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-kube-api-access-gqg7s\") pod \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " Nov 03 22:23:28.849817 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.849787 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-db\") pod \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " Nov 03 22:23:28.849899 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.849825 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-metrics-client-certs\") pod \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " Nov 03 22:23:28.849899 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.849886 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " Nov 03 22:23:28.850004 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.849916 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-configmap-metrics-client-ca\") pod \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " Nov 03 22:23:28.850004 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.849954 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-rulefiles-1\") pod \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " Nov 03 22:23:28.850004 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.849984 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-config-out\" (UniqueName: \"kubernetes.io/empty-dir/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-config-out\") pod \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " Nov 03 22:23:28.850004 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.849985 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-trusted-ca-bundle" pod "a83dfb48-acb6-4cbc-bfc0-61103dafa90f" (UID: "a83dfb48-acb6-4cbc-bfc0-61103dafa90f"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:23:28.850201 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.850012 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-thanos-prometheus-http-client-file\") pod \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " Nov 03 22:23:28.850201 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.850052 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-grpc-tls\") pod \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " Nov 03 22:23:28.850201 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.850093 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " Nov 03 22:23:28.850201 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.850130 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-config\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-config\") pod \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " Nov 03 22:23:28.850201 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.850163 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-rulefiles-2\") pod \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " Nov 03 22:23:28.850201 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.850194 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-tls-assets\" (UniqueName: \"kubernetes.io/projected/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-tls-assets\") pod \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " Nov 03 22:23:28.850470 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.850244 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-configmap-kubelet-serving-ca-bundle\") pod \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " Nov 03 22:23:28.850470 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.850292 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-configmap-serving-certs-ca-bundle\") pod \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\" (UID: \"a83dfb48-acb6-4cbc-bfc0-61103dafa90f\") " Nov 03 22:23:28.852163 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.851396 5196 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:28.852293 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.852261 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-db" pod "a83dfb48-acb6-4cbc-bfc0-61103dafa90f" (UID: "a83dfb48-acb6-4cbc-bfc0-61103dafa90f"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:23:28.852667 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.852639 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-rulefiles-2" pod "a83dfb48-acb6-4cbc-bfc0-61103dafa90f" (UID: "a83dfb48-acb6-4cbc-bfc0-61103dafa90f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:23:28.852776 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.852722 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-kube-api-access-gqg7s" pod "a83dfb48-acb6-4cbc-bfc0-61103dafa90f" (UID: "a83dfb48-acb6-4cbc-bfc0-61103dafa90f"). InnerVolumeSpecName "kube-api-access-gqg7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:28.853262 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.853225 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-configmap-kubelet-serving-ca-bundle" pod "a83dfb48-acb6-4cbc-bfc0-61103dafa90f" (UID: "a83dfb48-acb6-4cbc-bfc0-61103dafa90f"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:23:28.853521 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.853491 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-configmap-serving-certs-ca-bundle" pod "a83dfb48-acb6-4cbc-bfc0-61103dafa90f" (UID: "a83dfb48-acb6-4cbc-bfc0-61103dafa90f"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:23:28.855976 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.854882 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-configmap-metrics-client-ca" pod "a83dfb48-acb6-4cbc-bfc0-61103dafa90f" (UID: "a83dfb48-acb6-4cbc-bfc0-61103dafa90f"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:23:28.855976 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.855203 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-rulefiles-1" pod "a83dfb48-acb6-4cbc-bfc0-61103dafa90f" (UID: "a83dfb48-acb6-4cbc-bfc0-61103dafa90f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:23:28.855976 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.855628 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-kube-rbac-proxy" pod "a83dfb48-acb6-4cbc-bfc0-61103dafa90f" (UID: "a83dfb48-acb6-4cbc-bfc0-61103dafa90f"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:23:28.856154 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.856011 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-thanos-prometheus-http-client-file" pod "a83dfb48-acb6-4cbc-bfc0-61103dafa90f" (UID: "a83dfb48-acb6-4cbc-bfc0-61103dafa90f"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:23:28.859514 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.859377 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-prometheus-k8s-kube-rbac-proxy-web" pod "a83dfb48-acb6-4cbc-bfc0-61103dafa90f" (UID: "a83dfb48-acb6-4cbc-bfc0-61103dafa90f"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:23:28.859602 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.859557 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-config-out" pod "a83dfb48-acb6-4cbc-bfc0-61103dafa90f" (UID: "a83dfb48-acb6-4cbc-bfc0-61103dafa90f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:23:28.859602 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.859584 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-tls-assets" pod "a83dfb48-acb6-4cbc-bfc0-61103dafa90f" (UID: "a83dfb48-acb6-4cbc-bfc0-61103dafa90f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:28.859728 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.859583 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-metrics-client-certs" pod "a83dfb48-acb6-4cbc-bfc0-61103dafa90f" (UID: "a83dfb48-acb6-4cbc-bfc0-61103dafa90f"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:23:28.859728 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.859684 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-prometheus-k8s-thanos-sidecar-tls" pod "a83dfb48-acb6-4cbc-bfc0-61103dafa90f" (UID: "a83dfb48-acb6-4cbc-bfc0-61103dafa90f"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:23:28.859842 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.859743 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-prometheus-k8s-tls" pod "a83dfb48-acb6-4cbc-bfc0-61103dafa90f" (UID: "a83dfb48-acb6-4cbc-bfc0-61103dafa90f"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:23:28.859842 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.859740 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-rulefiles-0" pod "a83dfb48-acb6-4cbc-bfc0-61103dafa90f" (UID: "a83dfb48-acb6-4cbc-bfc0-61103dafa90f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:23:28.859842 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.859820 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-grpc-tls" pod "a83dfb48-acb6-4cbc-bfc0-61103dafa90f" (UID: "a83dfb48-acb6-4cbc-bfc0-61103dafa90f"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:23:28.860245 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.860226 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-config" pod "a83dfb48-acb6-4cbc-bfc0-61103dafa90f" (UID: "a83dfb48-acb6-4cbc-bfc0-61103dafa90f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:23:28.867294 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.867273 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-web-config" pod "a83dfb48-acb6-4cbc-bfc0-61103dafa90f" (UID: "a83dfb48-acb6-4cbc-bfc0-61103dafa90f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:23:28.951748 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.951715 5196 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-db\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:28.951748 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.951745 5196 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-metrics-client-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:28.951944 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.951756 5196 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:28.951944 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.951766 5196 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-configmap-metrics-client-ca\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:28.951944 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.951775 5196 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:28.951944 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.951784 5196 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-config-out\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:28.951944 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.951793 5196 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:28.951944 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.951807 5196 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-grpc-tls\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:28.951944 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.951820 5196 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:28.951944 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.951829 5196 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-config\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:28.951944 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.951837 5196 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:28.951944 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.951845 5196 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-tls-assets\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:28.951944 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.951872 5196 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:28.951944 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.951886 5196 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:28.951944 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.951898 5196 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:28.951944 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.951906 5196 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:28.951944 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.951933 5196 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-web-config\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:28.951944 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.951942 5196 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-secret-kube-rbac-proxy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:28.951944 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:28.951950 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqg7s\" (UniqueName: \"kubernetes.io/projected/a83dfb48-acb6-4cbc-bfc0-61103dafa90f-kube-api-access-gqg7s\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:23:29.015214 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.015183 5196 generic.go:356] "Generic (PLEG): container finished" podID="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" containerID="e0a99d3e39e5ffd83e6e7cf7f06f4523d1cd5da6801c34a306cad57614dddde6" exitCode=0 Nov 03 22:23:29.015214 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.015209 5196 generic.go:356] "Generic (PLEG): container finished" podID="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" containerID="271eaab0ca06566c95c2df8668bfc62f7c850b3ba50d6fe8c122c8f8313d4eeb" exitCode=0 Nov 03 22:23:29.015214 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.015215 5196 generic.go:356] "Generic (PLEG): container finished" podID="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" containerID="658e3f5c349132244c9d37d07235fdce87ee0c1108b4deb5747b830a68d5b892" exitCode=0 Nov 03 22:23:29.015214 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.015220 5196 generic.go:356] "Generic (PLEG): container finished" podID="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" containerID="c87203d96b6e0268ccac49203a40e7708c346175703815be5f4130a40cb9ebe5" exitCode=0 Nov 03 22:23:29.015539 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.015227 5196 generic.go:356] "Generic (PLEG): container finished" podID="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" containerID="88d04cbf2b22ea9f5917f15b5a4537e084b2380e5624151c4df5c28bdab98e5b" exitCode=0 Nov 03 22:23:29.015539 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.015232 5196 generic.go:356] "Generic (PLEG): container finished" podID="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" containerID="5c86517d0dcc6eb524b065e086aaed1fd2f51845d0f21e6050f636cec774f035" exitCode=0 Nov 03 22:23:29.015539 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.015268 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a83dfb48-acb6-4cbc-bfc0-61103dafa90f","Type":"ContainerDied","Data":"e0a99d3e39e5ffd83e6e7cf7f06f4523d1cd5da6801c34a306cad57614dddde6"} Nov 03 22:23:29.015539 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.015336 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.015539 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.015390 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a83dfb48-acb6-4cbc-bfc0-61103dafa90f","Type":"ContainerDied","Data":"271eaab0ca06566c95c2df8668bfc62f7c850b3ba50d6fe8c122c8f8313d4eeb"} Nov 03 22:23:29.015539 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.015441 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a83dfb48-acb6-4cbc-bfc0-61103dafa90f","Type":"ContainerDied","Data":"658e3f5c349132244c9d37d07235fdce87ee0c1108b4deb5747b830a68d5b892"} Nov 03 22:23:29.015539 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.015474 5196 scope.go:122] "RemoveContainer" containerID="e0a99d3e39e5ffd83e6e7cf7f06f4523d1cd5da6801c34a306cad57614dddde6" Nov 03 22:23:29.015539 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.015483 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a83dfb48-acb6-4cbc-bfc0-61103dafa90f","Type":"ContainerDied","Data":"c87203d96b6e0268ccac49203a40e7708c346175703815be5f4130a40cb9ebe5"} Nov 03 22:23:29.015539 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.015525 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a83dfb48-acb6-4cbc-bfc0-61103dafa90f","Type":"ContainerDied","Data":"88d04cbf2b22ea9f5917f15b5a4537e084b2380e5624151c4df5c28bdab98e5b"} Nov 03 22:23:29.015957 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.015563 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a83dfb48-acb6-4cbc-bfc0-61103dafa90f","Type":"ContainerDied","Data":"5c86517d0dcc6eb524b065e086aaed1fd2f51845d0f21e6050f636cec774f035"} Nov 03 22:23:29.015957 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.015605 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a83dfb48-acb6-4cbc-bfc0-61103dafa90f","Type":"ContainerDied","Data":"2b4d8589545194c7a73b28d8b87d606ffcb4749772644f101dd0d5956cf6cb4f"} Nov 03 22:23:29.031624 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.031609 5196 scope.go:122] "RemoveContainer" containerID="271eaab0ca06566c95c2df8668bfc62f7c850b3ba50d6fe8c122c8f8313d4eeb" Nov 03 22:23:29.038569 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.038552 5196 scope.go:122] "RemoveContainer" containerID="658e3f5c349132244c9d37d07235fdce87ee0c1108b4deb5747b830a68d5b892" Nov 03 22:23:29.045296 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.045277 5196 scope.go:122] "RemoveContainer" containerID="c87203d96b6e0268ccac49203a40e7708c346175703815be5f4130a40cb9ebe5" Nov 03 22:23:29.051641 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.051626 5196 scope.go:122] "RemoveContainer" containerID="88d04cbf2b22ea9f5917f15b5a4537e084b2380e5624151c4df5c28bdab98e5b" Nov 03 22:23:29.058002 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.057985 5196 scope.go:122] "RemoveContainer" containerID="5c86517d0dcc6eb524b065e086aaed1fd2f51845d0f21e6050f636cec774f035" Nov 03 22:23:29.064566 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.064548 5196 scope.go:122] "RemoveContainer" containerID="1bbcf6f7eeeec5392123a3b9318980d2d5b05a92d70b2c8aebf5d0ad0780c635" Nov 03 22:23:29.070722 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.070709 5196 scope.go:122] "RemoveContainer" containerID="e0a99d3e39e5ffd83e6e7cf7f06f4523d1cd5da6801c34a306cad57614dddde6" Nov 03 22:23:29.071000 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:23:29.070972 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0a99d3e39e5ffd83e6e7cf7f06f4523d1cd5da6801c34a306cad57614dddde6\": container with ID starting with e0a99d3e39e5ffd83e6e7cf7f06f4523d1cd5da6801c34a306cad57614dddde6 not found: ID does not exist" containerID="e0a99d3e39e5ffd83e6e7cf7f06f4523d1cd5da6801c34a306cad57614dddde6" Nov 03 22:23:29.071078 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.071012 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0a99d3e39e5ffd83e6e7cf7f06f4523d1cd5da6801c34a306cad57614dddde6"} err="failed to get container status \"e0a99d3e39e5ffd83e6e7cf7f06f4523d1cd5da6801c34a306cad57614dddde6\": rpc error: code = NotFound desc = could not find container \"e0a99d3e39e5ffd83e6e7cf7f06f4523d1cd5da6801c34a306cad57614dddde6\": container with ID starting with e0a99d3e39e5ffd83e6e7cf7f06f4523d1cd5da6801c34a306cad57614dddde6 not found: ID does not exist" Nov 03 22:23:29.071078 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.071052 5196 scope.go:122] "RemoveContainer" containerID="271eaab0ca06566c95c2df8668bfc62f7c850b3ba50d6fe8c122c8f8313d4eeb" Nov 03 22:23:29.071312 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:23:29.071295 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"271eaab0ca06566c95c2df8668bfc62f7c850b3ba50d6fe8c122c8f8313d4eeb\": container with ID starting with 271eaab0ca06566c95c2df8668bfc62f7c850b3ba50d6fe8c122c8f8313d4eeb not found: ID does not exist" containerID="271eaab0ca06566c95c2df8668bfc62f7c850b3ba50d6fe8c122c8f8313d4eeb" Nov 03 22:23:29.071369 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.071321 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"271eaab0ca06566c95c2df8668bfc62f7c850b3ba50d6fe8c122c8f8313d4eeb"} err="failed to get container status \"271eaab0ca06566c95c2df8668bfc62f7c850b3ba50d6fe8c122c8f8313d4eeb\": rpc error: code = NotFound desc = could not find container \"271eaab0ca06566c95c2df8668bfc62f7c850b3ba50d6fe8c122c8f8313d4eeb\": container with ID starting with 271eaab0ca06566c95c2df8668bfc62f7c850b3ba50d6fe8c122c8f8313d4eeb not found: ID does not exist" Nov 03 22:23:29.071369 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.071338 5196 scope.go:122] "RemoveContainer" containerID="658e3f5c349132244c9d37d07235fdce87ee0c1108b4deb5747b830a68d5b892" Nov 03 22:23:29.071583 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:23:29.071566 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"658e3f5c349132244c9d37d07235fdce87ee0c1108b4deb5747b830a68d5b892\": container with ID starting with 658e3f5c349132244c9d37d07235fdce87ee0c1108b4deb5747b830a68d5b892 not found: ID does not exist" containerID="658e3f5c349132244c9d37d07235fdce87ee0c1108b4deb5747b830a68d5b892" Nov 03 22:23:29.071634 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.071587 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"658e3f5c349132244c9d37d07235fdce87ee0c1108b4deb5747b830a68d5b892"} err="failed to get container status \"658e3f5c349132244c9d37d07235fdce87ee0c1108b4deb5747b830a68d5b892\": rpc error: code = NotFound desc = could not find container \"658e3f5c349132244c9d37d07235fdce87ee0c1108b4deb5747b830a68d5b892\": container with ID starting with 658e3f5c349132244c9d37d07235fdce87ee0c1108b4deb5747b830a68d5b892 not found: ID does not exist" Nov 03 22:23:29.071634 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.071599 5196 scope.go:122] "RemoveContainer" containerID="c87203d96b6e0268ccac49203a40e7708c346175703815be5f4130a40cb9ebe5" Nov 03 22:23:29.071812 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:23:29.071795 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c87203d96b6e0268ccac49203a40e7708c346175703815be5f4130a40cb9ebe5\": container with ID starting with c87203d96b6e0268ccac49203a40e7708c346175703815be5f4130a40cb9ebe5 not found: ID does not exist" containerID="c87203d96b6e0268ccac49203a40e7708c346175703815be5f4130a40cb9ebe5" Nov 03 22:23:29.071872 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.071818 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c87203d96b6e0268ccac49203a40e7708c346175703815be5f4130a40cb9ebe5"} err="failed to get container status \"c87203d96b6e0268ccac49203a40e7708c346175703815be5f4130a40cb9ebe5\": rpc error: code = NotFound desc = could not find container \"c87203d96b6e0268ccac49203a40e7708c346175703815be5f4130a40cb9ebe5\": container with ID starting with c87203d96b6e0268ccac49203a40e7708c346175703815be5f4130a40cb9ebe5 not found: ID does not exist" Nov 03 22:23:29.071872 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.071833 5196 scope.go:122] "RemoveContainer" containerID="88d04cbf2b22ea9f5917f15b5a4537e084b2380e5624151c4df5c28bdab98e5b" Nov 03 22:23:29.072052 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:23:29.072038 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88d04cbf2b22ea9f5917f15b5a4537e084b2380e5624151c4df5c28bdab98e5b\": container with ID starting with 88d04cbf2b22ea9f5917f15b5a4537e084b2380e5624151c4df5c28bdab98e5b not found: ID does not exist" containerID="88d04cbf2b22ea9f5917f15b5a4537e084b2380e5624151c4df5c28bdab98e5b" Nov 03 22:23:29.072092 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.072055 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88d04cbf2b22ea9f5917f15b5a4537e084b2380e5624151c4df5c28bdab98e5b"} err="failed to get container status \"88d04cbf2b22ea9f5917f15b5a4537e084b2380e5624151c4df5c28bdab98e5b\": rpc error: code = NotFound desc = could not find container \"88d04cbf2b22ea9f5917f15b5a4537e084b2380e5624151c4df5c28bdab98e5b\": container with ID starting with 88d04cbf2b22ea9f5917f15b5a4537e084b2380e5624151c4df5c28bdab98e5b not found: ID does not exist" Nov 03 22:23:29.072092 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.072067 5196 scope.go:122] "RemoveContainer" containerID="5c86517d0dcc6eb524b065e086aaed1fd2f51845d0f21e6050f636cec774f035" Nov 03 22:23:29.072254 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:23:29.072241 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c86517d0dcc6eb524b065e086aaed1fd2f51845d0f21e6050f636cec774f035\": container with ID starting with 5c86517d0dcc6eb524b065e086aaed1fd2f51845d0f21e6050f636cec774f035 not found: ID does not exist" containerID="5c86517d0dcc6eb524b065e086aaed1fd2f51845d0f21e6050f636cec774f035" Nov 03 22:23:29.072294 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.072256 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c86517d0dcc6eb524b065e086aaed1fd2f51845d0f21e6050f636cec774f035"} err="failed to get container status \"5c86517d0dcc6eb524b065e086aaed1fd2f51845d0f21e6050f636cec774f035\": rpc error: code = NotFound desc = could not find container \"5c86517d0dcc6eb524b065e086aaed1fd2f51845d0f21e6050f636cec774f035\": container with ID starting with 5c86517d0dcc6eb524b065e086aaed1fd2f51845d0f21e6050f636cec774f035 not found: ID does not exist" Nov 03 22:23:29.072294 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.072266 5196 scope.go:122] "RemoveContainer" containerID="1bbcf6f7eeeec5392123a3b9318980d2d5b05a92d70b2c8aebf5d0ad0780c635" Nov 03 22:23:29.072461 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:23:29.072449 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bbcf6f7eeeec5392123a3b9318980d2d5b05a92d70b2c8aebf5d0ad0780c635\": container with ID starting with 1bbcf6f7eeeec5392123a3b9318980d2d5b05a92d70b2c8aebf5d0ad0780c635 not found: ID does not exist" containerID="1bbcf6f7eeeec5392123a3b9318980d2d5b05a92d70b2c8aebf5d0ad0780c635" Nov 03 22:23:29.072506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.072463 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bbcf6f7eeeec5392123a3b9318980d2d5b05a92d70b2c8aebf5d0ad0780c635"} err="failed to get container status \"1bbcf6f7eeeec5392123a3b9318980d2d5b05a92d70b2c8aebf5d0ad0780c635\": rpc error: code = NotFound desc = could not find container \"1bbcf6f7eeeec5392123a3b9318980d2d5b05a92d70b2c8aebf5d0ad0780c635\": container with ID starting with 1bbcf6f7eeeec5392123a3b9318980d2d5b05a92d70b2c8aebf5d0ad0780c635 not found: ID does not exist" Nov 03 22:23:29.072506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.072472 5196 scope.go:122] "RemoveContainer" containerID="e0a99d3e39e5ffd83e6e7cf7f06f4523d1cd5da6801c34a306cad57614dddde6" Nov 03 22:23:29.072661 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.072643 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0a99d3e39e5ffd83e6e7cf7f06f4523d1cd5da6801c34a306cad57614dddde6"} err="failed to get container status \"e0a99d3e39e5ffd83e6e7cf7f06f4523d1cd5da6801c34a306cad57614dddde6\": rpc error: code = NotFound desc = could not find container \"e0a99d3e39e5ffd83e6e7cf7f06f4523d1cd5da6801c34a306cad57614dddde6\": container with ID starting with e0a99d3e39e5ffd83e6e7cf7f06f4523d1cd5da6801c34a306cad57614dddde6 not found: ID does not exist" Nov 03 22:23:29.072728 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.072664 5196 scope.go:122] "RemoveContainer" containerID="271eaab0ca06566c95c2df8668bfc62f7c850b3ba50d6fe8c122c8f8313d4eeb" Nov 03 22:23:29.072901 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.072852 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"271eaab0ca06566c95c2df8668bfc62f7c850b3ba50d6fe8c122c8f8313d4eeb"} err="failed to get container status \"271eaab0ca06566c95c2df8668bfc62f7c850b3ba50d6fe8c122c8f8313d4eeb\": rpc error: code = NotFound desc = could not find container \"271eaab0ca06566c95c2df8668bfc62f7c850b3ba50d6fe8c122c8f8313d4eeb\": container with ID starting with 271eaab0ca06566c95c2df8668bfc62f7c850b3ba50d6fe8c122c8f8313d4eeb not found: ID does not exist" Nov 03 22:23:29.072948 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.072907 5196 scope.go:122] "RemoveContainer" containerID="658e3f5c349132244c9d37d07235fdce87ee0c1108b4deb5747b830a68d5b892" Nov 03 22:23:29.073125 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.073108 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"658e3f5c349132244c9d37d07235fdce87ee0c1108b4deb5747b830a68d5b892"} err="failed to get container status \"658e3f5c349132244c9d37d07235fdce87ee0c1108b4deb5747b830a68d5b892\": rpc error: code = NotFound desc = could not find container \"658e3f5c349132244c9d37d07235fdce87ee0c1108b4deb5747b830a68d5b892\": container with ID starting with 658e3f5c349132244c9d37d07235fdce87ee0c1108b4deb5747b830a68d5b892 not found: ID does not exist" Nov 03 22:23:29.073172 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.073126 5196 scope.go:122] "RemoveContainer" containerID="c87203d96b6e0268ccac49203a40e7708c346175703815be5f4130a40cb9ebe5" Nov 03 22:23:29.073329 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.073314 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c87203d96b6e0268ccac49203a40e7708c346175703815be5f4130a40cb9ebe5"} err="failed to get container status \"c87203d96b6e0268ccac49203a40e7708c346175703815be5f4130a40cb9ebe5\": rpc error: code = NotFound desc = could not find container \"c87203d96b6e0268ccac49203a40e7708c346175703815be5f4130a40cb9ebe5\": container with ID starting with c87203d96b6e0268ccac49203a40e7708c346175703815be5f4130a40cb9ebe5 not found: ID does not exist" Nov 03 22:23:29.073370 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.073330 5196 scope.go:122] "RemoveContainer" containerID="88d04cbf2b22ea9f5917f15b5a4537e084b2380e5624151c4df5c28bdab98e5b" Nov 03 22:23:29.073544 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.073526 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88d04cbf2b22ea9f5917f15b5a4537e084b2380e5624151c4df5c28bdab98e5b"} err="failed to get container status \"88d04cbf2b22ea9f5917f15b5a4537e084b2380e5624151c4df5c28bdab98e5b\": rpc error: code = NotFound desc = could not find container \"88d04cbf2b22ea9f5917f15b5a4537e084b2380e5624151c4df5c28bdab98e5b\": container with ID starting with 88d04cbf2b22ea9f5917f15b5a4537e084b2380e5624151c4df5c28bdab98e5b not found: ID does not exist" Nov 03 22:23:29.073595 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.073545 5196 scope.go:122] "RemoveContainer" containerID="5c86517d0dcc6eb524b065e086aaed1fd2f51845d0f21e6050f636cec774f035" Nov 03 22:23:29.073744 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.073726 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c86517d0dcc6eb524b065e086aaed1fd2f51845d0f21e6050f636cec774f035"} err="failed to get container status \"5c86517d0dcc6eb524b065e086aaed1fd2f51845d0f21e6050f636cec774f035\": rpc error: code = NotFound desc = could not find container \"5c86517d0dcc6eb524b065e086aaed1fd2f51845d0f21e6050f636cec774f035\": container with ID starting with 5c86517d0dcc6eb524b065e086aaed1fd2f51845d0f21e6050f636cec774f035 not found: ID does not exist" Nov 03 22:23:29.073787 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.073743 5196 scope.go:122] "RemoveContainer" containerID="1bbcf6f7eeeec5392123a3b9318980d2d5b05a92d70b2c8aebf5d0ad0780c635" Nov 03 22:23:29.073962 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.073945 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bbcf6f7eeeec5392123a3b9318980d2d5b05a92d70b2c8aebf5d0ad0780c635"} err="failed to get container status \"1bbcf6f7eeeec5392123a3b9318980d2d5b05a92d70b2c8aebf5d0ad0780c635\": rpc error: code = NotFound desc = could not find container \"1bbcf6f7eeeec5392123a3b9318980d2d5b05a92d70b2c8aebf5d0ad0780c635\": container with ID starting with 1bbcf6f7eeeec5392123a3b9318980d2d5b05a92d70b2c8aebf5d0ad0780c635 not found: ID does not exist" Nov 03 22:23:29.074011 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.073964 5196 scope.go:122] "RemoveContainer" containerID="e0a99d3e39e5ffd83e6e7cf7f06f4523d1cd5da6801c34a306cad57614dddde6" Nov 03 22:23:29.074149 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.074133 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0a99d3e39e5ffd83e6e7cf7f06f4523d1cd5da6801c34a306cad57614dddde6"} err="failed to get container status \"e0a99d3e39e5ffd83e6e7cf7f06f4523d1cd5da6801c34a306cad57614dddde6\": rpc error: code = NotFound desc = could not find container \"e0a99d3e39e5ffd83e6e7cf7f06f4523d1cd5da6801c34a306cad57614dddde6\": container with ID starting with e0a99d3e39e5ffd83e6e7cf7f06f4523d1cd5da6801c34a306cad57614dddde6 not found: ID does not exist" Nov 03 22:23:29.074191 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.074149 5196 scope.go:122] "RemoveContainer" containerID="271eaab0ca06566c95c2df8668bfc62f7c850b3ba50d6fe8c122c8f8313d4eeb" Nov 03 22:23:29.074389 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.074371 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"271eaab0ca06566c95c2df8668bfc62f7c850b3ba50d6fe8c122c8f8313d4eeb"} err="failed to get container status \"271eaab0ca06566c95c2df8668bfc62f7c850b3ba50d6fe8c122c8f8313d4eeb\": rpc error: code = NotFound desc = could not find container \"271eaab0ca06566c95c2df8668bfc62f7c850b3ba50d6fe8c122c8f8313d4eeb\": container with ID starting with 271eaab0ca06566c95c2df8668bfc62f7c850b3ba50d6fe8c122c8f8313d4eeb not found: ID does not exist" Nov 03 22:23:29.074447 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.074390 5196 scope.go:122] "RemoveContainer" containerID="658e3f5c349132244c9d37d07235fdce87ee0c1108b4deb5747b830a68d5b892" Nov 03 22:23:29.074595 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.074579 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"658e3f5c349132244c9d37d07235fdce87ee0c1108b4deb5747b830a68d5b892"} err="failed to get container status \"658e3f5c349132244c9d37d07235fdce87ee0c1108b4deb5747b830a68d5b892\": rpc error: code = NotFound desc = could not find container \"658e3f5c349132244c9d37d07235fdce87ee0c1108b4deb5747b830a68d5b892\": container with ID starting with 658e3f5c349132244c9d37d07235fdce87ee0c1108b4deb5747b830a68d5b892 not found: ID does not exist" Nov 03 22:23:29.074636 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.074596 5196 scope.go:122] "RemoveContainer" containerID="c87203d96b6e0268ccac49203a40e7708c346175703815be5f4130a40cb9ebe5" Nov 03 22:23:29.074780 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.074765 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c87203d96b6e0268ccac49203a40e7708c346175703815be5f4130a40cb9ebe5"} err="failed to get container status \"c87203d96b6e0268ccac49203a40e7708c346175703815be5f4130a40cb9ebe5\": rpc error: code = NotFound desc = could not find container \"c87203d96b6e0268ccac49203a40e7708c346175703815be5f4130a40cb9ebe5\": container with ID starting with c87203d96b6e0268ccac49203a40e7708c346175703815be5f4130a40cb9ebe5 not found: ID does not exist" Nov 03 22:23:29.074819 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.074779 5196 scope.go:122] "RemoveContainer" containerID="88d04cbf2b22ea9f5917f15b5a4537e084b2380e5624151c4df5c28bdab98e5b" Nov 03 22:23:29.074999 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.074983 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88d04cbf2b22ea9f5917f15b5a4537e084b2380e5624151c4df5c28bdab98e5b"} err="failed to get container status \"88d04cbf2b22ea9f5917f15b5a4537e084b2380e5624151c4df5c28bdab98e5b\": rpc error: code = NotFound desc = could not find container \"88d04cbf2b22ea9f5917f15b5a4537e084b2380e5624151c4df5c28bdab98e5b\": container with ID starting with 88d04cbf2b22ea9f5917f15b5a4537e084b2380e5624151c4df5c28bdab98e5b not found: ID does not exist" Nov 03 22:23:29.075045 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.074999 5196 scope.go:122] "RemoveContainer" containerID="5c86517d0dcc6eb524b065e086aaed1fd2f51845d0f21e6050f636cec774f035" Nov 03 22:23:29.075191 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.075176 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c86517d0dcc6eb524b065e086aaed1fd2f51845d0f21e6050f636cec774f035"} err="failed to get container status \"5c86517d0dcc6eb524b065e086aaed1fd2f51845d0f21e6050f636cec774f035\": rpc error: code = NotFound desc = could not find container \"5c86517d0dcc6eb524b065e086aaed1fd2f51845d0f21e6050f636cec774f035\": container with ID starting with 5c86517d0dcc6eb524b065e086aaed1fd2f51845d0f21e6050f636cec774f035 not found: ID does not exist" Nov 03 22:23:29.075233 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.075190 5196 scope.go:122] "RemoveContainer" containerID="1bbcf6f7eeeec5392123a3b9318980d2d5b05a92d70b2c8aebf5d0ad0780c635" Nov 03 22:23:29.075407 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.075389 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bbcf6f7eeeec5392123a3b9318980d2d5b05a92d70b2c8aebf5d0ad0780c635"} err="failed to get container status \"1bbcf6f7eeeec5392123a3b9318980d2d5b05a92d70b2c8aebf5d0ad0780c635\": rpc error: code = NotFound desc = could not find container \"1bbcf6f7eeeec5392123a3b9318980d2d5b05a92d70b2c8aebf5d0ad0780c635\": container with ID starting with 1bbcf6f7eeeec5392123a3b9318980d2d5b05a92d70b2c8aebf5d0ad0780c635 not found: ID does not exist" Nov 03 22:23:29.075467 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.075409 5196 scope.go:122] "RemoveContainer" containerID="e0a99d3e39e5ffd83e6e7cf7f06f4523d1cd5da6801c34a306cad57614dddde6" Nov 03 22:23:29.075615 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.075594 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0a99d3e39e5ffd83e6e7cf7f06f4523d1cd5da6801c34a306cad57614dddde6"} err="failed to get container status \"e0a99d3e39e5ffd83e6e7cf7f06f4523d1cd5da6801c34a306cad57614dddde6\": rpc error: code = NotFound desc = could not find container \"e0a99d3e39e5ffd83e6e7cf7f06f4523d1cd5da6801c34a306cad57614dddde6\": container with ID starting with e0a99d3e39e5ffd83e6e7cf7f06f4523d1cd5da6801c34a306cad57614dddde6 not found: ID does not exist" Nov 03 22:23:29.075659 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.075616 5196 scope.go:122] "RemoveContainer" containerID="271eaab0ca06566c95c2df8668bfc62f7c850b3ba50d6fe8c122c8f8313d4eeb" Nov 03 22:23:29.075802 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.075785 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"271eaab0ca06566c95c2df8668bfc62f7c850b3ba50d6fe8c122c8f8313d4eeb"} err="failed to get container status \"271eaab0ca06566c95c2df8668bfc62f7c850b3ba50d6fe8c122c8f8313d4eeb\": rpc error: code = NotFound desc = could not find container \"271eaab0ca06566c95c2df8668bfc62f7c850b3ba50d6fe8c122c8f8313d4eeb\": container with ID starting with 271eaab0ca06566c95c2df8668bfc62f7c850b3ba50d6fe8c122c8f8313d4eeb not found: ID does not exist" Nov 03 22:23:29.075898 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.075804 5196 scope.go:122] "RemoveContainer" containerID="658e3f5c349132244c9d37d07235fdce87ee0c1108b4deb5747b830a68d5b892" Nov 03 22:23:29.076056 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.076040 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"658e3f5c349132244c9d37d07235fdce87ee0c1108b4deb5747b830a68d5b892"} err="failed to get container status \"658e3f5c349132244c9d37d07235fdce87ee0c1108b4deb5747b830a68d5b892\": rpc error: code = NotFound desc = could not find container \"658e3f5c349132244c9d37d07235fdce87ee0c1108b4deb5747b830a68d5b892\": container with ID starting with 658e3f5c349132244c9d37d07235fdce87ee0c1108b4deb5747b830a68d5b892 not found: ID does not exist" Nov 03 22:23:29.076056 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.076056 5196 scope.go:122] "RemoveContainer" containerID="c87203d96b6e0268ccac49203a40e7708c346175703815be5f4130a40cb9ebe5" Nov 03 22:23:29.076253 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.076235 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c87203d96b6e0268ccac49203a40e7708c346175703815be5f4130a40cb9ebe5"} err="failed to get container status \"c87203d96b6e0268ccac49203a40e7708c346175703815be5f4130a40cb9ebe5\": rpc error: code = NotFound desc = could not find container \"c87203d96b6e0268ccac49203a40e7708c346175703815be5f4130a40cb9ebe5\": container with ID starting with c87203d96b6e0268ccac49203a40e7708c346175703815be5f4130a40cb9ebe5 not found: ID does not exist" Nov 03 22:23:29.076296 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.076253 5196 scope.go:122] "RemoveContainer" containerID="88d04cbf2b22ea9f5917f15b5a4537e084b2380e5624151c4df5c28bdab98e5b" Nov 03 22:23:29.076484 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.076467 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88d04cbf2b22ea9f5917f15b5a4537e084b2380e5624151c4df5c28bdab98e5b"} err="failed to get container status \"88d04cbf2b22ea9f5917f15b5a4537e084b2380e5624151c4df5c28bdab98e5b\": rpc error: code = NotFound desc = could not find container \"88d04cbf2b22ea9f5917f15b5a4537e084b2380e5624151c4df5c28bdab98e5b\": container with ID starting with 88d04cbf2b22ea9f5917f15b5a4537e084b2380e5624151c4df5c28bdab98e5b not found: ID does not exist" Nov 03 22:23:29.076553 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.076485 5196 scope.go:122] "RemoveContainer" containerID="5c86517d0dcc6eb524b065e086aaed1fd2f51845d0f21e6050f636cec774f035" Nov 03 22:23:29.076716 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.076698 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c86517d0dcc6eb524b065e086aaed1fd2f51845d0f21e6050f636cec774f035"} err="failed to get container status \"5c86517d0dcc6eb524b065e086aaed1fd2f51845d0f21e6050f636cec774f035\": rpc error: code = NotFound desc = could not find container \"5c86517d0dcc6eb524b065e086aaed1fd2f51845d0f21e6050f636cec774f035\": container with ID starting with 5c86517d0dcc6eb524b065e086aaed1fd2f51845d0f21e6050f636cec774f035 not found: ID does not exist" Nov 03 22:23:29.076768 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.076717 5196 scope.go:122] "RemoveContainer" containerID="1bbcf6f7eeeec5392123a3b9318980d2d5b05a92d70b2c8aebf5d0ad0780c635" Nov 03 22:23:29.076935 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.076917 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bbcf6f7eeeec5392123a3b9318980d2d5b05a92d70b2c8aebf5d0ad0780c635"} err="failed to get container status \"1bbcf6f7eeeec5392123a3b9318980d2d5b05a92d70b2c8aebf5d0ad0780c635\": rpc error: code = NotFound desc = could not find container \"1bbcf6f7eeeec5392123a3b9318980d2d5b05a92d70b2c8aebf5d0ad0780c635\": container with ID starting with 1bbcf6f7eeeec5392123a3b9318980d2d5b05a92d70b2c8aebf5d0ad0780c635 not found: ID does not exist" Nov 03 22:23:29.077017 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.076936 5196 scope.go:122] "RemoveContainer" containerID="e0a99d3e39e5ffd83e6e7cf7f06f4523d1cd5da6801c34a306cad57614dddde6" Nov 03 22:23:29.077151 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.077137 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0a99d3e39e5ffd83e6e7cf7f06f4523d1cd5da6801c34a306cad57614dddde6"} err="failed to get container status \"e0a99d3e39e5ffd83e6e7cf7f06f4523d1cd5da6801c34a306cad57614dddde6\": rpc error: code = NotFound desc = could not find container \"e0a99d3e39e5ffd83e6e7cf7f06f4523d1cd5da6801c34a306cad57614dddde6\": container with ID starting with e0a99d3e39e5ffd83e6e7cf7f06f4523d1cd5da6801c34a306cad57614dddde6 not found: ID does not exist" Nov 03 22:23:29.077200 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.077151 5196 scope.go:122] "RemoveContainer" containerID="271eaab0ca06566c95c2df8668bfc62f7c850b3ba50d6fe8c122c8f8313d4eeb" Nov 03 22:23:29.077333 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.077317 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"271eaab0ca06566c95c2df8668bfc62f7c850b3ba50d6fe8c122c8f8313d4eeb"} err="failed to get container status \"271eaab0ca06566c95c2df8668bfc62f7c850b3ba50d6fe8c122c8f8313d4eeb\": rpc error: code = NotFound desc = could not find container \"271eaab0ca06566c95c2df8668bfc62f7c850b3ba50d6fe8c122c8f8313d4eeb\": container with ID starting with 271eaab0ca06566c95c2df8668bfc62f7c850b3ba50d6fe8c122c8f8313d4eeb not found: ID does not exist" Nov 03 22:23:29.077390 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.077333 5196 scope.go:122] "RemoveContainer" containerID="658e3f5c349132244c9d37d07235fdce87ee0c1108b4deb5747b830a68d5b892" Nov 03 22:23:29.077537 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.077516 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"658e3f5c349132244c9d37d07235fdce87ee0c1108b4deb5747b830a68d5b892"} err="failed to get container status \"658e3f5c349132244c9d37d07235fdce87ee0c1108b4deb5747b830a68d5b892\": rpc error: code = NotFound desc = could not find container \"658e3f5c349132244c9d37d07235fdce87ee0c1108b4deb5747b830a68d5b892\": container with ID starting with 658e3f5c349132244c9d37d07235fdce87ee0c1108b4deb5747b830a68d5b892 not found: ID does not exist" Nov 03 22:23:29.077611 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.077539 5196 scope.go:122] "RemoveContainer" containerID="c87203d96b6e0268ccac49203a40e7708c346175703815be5f4130a40cb9ebe5" Nov 03 22:23:29.077748 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.077733 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c87203d96b6e0268ccac49203a40e7708c346175703815be5f4130a40cb9ebe5"} err="failed to get container status \"c87203d96b6e0268ccac49203a40e7708c346175703815be5f4130a40cb9ebe5\": rpc error: code = NotFound desc = could not find container \"c87203d96b6e0268ccac49203a40e7708c346175703815be5f4130a40cb9ebe5\": container with ID starting with c87203d96b6e0268ccac49203a40e7708c346175703815be5f4130a40cb9ebe5 not found: ID does not exist" Nov 03 22:23:29.077794 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.077748 5196 scope.go:122] "RemoveContainer" containerID="88d04cbf2b22ea9f5917f15b5a4537e084b2380e5624151c4df5c28bdab98e5b" Nov 03 22:23:29.077998 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.077974 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88d04cbf2b22ea9f5917f15b5a4537e084b2380e5624151c4df5c28bdab98e5b"} err="failed to get container status \"88d04cbf2b22ea9f5917f15b5a4537e084b2380e5624151c4df5c28bdab98e5b\": rpc error: code = NotFound desc = could not find container \"88d04cbf2b22ea9f5917f15b5a4537e084b2380e5624151c4df5c28bdab98e5b\": container with ID starting with 88d04cbf2b22ea9f5917f15b5a4537e084b2380e5624151c4df5c28bdab98e5b not found: ID does not exist" Nov 03 22:23:29.078091 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.077999 5196 scope.go:122] "RemoveContainer" containerID="5c86517d0dcc6eb524b065e086aaed1fd2f51845d0f21e6050f636cec774f035" Nov 03 22:23:29.078226 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.078209 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c86517d0dcc6eb524b065e086aaed1fd2f51845d0f21e6050f636cec774f035"} err="failed to get container status \"5c86517d0dcc6eb524b065e086aaed1fd2f51845d0f21e6050f636cec774f035\": rpc error: code = NotFound desc = could not find container \"5c86517d0dcc6eb524b065e086aaed1fd2f51845d0f21e6050f636cec774f035\": container with ID starting with 5c86517d0dcc6eb524b065e086aaed1fd2f51845d0f21e6050f636cec774f035 not found: ID does not exist" Nov 03 22:23:29.078271 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.078227 5196 scope.go:122] "RemoveContainer" containerID="1bbcf6f7eeeec5392123a3b9318980d2d5b05a92d70b2c8aebf5d0ad0780c635" Nov 03 22:23:29.078431 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.078416 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bbcf6f7eeeec5392123a3b9318980d2d5b05a92d70b2c8aebf5d0ad0780c635"} err="failed to get container status \"1bbcf6f7eeeec5392123a3b9318980d2d5b05a92d70b2c8aebf5d0ad0780c635\": rpc error: code = NotFound desc = could not find container \"1bbcf6f7eeeec5392123a3b9318980d2d5b05a92d70b2c8aebf5d0ad0780c635\": container with ID starting with 1bbcf6f7eeeec5392123a3b9318980d2d5b05a92d70b2c8aebf5d0ad0780c635 not found: ID does not exist" Nov 03 22:23:29.078466 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.078433 5196 scope.go:122] "RemoveContainer" containerID="e0a99d3e39e5ffd83e6e7cf7f06f4523d1cd5da6801c34a306cad57614dddde6" Nov 03 22:23:29.078639 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.078622 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0a99d3e39e5ffd83e6e7cf7f06f4523d1cd5da6801c34a306cad57614dddde6"} err="failed to get container status \"e0a99d3e39e5ffd83e6e7cf7f06f4523d1cd5da6801c34a306cad57614dddde6\": rpc error: code = NotFound desc = could not find container \"e0a99d3e39e5ffd83e6e7cf7f06f4523d1cd5da6801c34a306cad57614dddde6\": container with ID starting with e0a99d3e39e5ffd83e6e7cf7f06f4523d1cd5da6801c34a306cad57614dddde6 not found: ID does not exist" Nov 03 22:23:29.078701 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.078639 5196 scope.go:122] "RemoveContainer" containerID="271eaab0ca06566c95c2df8668bfc62f7c850b3ba50d6fe8c122c8f8313d4eeb" Nov 03 22:23:29.078884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.078846 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"271eaab0ca06566c95c2df8668bfc62f7c850b3ba50d6fe8c122c8f8313d4eeb"} err="failed to get container status \"271eaab0ca06566c95c2df8668bfc62f7c850b3ba50d6fe8c122c8f8313d4eeb\": rpc error: code = NotFound desc = could not find container \"271eaab0ca06566c95c2df8668bfc62f7c850b3ba50d6fe8c122c8f8313d4eeb\": container with ID starting with 271eaab0ca06566c95c2df8668bfc62f7c850b3ba50d6fe8c122c8f8313d4eeb not found: ID does not exist" Nov 03 22:23:29.078884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.078882 5196 scope.go:122] "RemoveContainer" containerID="658e3f5c349132244c9d37d07235fdce87ee0c1108b4deb5747b830a68d5b892" Nov 03 22:23:29.079115 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.079099 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"658e3f5c349132244c9d37d07235fdce87ee0c1108b4deb5747b830a68d5b892"} err="failed to get container status \"658e3f5c349132244c9d37d07235fdce87ee0c1108b4deb5747b830a68d5b892\": rpc error: code = NotFound desc = could not find container \"658e3f5c349132244c9d37d07235fdce87ee0c1108b4deb5747b830a68d5b892\": container with ID starting with 658e3f5c349132244c9d37d07235fdce87ee0c1108b4deb5747b830a68d5b892 not found: ID does not exist" Nov 03 22:23:29.079162 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.079115 5196 scope.go:122] "RemoveContainer" containerID="c87203d96b6e0268ccac49203a40e7708c346175703815be5f4130a40cb9ebe5" Nov 03 22:23:29.079332 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.079313 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c87203d96b6e0268ccac49203a40e7708c346175703815be5f4130a40cb9ebe5"} err="failed to get container status \"c87203d96b6e0268ccac49203a40e7708c346175703815be5f4130a40cb9ebe5\": rpc error: code = NotFound desc = could not find container \"c87203d96b6e0268ccac49203a40e7708c346175703815be5f4130a40cb9ebe5\": container with ID starting with c87203d96b6e0268ccac49203a40e7708c346175703815be5f4130a40cb9ebe5 not found: ID does not exist" Nov 03 22:23:29.079369 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.079333 5196 scope.go:122] "RemoveContainer" containerID="88d04cbf2b22ea9f5917f15b5a4537e084b2380e5624151c4df5c28bdab98e5b" Nov 03 22:23:29.079533 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.079510 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88d04cbf2b22ea9f5917f15b5a4537e084b2380e5624151c4df5c28bdab98e5b"} err="failed to get container status \"88d04cbf2b22ea9f5917f15b5a4537e084b2380e5624151c4df5c28bdab98e5b\": rpc error: code = NotFound desc = could not find container \"88d04cbf2b22ea9f5917f15b5a4537e084b2380e5624151c4df5c28bdab98e5b\": container with ID starting with 88d04cbf2b22ea9f5917f15b5a4537e084b2380e5624151c4df5c28bdab98e5b not found: ID does not exist" Nov 03 22:23:29.079569 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.079533 5196 scope.go:122] "RemoveContainer" containerID="5c86517d0dcc6eb524b065e086aaed1fd2f51845d0f21e6050f636cec774f035" Nov 03 22:23:29.079734 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.079714 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c86517d0dcc6eb524b065e086aaed1fd2f51845d0f21e6050f636cec774f035"} err="failed to get container status \"5c86517d0dcc6eb524b065e086aaed1fd2f51845d0f21e6050f636cec774f035\": rpc error: code = NotFound desc = could not find container \"5c86517d0dcc6eb524b065e086aaed1fd2f51845d0f21e6050f636cec774f035\": container with ID starting with 5c86517d0dcc6eb524b065e086aaed1fd2f51845d0f21e6050f636cec774f035 not found: ID does not exist" Nov 03 22:23:29.079808 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.079735 5196 scope.go:122] "RemoveContainer" containerID="1bbcf6f7eeeec5392123a3b9318980d2d5b05a92d70b2c8aebf5d0ad0780c635" Nov 03 22:23:29.079963 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.079947 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bbcf6f7eeeec5392123a3b9318980d2d5b05a92d70b2c8aebf5d0ad0780c635"} err="failed to get container status \"1bbcf6f7eeeec5392123a3b9318980d2d5b05a92d70b2c8aebf5d0ad0780c635\": rpc error: code = NotFound desc = could not find container \"1bbcf6f7eeeec5392123a3b9318980d2d5b05a92d70b2c8aebf5d0ad0780c635\": container with ID starting with 1bbcf6f7eeeec5392123a3b9318980d2d5b05a92d70b2c8aebf5d0ad0780c635 not found: ID does not exist" Nov 03 22:23:29.187366 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.187336 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 22:23:29.190804 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.190779 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 22:23:29.234548 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.234521 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 22:23:29.234745 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.234734 5196 cpu_manager.go:414] "containerMap: removing container" podUID="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" containerName="config-reloader" Nov 03 22:23:29.234790 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.234746 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" containerName="config-reloader" Nov 03 22:23:29.234790 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.234762 5196 cpu_manager.go:414] "containerMap: removing container" podUID="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" containerName="kube-rbac-proxy" Nov 03 22:23:29.234790 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.234767 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" containerName="kube-rbac-proxy" Nov 03 22:23:29.234790 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.234774 5196 cpu_manager.go:414] "containerMap: removing container" podUID="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" containerName="init-config-reloader" Nov 03 22:23:29.234790 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.234779 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" containerName="init-config-reloader" Nov 03 22:23:29.234955 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.234794 5196 cpu_manager.go:414] "containerMap: removing container" podUID="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" containerName="prometheus" Nov 03 22:23:29.234955 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.234799 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" containerName="prometheus" Nov 03 22:23:29.234955 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.234804 5196 cpu_manager.go:414] "containerMap: removing container" podUID="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" containerName="thanos-sidecar" Nov 03 22:23:29.234955 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.234809 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" containerName="thanos-sidecar" Nov 03 22:23:29.234955 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.234814 5196 cpu_manager.go:414] "containerMap: removing container" podUID="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" containerName="kube-rbac-proxy-web" Nov 03 22:23:29.234955 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.234818 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" containerName="kube-rbac-proxy-web" Nov 03 22:23:29.234955 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.234824 5196 cpu_manager.go:414] "containerMap: removing container" podUID="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" containerName="kube-rbac-proxy-thanos" Nov 03 22:23:29.234955 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.234828 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" containerName="kube-rbac-proxy-thanos" Nov 03 22:23:29.234955 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.234881 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" containerName="kube-rbac-proxy" Nov 03 22:23:29.234955 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.234890 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" containerName="prometheus" Nov 03 22:23:29.234955 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.234896 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" containerName="thanos-sidecar" Nov 03 22:23:29.234955 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.234901 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" containerName="kube-rbac-proxy-web" Nov 03 22:23:29.234955 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.234906 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" containerName="kube-rbac-proxy-thanos" Nov 03 22:23:29.234955 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.234911 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" containerName="config-reloader" Nov 03 22:23:29.263582 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.263562 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.311596 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.311535 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 22:23:29.318936 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.318912 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 22:23:29.318936 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.318926 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 22:23:29.319144 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.318953 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-26t3ivs4b9nmd\"" Nov 03 22:23:29.319144 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.319004 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 22:23:29.319144 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.319046 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-t9khm\"" Nov 03 22:23:29.319144 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.319080 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 22:23:29.319144 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.319140 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 22:23:29.319334 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.319160 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 22:23:29.319543 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.319508 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 22:23:29.319608 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.319595 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 22:23:29.319776 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.319722 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 22:23:29.319776 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.319767 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 22:23:29.319776 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.319747 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 22:23:29.320039 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.319770 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 22:23:29.320039 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.319804 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 22:23:29.354848 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.354809 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a254ea4-03c8-4dbb-904c-a1def38b0d83-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.354848 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.354870 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1a254ea4-03c8-4dbb-904c-a1def38b0d83-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.355034 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.354887 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkdq6\" (UniqueName: \"kubernetes.io/projected/1a254ea4-03c8-4dbb-904c-a1def38b0d83-kube-api-access-jkdq6\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.355078 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.355042 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a254ea4-03c8-4dbb-904c-a1def38b0d83-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.355078 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.355061 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1a254ea4-03c8-4dbb-904c-a1def38b0d83-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.355157 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.355077 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1a254ea4-03c8-4dbb-904c-a1def38b0d83-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.355157 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.355094 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1a254ea4-03c8-4dbb-904c-a1def38b0d83-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.355218 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.355165 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1a254ea4-03c8-4dbb-904c-a1def38b0d83-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.355218 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.355211 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1a254ea4-03c8-4dbb-904c-a1def38b0d83-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.355279 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.355247 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a254ea4-03c8-4dbb-904c-a1def38b0d83-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.355311 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.355289 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1a254ea4-03c8-4dbb-904c-a1def38b0d83-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.355346 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.355311 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1a254ea4-03c8-4dbb-904c-a1def38b0d83-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.355346 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.355340 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1a254ea4-03c8-4dbb-904c-a1def38b0d83-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.355433 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.355368 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1a254ea4-03c8-4dbb-904c-a1def38b0d83-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.355433 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.355427 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1a254ea4-03c8-4dbb-904c-a1def38b0d83-config-out\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.355532 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.355511 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1a254ea4-03c8-4dbb-904c-a1def38b0d83-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.355588 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.355539 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a254ea4-03c8-4dbb-904c-a1def38b0d83-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.355588 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.355577 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1a254ea4-03c8-4dbb-904c-a1def38b0d83-web-config\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.355689 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.355605 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1a254ea4-03c8-4dbb-904c-a1def38b0d83-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.355689 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.355630 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1a254ea4-03c8-4dbb-904c-a1def38b0d83-config\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.370546 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.370518 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 22:23:29.370724 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.370704 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 22:23:29.456664 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.456631 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1a254ea4-03c8-4dbb-904c-a1def38b0d83-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.456664 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.456664 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1a254ea4-03c8-4dbb-904c-a1def38b0d83-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.456905 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.456683 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a254ea4-03c8-4dbb-904c-a1def38b0d83-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.456905 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.456809 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1a254ea4-03c8-4dbb-904c-a1def38b0d83-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.456905 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.456889 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1a254ea4-03c8-4dbb-904c-a1def38b0d83-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.457057 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.456917 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1a254ea4-03c8-4dbb-904c-a1def38b0d83-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.457057 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.457041 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1a254ea4-03c8-4dbb-904c-a1def38b0d83-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.457148 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.457102 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1a254ea4-03c8-4dbb-904c-a1def38b0d83-config-out\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.457148 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.457140 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1a254ea4-03c8-4dbb-904c-a1def38b0d83-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.457243 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.457187 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a254ea4-03c8-4dbb-904c-a1def38b0d83-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.457295 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.457231 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1a254ea4-03c8-4dbb-904c-a1def38b0d83-web-config\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.457295 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.457277 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1a254ea4-03c8-4dbb-904c-a1def38b0d83-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.457397 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.457301 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1a254ea4-03c8-4dbb-904c-a1def38b0d83-config\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.457397 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.457312 5196 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 22:23:29.457397 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.457332 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1a254ea4-03c8-4dbb-904c-a1def38b0d83-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.457397 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.457347 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a254ea4-03c8-4dbb-904c-a1def38b0d83-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.457397 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.457389 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1a254ea4-03c8-4dbb-904c-a1def38b0d83-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.459435 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.457414 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1a254ea4-03c8-4dbb-904c-a1def38b0d83-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.459435 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.457416 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkdq6\" (UniqueName: \"kubernetes.io/projected/1a254ea4-03c8-4dbb-904c-a1def38b0d83-kube-api-access-jkdq6\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.459435 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.457477 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a254ea4-03c8-4dbb-904c-a1def38b0d83-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.459435 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.457504 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1a254ea4-03c8-4dbb-904c-a1def38b0d83-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.459435 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.457530 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1a254ea4-03c8-4dbb-904c-a1def38b0d83-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.459435 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.457554 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1a254ea4-03c8-4dbb-904c-a1def38b0d83-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.459435 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.457848 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a254ea4-03c8-4dbb-904c-a1def38b0d83-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.459435 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.457995 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1a254ea4-03c8-4dbb-904c-a1def38b0d83-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.459435 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.458569 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a254ea4-03c8-4dbb-904c-a1def38b0d83-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.459435 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.458921 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a254ea4-03c8-4dbb-904c-a1def38b0d83-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.459435 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.459024 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a254ea4-03c8-4dbb-904c-a1def38b0d83-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.460738 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.460666 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1a254ea4-03c8-4dbb-904c-a1def38b0d83-config-out\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.461122 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.461097 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1a254ea4-03c8-4dbb-904c-a1def38b0d83-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.461208 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.461097 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1a254ea4-03c8-4dbb-904c-a1def38b0d83-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.461437 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.461413 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1a254ea4-03c8-4dbb-904c-a1def38b0d83-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.462120 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.462066 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1a254ea4-03c8-4dbb-904c-a1def38b0d83-web-config\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.462221 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.462154 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1a254ea4-03c8-4dbb-904c-a1def38b0d83-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.462221 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.462180 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1a254ea4-03c8-4dbb-904c-a1def38b0d83-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.462672 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.462624 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1a254ea4-03c8-4dbb-904c-a1def38b0d83-config\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.462777 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.462718 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1a254ea4-03c8-4dbb-904c-a1def38b0d83-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.462968 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.462946 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1a254ea4-03c8-4dbb-904c-a1def38b0d83-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.463122 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.463105 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1a254ea4-03c8-4dbb-904c-a1def38b0d83-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.463491 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.463472 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1a254ea4-03c8-4dbb-904c-a1def38b0d83-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.523028 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.522996 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkdq6\" (UniqueName: \"kubernetes.io/projected/1a254ea4-03c8-4dbb-904c-a1def38b0d83-kube-api-access-jkdq6\") pod \"prometheus-k8s-0\" (UID: \"1a254ea4-03c8-4dbb-904c-a1def38b0d83\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:29.679027 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:29.678992 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:30.177965 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:30.177933 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 22:23:30.181816 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:23:30.181783 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a254ea4_03c8_4dbb_904c_a1def38b0d83.slice/crio-3a553fa03eb14d90e39c1447691faf40c0fdbd65191a2bba54899637e850c756 WatchSource:0}: Error finding container 3a553fa03eb14d90e39c1447691faf40c0fdbd65191a2bba54899637e850c756: Status 404 returned error can't find the container with id 3a553fa03eb14d90e39c1447691faf40c0fdbd65191a2bba54899637e850c756 Nov 03 22:23:30.807920 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:30.807883 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:30.811933 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:30.811905 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a83dfb48-acb6-4cbc-bfc0-61103dafa90f" path="/var/lib/kubelet/pods/a83dfb48-acb6-4cbc-bfc0-61103dafa90f/volumes" Nov 03 22:23:31.027614 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:31.027577 5196 generic.go:356] "Generic (PLEG): container finished" podID="1a254ea4-03c8-4dbb-904c-a1def38b0d83" containerID="f6efc335d98c7d3dc53605f0743bfbaae9b1e61ee8af7b71caefcaa0591a624e" exitCode=0 Nov 03 22:23:31.027810 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:31.027624 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a254ea4-03c8-4dbb-904c-a1def38b0d83","Type":"ContainerDied","Data":"f6efc335d98c7d3dc53605f0743bfbaae9b1e61ee8af7b71caefcaa0591a624e"} Nov 03 22:23:31.027810 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:31.027685 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a254ea4-03c8-4dbb-904c-a1def38b0d83","Type":"ContainerStarted","Data":"3a553fa03eb14d90e39c1447691faf40c0fdbd65191a2bba54899637e850c756"} Nov 03 22:23:31.207464 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:31.207426 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:31.349211 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:31.349171 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:32.034310 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:32.034238 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a254ea4-03c8-4dbb-904c-a1def38b0d83","Type":"ContainerStarted","Data":"02ace6f696100751df84ba544b914930b7961d87a05cc9ac33a98d344a6e74e2"} Nov 03 22:23:32.034686 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:32.034324 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a254ea4-03c8-4dbb-904c-a1def38b0d83","Type":"ContainerStarted","Data":"3d8a80644fff681682c8184e6716265c4839bade0616f9efc1cd948f6302ae86"} Nov 03 22:23:32.034686 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:32.034355 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a254ea4-03c8-4dbb-904c-a1def38b0d83","Type":"ContainerStarted","Data":"8130b93923df7304f85be9f74bae341f7806edfdc8565ba5daf8fdaee4d0572c"} Nov 03 22:23:32.034686 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:32.034379 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a254ea4-03c8-4dbb-904c-a1def38b0d83","Type":"ContainerStarted","Data":"ffd3f2b9931256f486e18f8881f33f5f6eda7104b33a0bf5b5ff36d20404b1f8"} Nov 03 22:23:32.034686 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:32.034402 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a254ea4-03c8-4dbb-904c-a1def38b0d83","Type":"ContainerStarted","Data":"e8b3341c33aa34e11333a7f7c4ee96a7e085978d10d6a7a01dfda7e3516383b3"} Nov 03 22:23:32.034686 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:32.034427 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a254ea4-03c8-4dbb-904c-a1def38b0d83","Type":"ContainerStarted","Data":"b62e5f387d7fffa09aa2418202d60d574d42a9c24d090d6581f21e6a0e866558"} Nov 03 22:23:32.168229 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:32.168175 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.147187868 podStartE2EDuration="3.16816289s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.975022ms" isStatefulPod=false podCreationTimestamp="2026-11-03 22:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:32.167232027 +0000 UTC m=+41.476791321" watchObservedRunningTime="2026-11-03 22:23:32.16816289 +0000 UTC m=+41.477722183" Nov 03 22:23:34.680072 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:34.680033 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:23:43.268373 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:43.268330 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:44.108654 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:44.108615 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:23:55.916815 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:23:55.916779 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:23:56.282692 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:23:56.282598 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:24:16.243845 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:24:16.243817 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:24:16.259839 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:24:16.259811 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:24:17.191996 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:24:17.191961 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 22:24:18.320502 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:24:18.320465 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:24:18.466416 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:24:18.466380 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:24:25.916307 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:24:25.916260 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:24:31.003004 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:24:31.002912 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:24:31.148501 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:24:31.148461 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:24:39.348664 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:24:39.348629 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:24:39.498515 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:24:39.498488 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:24:42.519610 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:24:42.519568 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:24:42.663214 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:24:42.663169 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:24:43.800165 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:24:43.800128 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:24:43.948852 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:24:43.948807 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:24:44.159222 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:24:44.159192 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:24:44.305884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:24:44.305841 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:24:55.917388 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:24:55.917343 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:24:56.636452 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:24:56.636413 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:25:01.604256 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:25:01.604212 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:25:25.916447 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:25:25.916409 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:25:55.916819 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:25:55.916777 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:25:56.983419 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:25:56.983382 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:25:57.245201 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:25:57.245117 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:26:01.947960 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:01.947881 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:26:25.916777 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:26:25.916740 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:26:52.455448 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:52.455372 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:26:54.463666 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:54.463609 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:26:55.350058 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:55.350010 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rbv5k"] Nov 03 22:26:55.352149 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:55.352132 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rbv5k" Nov 03 22:26:55.463494 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:55.463471 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 22:26:55.463494 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:55.463470 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-7tfgh\"" Nov 03 22:26:55.463494 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:55.463468 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 22:26:55.464131 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:55.464114 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rbv5k"] Nov 03 22:26:55.483957 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:55.483919 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rbv5k\" (UID: \"7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rbv5k" Nov 03 22:26:55.484078 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:55.483985 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rbv5k\" (UID: \"7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rbv5k" Nov 03 22:26:55.484119 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:55.484094 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8ms7\" (UniqueName: \"kubernetes.io/projected/7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c-kube-api-access-p8ms7\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rbv5k\" (UID: \"7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rbv5k" Nov 03 22:26:55.585453 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:55.585423 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rbv5k\" (UID: \"7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rbv5k" Nov 03 22:26:55.585588 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:55.585468 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8ms7\" (UniqueName: \"kubernetes.io/projected/7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c-kube-api-access-p8ms7\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rbv5k\" (UID: \"7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rbv5k" Nov 03 22:26:55.585588 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:55.585491 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rbv5k\" (UID: \"7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rbv5k" Nov 03 22:26:55.585819 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:55.585802 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rbv5k\" (UID: \"7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rbv5k" Nov 03 22:26:55.585891 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:55.585829 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rbv5k\" (UID: \"7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rbv5k" Nov 03 22:26:55.646597 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:55.646507 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:26:55.648497 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:55.648471 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8ms7\" (UniqueName: \"kubernetes.io/projected/7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c-kube-api-access-p8ms7\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rbv5k\" (UID: \"7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rbv5k" Nov 03 22:26:55.664089 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:55.664066 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rbv5k" Nov 03 22:26:55.748685 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:55.748655 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79r6rv"] Nov 03 22:26:55.750968 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:55.750954 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79r6rv" Nov 03 22:26:55.804634 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:55.804611 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-9k8h2\"" Nov 03 22:26:55.812196 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:55.812173 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79r6rv"] Nov 03 22:26:55.842611 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:55.842587 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z47t8"] Nov 03 22:26:55.844621 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:55.844607 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z47t8" Nov 03 22:26:55.888461 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:55.888435 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3041da2a-28ee-437a-a3d0-93c0aad89939-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79r6rv\" (UID: \"3041da2a-28ee-437a-a3d0-93c0aad89939\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79r6rv" Nov 03 22:26:55.888583 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:55.888495 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2b94\" (UniqueName: \"kubernetes.io/projected/3041da2a-28ee-437a-a3d0-93c0aad89939-kube-api-access-k2b94\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79r6rv\" (UID: \"3041da2a-28ee-437a-a3d0-93c0aad89939\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79r6rv" Nov 03 22:26:55.888632 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:55.888582 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3041da2a-28ee-437a-a3d0-93c0aad89939-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79r6rv\" (UID: \"3041da2a-28ee-437a-a3d0-93c0aad89939\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79r6rv" Nov 03 22:26:55.905053 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:55.904993 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-jj5pb\"" Nov 03 22:26:55.906039 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:55.906016 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z47t8"] Nov 03 22:26:55.921209 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:26:55.921184 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:26:55.953460 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:55.953434 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6bvvt"] Nov 03 22:26:55.955532 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:55.955515 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6bvvt" Nov 03 22:26:55.989279 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:55.989258 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2b94\" (UniqueName: \"kubernetes.io/projected/3041da2a-28ee-437a-a3d0-93c0aad89939-kube-api-access-k2b94\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79r6rv\" (UID: \"3041da2a-28ee-437a-a3d0-93c0aad89939\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79r6rv" Nov 03 22:26:55.989384 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:55.989290 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d9f52e4-ae4e-4c9f-a477-fc1f1149791e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z47t8\" (UID: \"9d9f52e4-ae4e-4c9f-a477-fc1f1149791e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z47t8" Nov 03 22:26:55.989384 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:55.989319 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3041da2a-28ee-437a-a3d0-93c0aad89939-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79r6rv\" (UID: \"3041da2a-28ee-437a-a3d0-93c0aad89939\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79r6rv" Nov 03 22:26:55.989384 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:55.989343 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7l5j\" (UniqueName: \"kubernetes.io/projected/9d9f52e4-ae4e-4c9f-a477-fc1f1149791e-kube-api-access-b7l5j\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z47t8\" (UID: \"9d9f52e4-ae4e-4c9f-a477-fc1f1149791e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z47t8" Nov 03 22:26:55.989511 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:55.989390 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3041da2a-28ee-437a-a3d0-93c0aad89939-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79r6rv\" (UID: \"3041da2a-28ee-437a-a3d0-93c0aad89939\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79r6rv" Nov 03 22:26:55.989511 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:55.989474 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d9f52e4-ae4e-4c9f-a477-fc1f1149791e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z47t8\" (UID: \"9d9f52e4-ae4e-4c9f-a477-fc1f1149791e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z47t8" Nov 03 22:26:55.989701 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:55.989683 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3041da2a-28ee-437a-a3d0-93c0aad89939-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79r6rv\" (UID: \"3041da2a-28ee-437a-a3d0-93c0aad89939\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79r6rv" Nov 03 22:26:55.989743 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:55.989686 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3041da2a-28ee-437a-a3d0-93c0aad89939-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79r6rv\" (UID: \"3041da2a-28ee-437a-a3d0-93c0aad89939\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79r6rv" Nov 03 22:26:56.009231 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.009212 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-4rmt2\"" Nov 03 22:26:56.016483 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.016463 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6bvvt"] Nov 03 22:26:56.051120 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.051091 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2b94\" (UniqueName: \"kubernetes.io/projected/3041da2a-28ee-437a-a3d0-93c0aad89939-kube-api-access-k2b94\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79r6rv\" (UID: \"3041da2a-28ee-437a-a3d0-93c0aad89939\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79r6rv" Nov 03 22:26:56.053772 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.053750 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awrzmb"] Nov 03 22:26:56.055939 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.055920 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awrzmb" Nov 03 22:26:56.064175 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.064157 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79r6rv" Nov 03 22:26:56.079403 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.079380 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rbv5k"] Nov 03 22:26:56.082245 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:26:56.082223 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f307c0c_e2e6_42f1_92e3_d6c0b2ba876c.slice/crio-68f5065019543de9d1d48bbfc1b60be16b5c5833c6ba5ca8d43b4c33fbd1193c WatchSource:0}: Error finding container 68f5065019543de9d1d48bbfc1b60be16b5c5833c6ba5ca8d43b4c33fbd1193c: Status 404 returned error can't find the container with id 68f5065019543de9d1d48bbfc1b60be16b5c5833c6ba5ca8d43b4c33fbd1193c Nov 03 22:26:56.090325 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.090298 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqpv6\" (UniqueName: \"kubernetes.io/projected/6ece4112-1ac1-45a7-bd87-0361364a08a6-kube-api-access-kqpv6\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6bvvt\" (UID: \"6ece4112-1ac1-45a7-bd87-0361364a08a6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6bvvt" Nov 03 22:26:56.090434 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.090335 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7l5j\" (UniqueName: \"kubernetes.io/projected/9d9f52e4-ae4e-4c9f-a477-fc1f1149791e-kube-api-access-b7l5j\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z47t8\" (UID: \"9d9f52e4-ae4e-4c9f-a477-fc1f1149791e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z47t8" Nov 03 22:26:56.090434 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.090376 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d9f52e4-ae4e-4c9f-a477-fc1f1149791e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z47t8\" (UID: \"9d9f52e4-ae4e-4c9f-a477-fc1f1149791e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z47t8" Nov 03 22:26:56.090434 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.090418 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ece4112-1ac1-45a7-bd87-0361364a08a6-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6bvvt\" (UID: \"6ece4112-1ac1-45a7-bd87-0361364a08a6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6bvvt" Nov 03 22:26:56.090620 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.090475 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ece4112-1ac1-45a7-bd87-0361364a08a6-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6bvvt\" (UID: \"6ece4112-1ac1-45a7-bd87-0361364a08a6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6bvvt" Nov 03 22:26:56.090620 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.090496 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d9f52e4-ae4e-4c9f-a477-fc1f1149791e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z47t8\" (UID: \"9d9f52e4-ae4e-4c9f-a477-fc1f1149791e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z47t8" Nov 03 22:26:56.090829 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.090808 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d9f52e4-ae4e-4c9f-a477-fc1f1149791e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z47t8\" (UID: \"9d9f52e4-ae4e-4c9f-a477-fc1f1149791e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z47t8" Nov 03 22:26:56.090916 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.090835 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d9f52e4-ae4e-4c9f-a477-fc1f1149791e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z47t8\" (UID: \"9d9f52e4-ae4e-4c9f-a477-fc1f1149791e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z47t8" Nov 03 22:26:56.111932 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.111909 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-qst65\"" Nov 03 22:26:56.119885 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.118590 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awrzmb"] Nov 03 22:26:56.152638 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.152611 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7l5j\" (UniqueName: \"kubernetes.io/projected/9d9f52e4-ae4e-4c9f-a477-fc1f1149791e-kube-api-access-b7l5j\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z47t8\" (UID: \"9d9f52e4-ae4e-4c9f-a477-fc1f1149791e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z47t8" Nov 03 22:26:56.156471 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.156432 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z47t8" Nov 03 22:26:56.191835 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.191808 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqpv6\" (UniqueName: \"kubernetes.io/projected/6ece4112-1ac1-45a7-bd87-0361364a08a6-kube-api-access-kqpv6\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6bvvt\" (UID: \"6ece4112-1ac1-45a7-bd87-0361364a08a6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6bvvt" Nov 03 22:26:56.191972 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.191846 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e48ce008-81c3-4b1d-b67c-ea51fd6f0279-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awrzmb\" (UID: \"e48ce008-81c3-4b1d-b67c-ea51fd6f0279\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awrzmb" Nov 03 22:26:56.191972 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.191914 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e48ce008-81c3-4b1d-b67c-ea51fd6f0279-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awrzmb\" (UID: \"e48ce008-81c3-4b1d-b67c-ea51fd6f0279\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awrzmb" Nov 03 22:26:56.192080 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.192066 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ece4112-1ac1-45a7-bd87-0361364a08a6-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6bvvt\" (UID: \"6ece4112-1ac1-45a7-bd87-0361364a08a6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6bvvt" Nov 03 22:26:56.192132 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.192097 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tns9k\" (UniqueName: \"kubernetes.io/projected/e48ce008-81c3-4b1d-b67c-ea51fd6f0279-kube-api-access-tns9k\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awrzmb\" (UID: \"e48ce008-81c3-4b1d-b67c-ea51fd6f0279\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awrzmb" Nov 03 22:26:56.192203 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.192182 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ece4112-1ac1-45a7-bd87-0361364a08a6-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6bvvt\" (UID: \"6ece4112-1ac1-45a7-bd87-0361364a08a6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6bvvt" Nov 03 22:26:56.192421 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.192399 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ece4112-1ac1-45a7-bd87-0361364a08a6-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6bvvt\" (UID: \"6ece4112-1ac1-45a7-bd87-0361364a08a6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6bvvt" Nov 03 22:26:56.192491 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.192442 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ece4112-1ac1-45a7-bd87-0361364a08a6-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6bvvt\" (UID: \"6ece4112-1ac1-45a7-bd87-0361364a08a6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6bvvt" Nov 03 22:26:56.254014 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.253989 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqpv6\" (UniqueName: \"kubernetes.io/projected/6ece4112-1ac1-45a7-bd87-0361364a08a6-kube-api-access-kqpv6\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6bvvt\" (UID: \"6ece4112-1ac1-45a7-bd87-0361364a08a6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6bvvt" Nov 03 22:26:56.267931 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.267912 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6bvvt" Nov 03 22:26:56.293259 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.293232 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e48ce008-81c3-4b1d-b67c-ea51fd6f0279-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awrzmb\" (UID: \"e48ce008-81c3-4b1d-b67c-ea51fd6f0279\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awrzmb" Nov 03 22:26:56.293396 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.293276 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e48ce008-81c3-4b1d-b67c-ea51fd6f0279-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awrzmb\" (UID: \"e48ce008-81c3-4b1d-b67c-ea51fd6f0279\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awrzmb" Nov 03 22:26:56.293396 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.293308 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tns9k\" (UniqueName: \"kubernetes.io/projected/e48ce008-81c3-4b1d-b67c-ea51fd6f0279-kube-api-access-tns9k\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awrzmb\" (UID: \"e48ce008-81c3-4b1d-b67c-ea51fd6f0279\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awrzmb" Nov 03 22:26:56.293659 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.293634 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e48ce008-81c3-4b1d-b67c-ea51fd6f0279-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awrzmb\" (UID: \"e48ce008-81c3-4b1d-b67c-ea51fd6f0279\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awrzmb" Nov 03 22:26:56.293731 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.293640 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e48ce008-81c3-4b1d-b67c-ea51fd6f0279-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awrzmb\" (UID: \"e48ce008-81c3-4b1d-b67c-ea51fd6f0279\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awrzmb" Nov 03 22:26:56.355758 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.355731 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tns9k\" (UniqueName: \"kubernetes.io/projected/e48ce008-81c3-4b1d-b67c-ea51fd6f0279-kube-api-access-tns9k\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awrzmb\" (UID: \"e48ce008-81c3-4b1d-b67c-ea51fd6f0279\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awrzmb" Nov 03 22:26:56.368459 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.368436 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awrzmb" Nov 03 22:26:56.439548 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.439515 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79r6rv"] Nov 03 22:26:56.442614 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:26:56.442588 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3041da2a_28ee_437a_a3d0_93c0aad89939.slice/crio-4e4addc45cde85a59022cc82bf25f93bc3f1df2ce91e756db312a0ad1214ab23 WatchSource:0}: Error finding container 4e4addc45cde85a59022cc82bf25f93bc3f1df2ce91e756db312a0ad1214ab23: Status 404 returned error can't find the container with id 4e4addc45cde85a59022cc82bf25f93bc3f1df2ce91e756db312a0ad1214ab23 Nov 03 22:26:56.546194 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.546174 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z47t8"] Nov 03 22:26:56.548426 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:26:56.548405 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d9f52e4_ae4e_4c9f_a477_fc1f1149791e.slice/crio-97d640211e1d03e21eb9bf049461cf7f4522e767df8a8b7cb441c0a17fc93cb9 WatchSource:0}: Error finding container 97d640211e1d03e21eb9bf049461cf7f4522e767df8a8b7cb441c0a17fc93cb9: Status 404 returned error can't find the container with id 97d640211e1d03e21eb9bf049461cf7f4522e767df8a8b7cb441c0a17fc93cb9 Nov 03 22:26:56.640615 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.640590 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6bvvt"] Nov 03 22:26:56.645701 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:26:56.645103 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ece4112_1ac1_45a7_bd87_0361364a08a6.slice/crio-1e7259f3c262aaf2d4a097eb8b49ef3bdb12a727e389e4a1520927b5b4f714db WatchSource:0}: Error finding container 1e7259f3c262aaf2d4a097eb8b49ef3bdb12a727e389e4a1520927b5b4f714db: Status 404 returned error can't find the container with id 1e7259f3c262aaf2d4a097eb8b49ef3bdb12a727e389e4a1520927b5b4f714db Nov 03 22:26:56.678803 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.678729 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rbv5k" event={"ID":"7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c","Type":"ContainerStarted","Data":"68f5065019543de9d1d48bbfc1b60be16b5c5833c6ba5ca8d43b4c33fbd1193c"} Nov 03 22:26:56.679908 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.679885 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79r6rv" event={"ID":"3041da2a-28ee-437a-a3d0-93c0aad89939","Type":"ContainerStarted","Data":"4e4addc45cde85a59022cc82bf25f93bc3f1df2ce91e756db312a0ad1214ab23"} Nov 03 22:26:56.680956 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.680935 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6bvvt" event={"ID":"6ece4112-1ac1-45a7-bd87-0361364a08a6","Type":"ContainerStarted","Data":"1e7259f3c262aaf2d4a097eb8b49ef3bdb12a727e389e4a1520927b5b4f714db"} Nov 03 22:26:56.681949 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.681930 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z47t8" event={"ID":"9d9f52e4-ae4e-4c9f-a477-fc1f1149791e","Type":"ContainerStarted","Data":"97d640211e1d03e21eb9bf049461cf7f4522e767df8a8b7cb441c0a17fc93cb9"} Nov 03 22:26:56.719576 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.719551 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctrt6k"] Nov 03 22:26:56.722410 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.722394 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctrt6k" Nov 03 22:26:56.744596 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.744574 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awrzmb"] Nov 03 22:26:56.747481 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:26:56.747459 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode48ce008_81c3_4b1d_b67c_ea51fd6f0279.slice/crio-bcc7584183fba15d239bfc8e86a810f9f39cb33cdb297f14fafaa8eb6b83a23d WatchSource:0}: Error finding container bcc7584183fba15d239bfc8e86a810f9f39cb33cdb297f14fafaa8eb6b83a23d: Status 404 returned error can't find the container with id bcc7584183fba15d239bfc8e86a810f9f39cb33cdb297f14fafaa8eb6b83a23d Nov 03 22:26:56.775775 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.775755 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-kw9n8\"" Nov 03 22:26:56.781841 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.781818 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctrt6k"] Nov 03 22:26:56.796501 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.796482 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5a56ae5-b5db-4e0e-b9f7-2b452d333c86-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctrt6k\" (UID: \"b5a56ae5-b5db-4e0e-b9f7-2b452d333c86\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctrt6k" Nov 03 22:26:56.796608 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.796519 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5a56ae5-b5db-4e0e-b9f7-2b452d333c86-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctrt6k\" (UID: \"b5a56ae5-b5db-4e0e-b9f7-2b452d333c86\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctrt6k" Nov 03 22:26:56.796608 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.796555 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcpbg\" (UniqueName: \"kubernetes.io/projected/b5a56ae5-b5db-4e0e-b9f7-2b452d333c86-kube-api-access-bcpbg\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctrt6k\" (UID: \"b5a56ae5-b5db-4e0e-b9f7-2b452d333c86\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctrt6k" Nov 03 22:26:56.897373 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.897345 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5a56ae5-b5db-4e0e-b9f7-2b452d333c86-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctrt6k\" (UID: \"b5a56ae5-b5db-4e0e-b9f7-2b452d333c86\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctrt6k" Nov 03 22:26:56.897514 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.897384 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5a56ae5-b5db-4e0e-b9f7-2b452d333c86-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctrt6k\" (UID: \"b5a56ae5-b5db-4e0e-b9f7-2b452d333c86\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctrt6k" Nov 03 22:26:56.897514 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.897409 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcpbg\" (UniqueName: \"kubernetes.io/projected/b5a56ae5-b5db-4e0e-b9f7-2b452d333c86-kube-api-access-bcpbg\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctrt6k\" (UID: \"b5a56ae5-b5db-4e0e-b9f7-2b452d333c86\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctrt6k" Nov 03 22:26:56.897725 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.897706 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5a56ae5-b5db-4e0e-b9f7-2b452d333c86-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctrt6k\" (UID: \"b5a56ae5-b5db-4e0e-b9f7-2b452d333c86\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctrt6k" Nov 03 22:26:56.897793 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.897757 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5a56ae5-b5db-4e0e-b9f7-2b452d333c86-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctrt6k\" (UID: \"b5a56ae5-b5db-4e0e-b9f7-2b452d333c86\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctrt6k" Nov 03 22:26:56.959820 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:56.959753 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcpbg\" (UniqueName: \"kubernetes.io/projected/b5a56ae5-b5db-4e0e-b9f7-2b452d333c86-kube-api-access-bcpbg\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctrt6k\" (UID: \"b5a56ae5-b5db-4e0e-b9f7-2b452d333c86\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctrt6k" Nov 03 22:26:57.034454 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:57.034419 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctrt6k" Nov 03 22:26:57.415789 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:57.415747 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctrt6k"] Nov 03 22:26:57.418149 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:57.418117 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:26:57.419341 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:26:57.419313 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a56ae5_b5db_4e0e_b9f7_2b452d333c86.slice/crio-6153c4e733678e0332267af422e8f782a30b26e09b0d31441770b80b04581129 WatchSource:0}: Error finding container 6153c4e733678e0332267af422e8f782a30b26e09b0d31441770b80b04581129: Status 404 returned error can't find the container with id 6153c4e733678e0332267af422e8f782a30b26e09b0d31441770b80b04581129 Nov 03 22:26:57.688493 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:57.688409 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awrzmb" event={"ID":"e48ce008-81c3-4b1d-b67c-ea51fd6f0279","Type":"ContainerStarted","Data":"bcc7584183fba15d239bfc8e86a810f9f39cb33cdb297f14fafaa8eb6b83a23d"} Nov 03 22:26:57.691473 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:57.691443 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctrt6k" event={"ID":"b5a56ae5-b5db-4e0e-b9f7-2b452d333c86","Type":"ContainerStarted","Data":"6153c4e733678e0332267af422e8f782a30b26e09b0d31441770b80b04581129"} Nov 03 22:26:59.816233 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:59.816179 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:26:59.881616 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:59.880050 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wqwdz"] Nov 03 22:26:59.882819 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:59.882796 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wqwdz" Nov 03 22:26:59.962397 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:59.962353 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:26:59.988146 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:59.988113 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wqwdz"] Nov 03 22:26:59.988304 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:26:59.988271 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-rsbsr\"" Nov 03 22:27:00.026364 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.026315 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bgpk\" (UniqueName: \"kubernetes.io/projected/4e0857a6-bb8a-4563-91c1-4e3ff43d1078-kube-api-access-4bgpk\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wqwdz\" (UID: \"4e0857a6-bb8a-4563-91c1-4e3ff43d1078\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wqwdz" Nov 03 22:27:00.026518 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.026399 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e0857a6-bb8a-4563-91c1-4e3ff43d1078-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wqwdz\" (UID: \"4e0857a6-bb8a-4563-91c1-4e3ff43d1078\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wqwdz" Nov 03 22:27:00.026580 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.026533 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e0857a6-bb8a-4563-91c1-4e3ff43d1078-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wqwdz\" (UID: \"4e0857a6-bb8a-4563-91c1-4e3ff43d1078\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wqwdz" Nov 03 22:27:00.127952 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.127916 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e0857a6-bb8a-4563-91c1-4e3ff43d1078-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wqwdz\" (UID: \"4e0857a6-bb8a-4563-91c1-4e3ff43d1078\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wqwdz" Nov 03 22:27:00.128121 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.127999 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bgpk\" (UniqueName: \"kubernetes.io/projected/4e0857a6-bb8a-4563-91c1-4e3ff43d1078-kube-api-access-4bgpk\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wqwdz\" (UID: \"4e0857a6-bb8a-4563-91c1-4e3ff43d1078\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wqwdz" Nov 03 22:27:00.128121 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.128043 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e0857a6-bb8a-4563-91c1-4e3ff43d1078-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wqwdz\" (UID: \"4e0857a6-bb8a-4563-91c1-4e3ff43d1078\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wqwdz" Nov 03 22:27:00.128593 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.128568 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e0857a6-bb8a-4563-91c1-4e3ff43d1078-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wqwdz\" (UID: \"4e0857a6-bb8a-4563-91c1-4e3ff43d1078\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wqwdz" Nov 03 22:27:00.128815 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.128795 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e0857a6-bb8a-4563-91c1-4e3ff43d1078-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wqwdz\" (UID: \"4e0857a6-bb8a-4563-91c1-4e3ff43d1078\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wqwdz" Nov 03 22:27:00.214794 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.214764 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bgpk\" (UniqueName: \"kubernetes.io/projected/4e0857a6-bb8a-4563-91c1-4e3ff43d1078-kube-api-access-4bgpk\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wqwdz\" (UID: \"4e0857a6-bb8a-4563-91c1-4e3ff43d1078\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wqwdz" Nov 03 22:27:00.314793 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.314761 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk"] Nov 03 22:27:00.317460 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.317436 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:27:00.382418 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.371439 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 22:27:00.382418 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.371527 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-89xv7\"" Nov 03 22:27:00.382418 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.372148 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 22:27:00.382418 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.372213 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 22:27:00.410764 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.410723 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:00.429749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.429700 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk"] Nov 03 22:27:00.434098 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.433666 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa3310c5-bd35-4bef-b838-1722c7143cf5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nqmgk\" (UID: \"aa3310c5-bd35-4bef-b838-1722c7143cf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:27:00.434098 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.433704 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa3310c5-bd35-4bef-b838-1722c7143cf5-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nqmgk\" (UID: \"aa3310c5-bd35-4bef-b838-1722c7143cf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:27:00.434098 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.433730 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3310c5-bd35-4bef-b838-1722c7143cf5-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nqmgk\" (UID: \"aa3310c5-bd35-4bef-b838-1722c7143cf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:27:00.434098 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.433745 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa3310c5-bd35-4bef-b838-1722c7143cf5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nqmgk\" (UID: \"aa3310c5-bd35-4bef-b838-1722c7143cf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:27:00.434098 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.433764 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbbw2\" (UniqueName: \"kubernetes.io/projected/aa3310c5-bd35-4bef-b838-1722c7143cf5-kube-api-access-nbbw2\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nqmgk\" (UID: \"aa3310c5-bd35-4bef-b838-1722c7143cf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:27:00.434098 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.433793 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3310c5-bd35-4bef-b838-1722c7143cf5-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nqmgk\" (UID: \"aa3310c5-bd35-4bef-b838-1722c7143cf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:27:00.434098 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.433808 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa3310c5-bd35-4bef-b838-1722c7143cf5-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nqmgk\" (UID: \"aa3310c5-bd35-4bef-b838-1722c7143cf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:27:00.434098 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.433826 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa3310c5-bd35-4bef-b838-1722c7143cf5-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nqmgk\" (UID: \"aa3310c5-bd35-4bef-b838-1722c7143cf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:27:00.434098 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.433847 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa3310c5-bd35-4bef-b838-1722c7143cf5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nqmgk\" (UID: \"aa3310c5-bd35-4bef-b838-1722c7143cf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:27:00.502551 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.502308 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wqwdz" Nov 03 22:27:00.536148 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.534354 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa3310c5-bd35-4bef-b838-1722c7143cf5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nqmgk\" (UID: \"aa3310c5-bd35-4bef-b838-1722c7143cf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:27:00.536148 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.534401 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa3310c5-bd35-4bef-b838-1722c7143cf5-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nqmgk\" (UID: \"aa3310c5-bd35-4bef-b838-1722c7143cf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:27:00.536148 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.534439 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3310c5-bd35-4bef-b838-1722c7143cf5-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nqmgk\" (UID: \"aa3310c5-bd35-4bef-b838-1722c7143cf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:27:00.536148 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.534463 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa3310c5-bd35-4bef-b838-1722c7143cf5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nqmgk\" (UID: \"aa3310c5-bd35-4bef-b838-1722c7143cf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:27:00.536148 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.534492 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbbw2\" (UniqueName: \"kubernetes.io/projected/aa3310c5-bd35-4bef-b838-1722c7143cf5-kube-api-access-nbbw2\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nqmgk\" (UID: \"aa3310c5-bd35-4bef-b838-1722c7143cf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:27:00.536148 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.534540 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3310c5-bd35-4bef-b838-1722c7143cf5-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nqmgk\" (UID: \"aa3310c5-bd35-4bef-b838-1722c7143cf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:27:00.536148 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.534566 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa3310c5-bd35-4bef-b838-1722c7143cf5-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nqmgk\" (UID: \"aa3310c5-bd35-4bef-b838-1722c7143cf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:27:00.536148 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.534590 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa3310c5-bd35-4bef-b838-1722c7143cf5-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nqmgk\" (UID: \"aa3310c5-bd35-4bef-b838-1722c7143cf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:27:00.536148 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.534623 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa3310c5-bd35-4bef-b838-1722c7143cf5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nqmgk\" (UID: \"aa3310c5-bd35-4bef-b838-1722c7143cf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:27:00.536148 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.535810 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa3310c5-bd35-4bef-b838-1722c7143cf5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nqmgk\" (UID: \"aa3310c5-bd35-4bef-b838-1722c7143cf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:27:00.538148 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.537133 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3310c5-bd35-4bef-b838-1722c7143cf5-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nqmgk\" (UID: \"aa3310c5-bd35-4bef-b838-1722c7143cf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:27:00.538148 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.537467 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3310c5-bd35-4bef-b838-1722c7143cf5-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nqmgk\" (UID: \"aa3310c5-bd35-4bef-b838-1722c7143cf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:27:00.538148 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.537808 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa3310c5-bd35-4bef-b838-1722c7143cf5-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nqmgk\" (UID: \"aa3310c5-bd35-4bef-b838-1722c7143cf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:27:00.538148 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.538051 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa3310c5-bd35-4bef-b838-1722c7143cf5-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nqmgk\" (UID: \"aa3310c5-bd35-4bef-b838-1722c7143cf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:27:00.541366 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.541261 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa3310c5-bd35-4bef-b838-1722c7143cf5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nqmgk\" (UID: \"aa3310c5-bd35-4bef-b838-1722c7143cf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:27:00.542747 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.542727 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa3310c5-bd35-4bef-b838-1722c7143cf5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nqmgk\" (UID: \"aa3310c5-bd35-4bef-b838-1722c7143cf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:27:00.652683 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.652602 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbbw2\" (UniqueName: \"kubernetes.io/projected/aa3310c5-bd35-4bef-b838-1722c7143cf5-kube-api-access-nbbw2\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nqmgk\" (UID: \"aa3310c5-bd35-4bef-b838-1722c7143cf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:27:00.653278 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.653234 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa3310c5-bd35-4bef-b838-1722c7143cf5-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nqmgk\" (UID: \"aa3310c5-bd35-4bef-b838-1722c7143cf5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:27:00.697715 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.697611 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:00.891022 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.890942 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:00.934968 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.934934 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:27:00.977996 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:00.977915 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wqwdz"] Nov 03 22:27:00.986330 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:27:00.986299 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e0857a6_bb8a_4563_91c1_4e3ff43d1078.slice/crio-0e5daf0249b99eab84dd41e351a36d274f84af3508db78c2d9c91079606f567d WatchSource:0}: Error finding container 0e5daf0249b99eab84dd41e351a36d274f84af3508db78c2d9c91079606f567d: Status 404 returned error can't find the container with id 0e5daf0249b99eab84dd41e351a36d274f84af3508db78c2d9c91079606f567d Nov 03 22:27:01.036976 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:01.036937 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:01.424243 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:01.424208 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk"] Nov 03 22:27:01.435114 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:27:01.435051 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa3310c5_bd35_4bef_b838_1722c7143cf5.slice/crio-a2eb689fb6747f6a72b074f6587e3a0ad284188879a841eaabe0358f13b38053 WatchSource:0}: Error finding container a2eb689fb6747f6a72b074f6587e3a0ad284188879a841eaabe0358f13b38053: Status 404 returned error can't find the container with id a2eb689fb6747f6a72b074f6587e3a0ad284188879a841eaabe0358f13b38053 Nov 03 22:27:01.722466 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:01.722372 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wqwdz" event={"ID":"4e0857a6-bb8a-4563-91c1-4e3ff43d1078","Type":"ContainerStarted","Data":"0e5daf0249b99eab84dd41e351a36d274f84af3508db78c2d9c91079606f567d"} Nov 03 22:27:01.724838 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:01.724803 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" event={"ID":"aa3310c5-bd35-4bef-b838-1722c7143cf5","Type":"ContainerStarted","Data":"a2eb689fb6747f6a72b074f6587e3a0ad284188879a841eaabe0358f13b38053"} Nov 03 22:27:02.296369 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:02.296335 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:04.805948 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:04.805836 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:09.369735 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:09.369700 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:27:09.370148 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:09.369793 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:27:09.370148 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:09.369840 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:27:09.805539 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:09.800042 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" event={"ID":"aa3310c5-bd35-4bef-b838-1722c7143cf5","Type":"ContainerStarted","Data":"94185fe4b0d8ffd8765aacdeece9a13a1b59a1924ca7c60218f604135935965c"} Nov 03 22:27:09.936168 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:09.936136 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:27:09.974167 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:09.974107 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" podStartSLOduration=2.0423048 podStartE2EDuration="9.974090478s" totalImagesPullingTime="7.931785678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:09.973361606 +0000 UTC m=+259.282920909" watchObservedRunningTime="2026-11-03 22:27:09.974090478 +0000 UTC m=+259.283649771" Nov 03 22:27:10.817149 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:10.817115 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rbv5k" event={"ID":"7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c","Type":"ContainerStarted","Data":"96d52aecad24f31d9d5aa90cb38f566a01c76923944bc0d7c20a1281b1b88b25"} Nov 03 22:27:10.943710 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:10.943642 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:27:11.818940 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:11.818912 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z47t8" event={"ID":"9d9f52e4-ae4e-4c9f-a477-fc1f1149791e","Type":"ContainerStarted","Data":"e520b56f787d605270e73633742118383b3689a7c0562b4d61e73ae4dd28ebf6"} Nov 03 22:27:11.820502 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:11.820478 5196 generic.go:356] "Generic (PLEG): container finished" podID="7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c" containerID="96d52aecad24f31d9d5aa90cb38f566a01c76923944bc0d7c20a1281b1b88b25" exitCode=0 Nov 03 22:27:11.820599 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:11.820554 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rbv5k" event={"ID":"7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c","Type":"ContainerDied","Data":"96d52aecad24f31d9d5aa90cb38f566a01c76923944bc0d7c20a1281b1b88b25"} Nov 03 22:27:11.821842 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:11.821823 5196 generic.go:356] "Generic (PLEG): container finished" podID="3041da2a-28ee-437a-a3d0-93c0aad89939" containerID="e679c52f9289424cfcf122dcbb8de3a6e4b2010c61aa0990fa93683cd83c4494" exitCode=0 Nov 03 22:27:11.821906 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:11.821890 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79r6rv" event={"ID":"3041da2a-28ee-437a-a3d0-93c0aad89939","Type":"ContainerDied","Data":"e679c52f9289424cfcf122dcbb8de3a6e4b2010c61aa0990fa93683cd83c4494"} Nov 03 22:27:11.823386 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:11.823364 5196 generic.go:356] "Generic (PLEG): container finished" podID="4e0857a6-bb8a-4563-91c1-4e3ff43d1078" containerID="554db8494569502a525e2be3181ab6927033ad5058aedabbf2d4c6c9499f30a2" exitCode=0 Nov 03 22:27:11.823504 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:11.823487 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wqwdz" event={"ID":"4e0857a6-bb8a-4563-91c1-4e3ff43d1078","Type":"ContainerDied","Data":"554db8494569502a525e2be3181ab6927033ad5058aedabbf2d4c6c9499f30a2"} Nov 03 22:27:11.823667 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:11.823655 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:27:11.824775 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:11.824762 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:27:12.828255 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:12.828232 5196 generic.go:356] "Generic (PLEG): container finished" podID="4e0857a6-bb8a-4563-91c1-4e3ff43d1078" containerID="fe6e56674d7a0c03782df1117f1d6e218d00c9eef5db50e3ce20f1086d8ca402" exitCode=0 Nov 03 22:27:12.828567 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:12.828311 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wqwdz" event={"ID":"4e0857a6-bb8a-4563-91c1-4e3ff43d1078","Type":"ContainerDied","Data":"fe6e56674d7a0c03782df1117f1d6e218d00c9eef5db50e3ce20f1086d8ca402"} Nov 03 22:27:12.829911 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:12.829886 5196 generic.go:356] "Generic (PLEG): container finished" podID="6ece4112-1ac1-45a7-bd87-0361364a08a6" containerID="02c4db153d0b877228a8a254f7563f59d8f7b03dc51361e2973a2629d3206ccc" exitCode=0 Nov 03 22:27:12.830021 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:12.829910 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6bvvt" event={"ID":"6ece4112-1ac1-45a7-bd87-0361364a08a6","Type":"ContainerDied","Data":"02c4db153d0b877228a8a254f7563f59d8f7b03dc51361e2973a2629d3206ccc"} Nov 03 22:27:12.831547 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:12.831520 5196 generic.go:356] "Generic (PLEG): container finished" podID="9d9f52e4-ae4e-4c9f-a477-fc1f1149791e" containerID="e520b56f787d605270e73633742118383b3689a7c0562b4d61e73ae4dd28ebf6" exitCode=0 Nov 03 22:27:12.831648 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:12.831611 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z47t8" event={"ID":"9d9f52e4-ae4e-4c9f-a477-fc1f1149791e","Type":"ContainerDied","Data":"e520b56f787d605270e73633742118383b3689a7c0562b4d61e73ae4dd28ebf6"} Nov 03 22:27:12.833207 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:12.833185 5196 generic.go:356] "Generic (PLEG): container finished" podID="e48ce008-81c3-4b1d-b67c-ea51fd6f0279" containerID="a4906741e8545ce516100a7f9597fefd245710f5429d9b46ec71775110c7960b" exitCode=0 Nov 03 22:27:12.833309 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:12.833208 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awrzmb" event={"ID":"e48ce008-81c3-4b1d-b67c-ea51fd6f0279","Type":"ContainerDied","Data":"a4906741e8545ce516100a7f9597fefd245710f5429d9b46ec71775110c7960b"} Nov 03 22:27:12.834975 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:12.834912 5196 generic.go:356] "Generic (PLEG): container finished" podID="b5a56ae5-b5db-4e0e-b9f7-2b452d333c86" containerID="1456d0952f7a9cff3f9ab0564216d3a31fab77fdbc6e9bb7f6313fa7ea4db4a6" exitCode=0 Nov 03 22:27:12.835026 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:12.834988 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctrt6k" event={"ID":"b5a56ae5-b5db-4e0e-b9f7-2b452d333c86","Type":"ContainerDied","Data":"1456d0952f7a9cff3f9ab0564216d3a31fab77fdbc6e9bb7f6313fa7ea4db4a6"} Nov 03 22:27:14.848992 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:14.848823 5196 generic.go:356] "Generic (PLEG): container finished" podID="3041da2a-28ee-437a-a3d0-93c0aad89939" containerID="286404e8a27378b475ef08619bf51578b2241a4f555bda8d2a43917a9899f4eb" exitCode=0 Nov 03 22:27:14.848992 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:14.848887 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79r6rv" event={"ID":"3041da2a-28ee-437a-a3d0-93c0aad89939","Type":"ContainerDied","Data":"286404e8a27378b475ef08619bf51578b2241a4f555bda8d2a43917a9899f4eb"} Nov 03 22:27:14.852032 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:14.852004 5196 generic.go:356] "Generic (PLEG): container finished" podID="6ece4112-1ac1-45a7-bd87-0361364a08a6" containerID="7cbb5bb5aa4b2aba35857b4250b85e421b6ffbd059b9842c66fbe447b7e57fd8" exitCode=0 Nov 03 22:27:14.852155 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:14.852134 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6bvvt" event={"ID":"6ece4112-1ac1-45a7-bd87-0361364a08a6","Type":"ContainerDied","Data":"7cbb5bb5aa4b2aba35857b4250b85e421b6ffbd059b9842c66fbe447b7e57fd8"} Nov 03 22:27:14.854602 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:14.854576 5196 generic.go:356] "Generic (PLEG): container finished" podID="9d9f52e4-ae4e-4c9f-a477-fc1f1149791e" containerID="f6a0f820a36755b6dc3e0e7924116f3b80f4fc10641c1ae47bfc8b42d7eff189" exitCode=0 Nov 03 22:27:14.855241 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:14.854729 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z47t8" event={"ID":"9d9f52e4-ae4e-4c9f-a477-fc1f1149791e","Type":"ContainerDied","Data":"f6a0f820a36755b6dc3e0e7924116f3b80f4fc10641c1ae47bfc8b42d7eff189"} Nov 03 22:27:14.856784 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:14.856765 5196 generic.go:356] "Generic (PLEG): container finished" podID="e48ce008-81c3-4b1d-b67c-ea51fd6f0279" containerID="a5a9a86d408abde34d8539067d64ff9b277146bf3453ff725b220f3160307197" exitCode=0 Nov 03 22:27:14.856893 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:14.856805 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awrzmb" event={"ID":"e48ce008-81c3-4b1d-b67c-ea51fd6f0279","Type":"ContainerDied","Data":"a5a9a86d408abde34d8539067d64ff9b277146bf3453ff725b220f3160307197"} Nov 03 22:27:15.869198 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:15.869160 5196 generic.go:356] "Generic (PLEG): container finished" podID="7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c" containerID="f8fb8da765bd079e6cd7b4787350de3d6efda603674ca2ed141dab7b6988cd51" exitCode=0 Nov 03 22:27:15.869690 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:15.869267 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rbv5k" event={"ID":"7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c","Type":"ContainerDied","Data":"f8fb8da765bd079e6cd7b4787350de3d6efda603674ca2ed141dab7b6988cd51"} Nov 03 22:27:22.919550 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:22.918631 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctrt6k" event={"ID":"b5a56ae5-b5db-4e0e-b9f7-2b452d333c86","Type":"ContainerStarted","Data":"61ad7baeae3c010f2e367e051b43fefb89f5d3870401aec378eafe23333613f4"} Nov 03 22:27:23.927154 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:23.927112 5196 generic.go:356] "Generic (PLEG): container finished" podID="b5a56ae5-b5db-4e0e-b9f7-2b452d333c86" containerID="61ad7baeae3c010f2e367e051b43fefb89f5d3870401aec378eafe23333613f4" exitCode=0 Nov 03 22:27:23.927681 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:23.927233 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctrt6k" event={"ID":"b5a56ae5-b5db-4e0e-b9f7-2b452d333c86","Type":"ContainerDied","Data":"61ad7baeae3c010f2e367e051b43fefb89f5d3870401aec378eafe23333613f4"} Nov 03 22:27:25.945509 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:27:25.945467 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:27:25.953924 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:25.953888 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wqwdz" event={"ID":"4e0857a6-bb8a-4563-91c1-4e3ff43d1078","Type":"ContainerStarted","Data":"de86c5c7176efa6da9443b14e4902eeb57fce61bf0296f65cb2ad51d45182dfd"} Nov 03 22:27:26.076245 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:26.076033 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wqwdz" podStartSLOduration=3.0703236 podStartE2EDuration="27.076010173s" totalImagesPullingTime="24.005686573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:26:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:26.074561724 +0000 UTC m=+275.384121017" watchObservedRunningTime="2026-11-03 22:27:26.076010173 +0000 UTC m=+275.385569467" Nov 03 22:27:26.966221 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:26.966183 5196 generic.go:356] "Generic (PLEG): container finished" podID="4e0857a6-bb8a-4563-91c1-4e3ff43d1078" containerID="de86c5c7176efa6da9443b14e4902eeb57fce61bf0296f65cb2ad51d45182dfd" exitCode=0 Nov 03 22:27:26.966687 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:26.966288 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wqwdz" event={"ID":"4e0857a6-bb8a-4563-91c1-4e3ff43d1078","Type":"ContainerDied","Data":"de86c5c7176efa6da9443b14e4902eeb57fce61bf0296f65cb2ad51d45182dfd"} Nov 03 22:27:28.183833 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:28.183526 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wqwdz" Nov 03 22:27:28.322044 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:28.321457 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e0857a6-bb8a-4563-91c1-4e3ff43d1078-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e0857a6-bb8a-4563-91c1-4e3ff43d1078-bundle\") pod \"4e0857a6-bb8a-4563-91c1-4e3ff43d1078\" (UID: \"4e0857a6-bb8a-4563-91c1-4e3ff43d1078\") " Nov 03 22:27:28.322044 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:28.321515 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e0857a6-bb8a-4563-91c1-4e3ff43d1078-kube-api-access-4bgpk\" (UniqueName: \"kubernetes.io/projected/4e0857a6-bb8a-4563-91c1-4e3ff43d1078-kube-api-access-4bgpk\") pod \"4e0857a6-bb8a-4563-91c1-4e3ff43d1078\" (UID: \"4e0857a6-bb8a-4563-91c1-4e3ff43d1078\") " Nov 03 22:27:28.322044 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:28.321559 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e0857a6-bb8a-4563-91c1-4e3ff43d1078-util\" (UniqueName: \"kubernetes.io/empty-dir/4e0857a6-bb8a-4563-91c1-4e3ff43d1078-util\") pod \"4e0857a6-bb8a-4563-91c1-4e3ff43d1078\" (UID: \"4e0857a6-bb8a-4563-91c1-4e3ff43d1078\") " Nov 03 22:27:28.328081 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:28.328054 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e0857a6-bb8a-4563-91c1-4e3ff43d1078-util" pod "4e0857a6-bb8a-4563-91c1-4e3ff43d1078" (UID: "4e0857a6-bb8a-4563-91c1-4e3ff43d1078"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:27:28.328793 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:28.328772 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e0857a6-bb8a-4563-91c1-4e3ff43d1078-bundle" pod "4e0857a6-bb8a-4563-91c1-4e3ff43d1078" (UID: "4e0857a6-bb8a-4563-91c1-4e3ff43d1078"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:27:28.336638 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:28.336603 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e0857a6-bb8a-4563-91c1-4e3ff43d1078-kube-api-access-4bgpk" pod "4e0857a6-bb8a-4563-91c1-4e3ff43d1078" (UID: "4e0857a6-bb8a-4563-91c1-4e3ff43d1078"). InnerVolumeSpecName "kube-api-access-4bgpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:27:28.423214 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:28.423178 5196 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e0857a6-bb8a-4563-91c1-4e3ff43d1078-bundle\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:27:28.423214 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:28.423217 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bgpk\" (UniqueName: \"kubernetes.io/projected/4e0857a6-bb8a-4563-91c1-4e3ff43d1078-kube-api-access-4bgpk\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:27:28.423418 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:28.423233 5196 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e0857a6-bb8a-4563-91c1-4e3ff43d1078-util\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:27:28.982930 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:28.982122 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wqwdz" event={"ID":"4e0857a6-bb8a-4563-91c1-4e3ff43d1078","Type":"ContainerDied","Data":"0e5daf0249b99eab84dd41e351a36d274f84af3508db78c2d9c91079606f567d"} Nov 03 22:27:28.982930 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:28.982180 5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e5daf0249b99eab84dd41e351a36d274f84af3508db78c2d9c91079606f567d" Nov 03 22:27:28.982930 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:28.982259 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wqwdz" Nov 03 22:27:29.989318 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:29.989048 5196 generic.go:356] "Generic (PLEG): container finished" podID="6ece4112-1ac1-45a7-bd87-0361364a08a6" containerID="06ec430dae681e783c2607ec7477612fa9af62f2c6267a2957c62843b654e872" exitCode=0 Nov 03 22:27:29.989318 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:29.989170 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6bvvt" event={"ID":"6ece4112-1ac1-45a7-bd87-0361364a08a6","Type":"ContainerDied","Data":"06ec430dae681e783c2607ec7477612fa9af62f2c6267a2957c62843b654e872"} Nov 03 22:27:29.991886 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:29.991835 5196 generic.go:356] "Generic (PLEG): container finished" podID="9d9f52e4-ae4e-4c9f-a477-fc1f1149791e" containerID="e512db1db36f588d9b0e3ec4557683ecd40c97a8d2124f1e4e27d0d836354883" exitCode=0 Nov 03 22:27:29.992014 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:29.991895 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z47t8" event={"ID":"9d9f52e4-ae4e-4c9f-a477-fc1f1149791e","Type":"ContainerDied","Data":"e512db1db36f588d9b0e3ec4557683ecd40c97a8d2124f1e4e27d0d836354883"} Nov 03 22:27:29.997476 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:29.997447 5196 generic.go:356] "Generic (PLEG): container finished" podID="e48ce008-81c3-4b1d-b67c-ea51fd6f0279" containerID="4a6f5e869771c19ccc332002a5dbae10ee64d6264026d50e18023ce6af6b8783" exitCode=0 Nov 03 22:27:29.997873 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:29.997836 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awrzmb" event={"ID":"e48ce008-81c3-4b1d-b67c-ea51fd6f0279","Type":"ContainerDied","Data":"4a6f5e869771c19ccc332002a5dbae10ee64d6264026d50e18023ce6af6b8783"} Nov 03 22:27:30.000358 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:30.000333 5196 generic.go:356] "Generic (PLEG): container finished" podID="3041da2a-28ee-437a-a3d0-93c0aad89939" containerID="6acf34d64702c9a8031d861af1e92f59e269e0c93db853ef2f6ad90716a050df" exitCode=0 Nov 03 22:27:30.000455 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:30.000391 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79r6rv" event={"ID":"3041da2a-28ee-437a-a3d0-93c0aad89939","Type":"ContainerDied","Data":"6acf34d64702c9a8031d861af1e92f59e269e0c93db853ef2f6ad90716a050df"} Nov 03 22:27:30.473939 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:30.472937 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rbv5k" podStartSLOduration=3.726477022 podStartE2EDuration="35.472918704s" totalImagesPullingTime="31.746441682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:26:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:30.472100667 +0000 UTC m=+279.781659984" watchObservedRunningTime="2026-11-03 22:27:30.472918704 +0000 UTC m=+279.782477999" Nov 03 22:27:31.006231 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.006203 5196 generic.go:356] "Generic (PLEG): container finished" podID="7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c" containerID="8ee8ac119d5450ed023d3f149f3771e2c29c2932de69561e3403f7e9502c73bd" exitCode=0 Nov 03 22:27:31.006648 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.006275 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rbv5k" event={"ID":"7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c","Type":"ContainerDied","Data":"8ee8ac119d5450ed023d3f149f3771e2c29c2932de69561e3403f7e9502c73bd"} Nov 03 22:27:31.008388 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.008348 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctrt6k" event={"ID":"b5a56ae5-b5db-4e0e-b9f7-2b452d333c86","Type":"ContainerStarted","Data":"27a7494f69691d9767f83344cd4f20753705613d0e60d7b2f6f82bb940f40c5b"} Nov 03 22:27:31.142736 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.142713 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79r6rv" Nov 03 22:27:31.176619 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.176589 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awrzmb" Nov 03 22:27:31.197273 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.197249 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6bvvt" Nov 03 22:27:31.203891 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.203875 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z47t8" Nov 03 22:27:31.255144 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.255116 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3041da2a-28ee-437a-a3d0-93c0aad89939-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3041da2a-28ee-437a-a3d0-93c0aad89939-bundle\") pod \"3041da2a-28ee-437a-a3d0-93c0aad89939\" (UID: \"3041da2a-28ee-437a-a3d0-93c0aad89939\") " Nov 03 22:27:31.255282 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.255155 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3041da2a-28ee-437a-a3d0-93c0aad89939-kube-api-access-k2b94\" (UniqueName: \"kubernetes.io/projected/3041da2a-28ee-437a-a3d0-93c0aad89939-kube-api-access-k2b94\") pod \"3041da2a-28ee-437a-a3d0-93c0aad89939\" (UID: \"3041da2a-28ee-437a-a3d0-93c0aad89939\") " Nov 03 22:27:31.255282 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.255200 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ece4112-1ac1-45a7-bd87-0361364a08a6-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ece4112-1ac1-45a7-bd87-0361364a08a6-bundle\") pod \"6ece4112-1ac1-45a7-bd87-0361364a08a6\" (UID: \"6ece4112-1ac1-45a7-bd87-0361364a08a6\") " Nov 03 22:27:31.255282 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.255235 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ece4112-1ac1-45a7-bd87-0361364a08a6-kube-api-access-kqpv6\" (UniqueName: \"kubernetes.io/projected/6ece4112-1ac1-45a7-bd87-0361364a08a6-kube-api-access-kqpv6\") pod \"6ece4112-1ac1-45a7-bd87-0361364a08a6\" (UID: \"6ece4112-1ac1-45a7-bd87-0361364a08a6\") " Nov 03 22:27:31.255282 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.255274 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e48ce008-81c3-4b1d-b67c-ea51fd6f0279-kube-api-access-tns9k\" (UniqueName: \"kubernetes.io/projected/e48ce008-81c3-4b1d-b67c-ea51fd6f0279-kube-api-access-tns9k\") pod \"e48ce008-81c3-4b1d-b67c-ea51fd6f0279\" (UID: \"e48ce008-81c3-4b1d-b67c-ea51fd6f0279\") " Nov 03 22:27:31.255505 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.255308 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d9f52e4-ae4e-4c9f-a477-fc1f1149791e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d9f52e4-ae4e-4c9f-a477-fc1f1149791e-bundle\") pod \"9d9f52e4-ae4e-4c9f-a477-fc1f1149791e\" (UID: \"9d9f52e4-ae4e-4c9f-a477-fc1f1149791e\") " Nov 03 22:27:31.255505 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.255332 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e48ce008-81c3-4b1d-b67c-ea51fd6f0279-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e48ce008-81c3-4b1d-b67c-ea51fd6f0279-bundle\") pod \"e48ce008-81c3-4b1d-b67c-ea51fd6f0279\" (UID: \"e48ce008-81c3-4b1d-b67c-ea51fd6f0279\") " Nov 03 22:27:31.255505 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.255358 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d9f52e4-ae4e-4c9f-a477-fc1f1149791e-kube-api-access-b7l5j\" (UniqueName: \"kubernetes.io/projected/9d9f52e4-ae4e-4c9f-a477-fc1f1149791e-kube-api-access-b7l5j\") pod \"9d9f52e4-ae4e-4c9f-a477-fc1f1149791e\" (UID: \"9d9f52e4-ae4e-4c9f-a477-fc1f1149791e\") " Nov 03 22:27:31.255505 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.255425 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d9f52e4-ae4e-4c9f-a477-fc1f1149791e-util\" (UniqueName: \"kubernetes.io/empty-dir/9d9f52e4-ae4e-4c9f-a477-fc1f1149791e-util\") pod \"9d9f52e4-ae4e-4c9f-a477-fc1f1149791e\" (UID: \"9d9f52e4-ae4e-4c9f-a477-fc1f1149791e\") " Nov 03 22:27:31.255505 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.255470 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e48ce008-81c3-4b1d-b67c-ea51fd6f0279-util\" (UniqueName: \"kubernetes.io/empty-dir/e48ce008-81c3-4b1d-b67c-ea51fd6f0279-util\") pod \"e48ce008-81c3-4b1d-b67c-ea51fd6f0279\" (UID: \"e48ce008-81c3-4b1d-b67c-ea51fd6f0279\") " Nov 03 22:27:31.255733 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.255523 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ece4112-1ac1-45a7-bd87-0361364a08a6-util\" (UniqueName: \"kubernetes.io/empty-dir/6ece4112-1ac1-45a7-bd87-0361364a08a6-util\") pod \"6ece4112-1ac1-45a7-bd87-0361364a08a6\" (UID: \"6ece4112-1ac1-45a7-bd87-0361364a08a6\") " Nov 03 22:27:31.255733 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.255568 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3041da2a-28ee-437a-a3d0-93c0aad89939-util\" (UniqueName: \"kubernetes.io/empty-dir/3041da2a-28ee-437a-a3d0-93c0aad89939-util\") pod \"3041da2a-28ee-437a-a3d0-93c0aad89939\" (UID: \"3041da2a-28ee-437a-a3d0-93c0aad89939\") " Nov 03 22:27:31.255875 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.255832 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3041da2a-28ee-437a-a3d0-93c0aad89939-bundle" pod "3041da2a-28ee-437a-a3d0-93c0aad89939" (UID: "3041da2a-28ee-437a-a3d0-93c0aad89939"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:27:31.255953 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.255907 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d9f52e4-ae4e-4c9f-a477-fc1f1149791e-bundle" pod "9d9f52e4-ae4e-4c9f-a477-fc1f1149791e" (UID: "9d9f52e4-ae4e-4c9f-a477-fc1f1149791e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:27:31.255953 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.255930 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ece4112-1ac1-45a7-bd87-0361364a08a6-bundle" pod "6ece4112-1ac1-45a7-bd87-0361364a08a6" (UID: "6ece4112-1ac1-45a7-bd87-0361364a08a6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:27:31.256696 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.256668 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e48ce008-81c3-4b1d-b67c-ea51fd6f0279-bundle" pod "e48ce008-81c3-4b1d-b67c-ea51fd6f0279" (UID: "e48ce008-81c3-4b1d-b67c-ea51fd6f0279"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:27:31.258196 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.258171 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e48ce008-81c3-4b1d-b67c-ea51fd6f0279-kube-api-access-tns9k" pod "e48ce008-81c3-4b1d-b67c-ea51fd6f0279" (UID: "e48ce008-81c3-4b1d-b67c-ea51fd6f0279"). InnerVolumeSpecName "kube-api-access-tns9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:27:31.258713 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.258689 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ece4112-1ac1-45a7-bd87-0361364a08a6-kube-api-access-kqpv6" pod "6ece4112-1ac1-45a7-bd87-0361364a08a6" (UID: "6ece4112-1ac1-45a7-bd87-0361364a08a6"). InnerVolumeSpecName "kube-api-access-kqpv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:27:31.259414 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.259379 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d9f52e4-ae4e-4c9f-a477-fc1f1149791e-kube-api-access-b7l5j" pod "9d9f52e4-ae4e-4c9f-a477-fc1f1149791e" (UID: "9d9f52e4-ae4e-4c9f-a477-fc1f1149791e"). InnerVolumeSpecName "kube-api-access-b7l5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:27:31.259496 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.259443 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3041da2a-28ee-437a-a3d0-93c0aad89939-kube-api-access-k2b94" pod "3041da2a-28ee-437a-a3d0-93c0aad89939" (UID: "3041da2a-28ee-437a-a3d0-93c0aad89939"). InnerVolumeSpecName "kube-api-access-k2b94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:27:31.261090 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.261067 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3041da2a-28ee-437a-a3d0-93c0aad89939-util" pod "3041da2a-28ee-437a-a3d0-93c0aad89939" (UID: "3041da2a-28ee-437a-a3d0-93c0aad89939"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:27:31.262043 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.262022 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ece4112-1ac1-45a7-bd87-0361364a08a6-util" pod "6ece4112-1ac1-45a7-bd87-0361364a08a6" (UID: "6ece4112-1ac1-45a7-bd87-0361364a08a6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:27:31.262896 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.262877 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d9f52e4-ae4e-4c9f-a477-fc1f1149791e-util" pod "9d9f52e4-ae4e-4c9f-a477-fc1f1149791e" (UID: "9d9f52e4-ae4e-4c9f-a477-fc1f1149791e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:27:31.264799 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.264773 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e48ce008-81c3-4b1d-b67c-ea51fd6f0279-util" pod "e48ce008-81c3-4b1d-b67c-ea51fd6f0279" (UID: "e48ce008-81c3-4b1d-b67c-ea51fd6f0279"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:27:31.276160 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.276112 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctrt6k" podStartSLOduration=3.713031711 podStartE2EDuration="35.276095345s" totalImagesPullingTime="31.563063634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:26:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:31.275493648 +0000 UTC m=+280.585052942" watchObservedRunningTime="2026-11-03 22:27:31.276095345 +0000 UTC m=+280.585654638" Nov 03 22:27:31.357079 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.357059 5196 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3041da2a-28ee-437a-a3d0-93c0aad89939-util\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:27:31.357079 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.357077 5196 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3041da2a-28ee-437a-a3d0-93c0aad89939-bundle\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:27:31.357210 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.357088 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2b94\" (UniqueName: \"kubernetes.io/projected/3041da2a-28ee-437a-a3d0-93c0aad89939-kube-api-access-k2b94\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:27:31.357210 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.357097 5196 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ece4112-1ac1-45a7-bd87-0361364a08a6-bundle\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:27:31.357210 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.357107 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqpv6\" (UniqueName: \"kubernetes.io/projected/6ece4112-1ac1-45a7-bd87-0361364a08a6-kube-api-access-kqpv6\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:27:31.357210 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.357115 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tns9k\" (UniqueName: \"kubernetes.io/projected/e48ce008-81c3-4b1d-b67c-ea51fd6f0279-kube-api-access-tns9k\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:27:31.357210 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.357125 5196 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d9f52e4-ae4e-4c9f-a477-fc1f1149791e-bundle\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:27:31.357210 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.357133 5196 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e48ce008-81c3-4b1d-b67c-ea51fd6f0279-bundle\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:27:31.357210 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.357140 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7l5j\" (UniqueName: \"kubernetes.io/projected/9d9f52e4-ae4e-4c9f-a477-fc1f1149791e-kube-api-access-b7l5j\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:27:31.357210 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.357147 5196 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d9f52e4-ae4e-4c9f-a477-fc1f1149791e-util\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:27:31.357210 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.357155 5196 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e48ce008-81c3-4b1d-b67c-ea51fd6f0279-util\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:27:31.357210 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:31.357162 5196 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ece4112-1ac1-45a7-bd87-0361364a08a6-util\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:27:32.014106 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:32.014070 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79r6rv" event={"ID":"3041da2a-28ee-437a-a3d0-93c0aad89939","Type":"ContainerDied","Data":"4e4addc45cde85a59022cc82bf25f93bc3f1df2ce91e756db312a0ad1214ab23"} Nov 03 22:27:32.014569 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:32.014125 5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e4addc45cde85a59022cc82bf25f93bc3f1df2ce91e756db312a0ad1214ab23" Nov 03 22:27:32.014569 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:32.014080 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79r6rv" Nov 03 22:27:32.016405 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:32.016380 5196 generic.go:356] "Generic (PLEG): container finished" podID="b5a56ae5-b5db-4e0e-b9f7-2b452d333c86" containerID="27a7494f69691d9767f83344cd4f20753705613d0e60d7b2f6f82bb940f40c5b" exitCode=0 Nov 03 22:27:32.016534 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:32.016460 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctrt6k" event={"ID":"b5a56ae5-b5db-4e0e-b9f7-2b452d333c86","Type":"ContainerDied","Data":"27a7494f69691d9767f83344cd4f20753705613d0e60d7b2f6f82bb940f40c5b"} Nov 03 22:27:32.018387 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:32.018370 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6bvvt" Nov 03 22:27:32.018472 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:32.018388 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6bvvt" event={"ID":"6ece4112-1ac1-45a7-bd87-0361364a08a6","Type":"ContainerDied","Data":"1e7259f3c262aaf2d4a097eb8b49ef3bdb12a727e389e4a1520927b5b4f714db"} Nov 03 22:27:32.018472 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:32.018420 5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e7259f3c262aaf2d4a097eb8b49ef3bdb12a727e389e4a1520927b5b4f714db" Nov 03 22:27:32.020498 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:32.020477 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z47t8" event={"ID":"9d9f52e4-ae4e-4c9f-a477-fc1f1149791e","Type":"ContainerDied","Data":"97d640211e1d03e21eb9bf049461cf7f4522e767df8a8b7cb441c0a17fc93cb9"} Nov 03 22:27:32.020589 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:32.020516 5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97d640211e1d03e21eb9bf049461cf7f4522e767df8a8b7cb441c0a17fc93cb9" Nov 03 22:27:32.020589 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:32.020487 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6z47t8" Nov 03 22:27:32.030280 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:32.030254 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awrzmb" event={"ID":"e48ce008-81c3-4b1d-b67c-ea51fd6f0279","Type":"ContainerDied","Data":"bcc7584183fba15d239bfc8e86a810f9f39cb33cdb297f14fafaa8eb6b83a23d"} Nov 03 22:27:32.030365 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:32.030327 5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcc7584183fba15d239bfc8e86a810f9f39cb33cdb297f14fafaa8eb6b83a23d" Nov 03 22:27:32.030443 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:32.030419 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awrzmb" Nov 03 22:27:32.150043 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:32.150021 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rbv5k" Nov 03 22:27:32.266040 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:32.265962 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c-kube-api-access-p8ms7\" (UniqueName: \"kubernetes.io/projected/7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c-kube-api-access-p8ms7\") pod \"7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c\" (UID: \"7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c\") " Nov 03 22:27:32.266177 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:32.266069 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c-bundle\") pod \"7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c\" (UID: \"7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c\") " Nov 03 22:27:32.266177 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:32.266107 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c-util\" (UniqueName: \"kubernetes.io/empty-dir/7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c-util\") pod \"7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c\" (UID: \"7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c\") " Nov 03 22:27:32.266475 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:32.266454 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c-bundle" pod "7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c" (UID: "7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:27:32.268231 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:32.268204 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c-kube-api-access-p8ms7" pod "7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c" (UID: "7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c"). InnerVolumeSpecName "kube-api-access-p8ms7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:27:32.271275 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:32.271254 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c-util" pod "7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c" (UID: "7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:27:32.366988 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:32.366964 5196 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c-bundle\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:27:32.366988 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:32.366985 5196 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c-util\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:27:32.367163 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:32.366995 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8ms7\" (UniqueName: \"kubernetes.io/projected/7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c-kube-api-access-p8ms7\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:27:33.035833 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:33.035796 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rbv5k" Nov 03 22:27:33.035833 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:33.035812 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rbv5k" event={"ID":"7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c","Type":"ContainerDied","Data":"68f5065019543de9d1d48bbfc1b60be16b5c5833c6ba5ca8d43b4c33fbd1193c"} Nov 03 22:27:33.036290 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:33.035875 5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68f5065019543de9d1d48bbfc1b60be16b5c5833c6ba5ca8d43b4c33fbd1193c" Nov 03 22:27:33.154391 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:33.154366 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctrt6k" Nov 03 22:27:33.275070 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:33.275034 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a56ae5-b5db-4e0e-b9f7-2b452d333c86-util\" (UniqueName: \"kubernetes.io/empty-dir/b5a56ae5-b5db-4e0e-b9f7-2b452d333c86-util\") pod \"b5a56ae5-b5db-4e0e-b9f7-2b452d333c86\" (UID: \"b5a56ae5-b5db-4e0e-b9f7-2b452d333c86\") " Nov 03 22:27:33.275267 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:33.275092 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5a56ae5-b5db-4e0e-b9f7-2b452d333c86-kube-api-access-bcpbg\" (UniqueName: \"kubernetes.io/projected/b5a56ae5-b5db-4e0e-b9f7-2b452d333c86-kube-api-access-bcpbg\") pod \"b5a56ae5-b5db-4e0e-b9f7-2b452d333c86\" (UID: \"b5a56ae5-b5db-4e0e-b9f7-2b452d333c86\") " Nov 03 22:27:33.275267 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:33.275149 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a56ae5-b5db-4e0e-b9f7-2b452d333c86-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5a56ae5-b5db-4e0e-b9f7-2b452d333c86-bundle\") pod \"b5a56ae5-b5db-4e0e-b9f7-2b452d333c86\" (UID: \"b5a56ae5-b5db-4e0e-b9f7-2b452d333c86\") " Nov 03 22:27:33.276277 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:33.276247 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a56ae5-b5db-4e0e-b9f7-2b452d333c86-bundle" pod "b5a56ae5-b5db-4e0e-b9f7-2b452d333c86" (UID: "b5a56ae5-b5db-4e0e-b9f7-2b452d333c86"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:27:33.277498 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:33.277466 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a56ae5-b5db-4e0e-b9f7-2b452d333c86-kube-api-access-bcpbg" pod "b5a56ae5-b5db-4e0e-b9f7-2b452d333c86" (UID: "b5a56ae5-b5db-4e0e-b9f7-2b452d333c86"). InnerVolumeSpecName "kube-api-access-bcpbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:27:33.279713 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:33.279689 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a56ae5-b5db-4e0e-b9f7-2b452d333c86-util" pod "b5a56ae5-b5db-4e0e-b9f7-2b452d333c86" (UID: "b5a56ae5-b5db-4e0e-b9f7-2b452d333c86"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:27:33.375722 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:33.375695 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcpbg\" (UniqueName: \"kubernetes.io/projected/b5a56ae5-b5db-4e0e-b9f7-2b452d333c86-kube-api-access-bcpbg\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:27:33.375722 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:33.375722 5196 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5a56ae5-b5db-4e0e-b9f7-2b452d333c86-bundle\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:27:33.375913 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:33.375732 5196 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5a56ae5-b5db-4e0e-b9f7-2b452d333c86-util\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:27:33.468995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:33.468951 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:34.041695 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:34.041668 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctrt6k" Nov 03 22:27:34.042105 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:34.041668 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ctrt6k" event={"ID":"b5a56ae5-b5db-4e0e-b9f7-2b452d333c86","Type":"ContainerDied","Data":"6153c4e733678e0332267af422e8f782a30b26e09b0d31441770b80b04581129"} Nov 03 22:27:34.042105 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:34.041785 5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6153c4e733678e0332267af422e8f782a30b26e09b0d31441770b80b04581129" Nov 03 22:27:36.497032 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.496999 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-dr2qp"] Nov 03 22:27:36.497408 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497268 5196 cpu_manager.go:414] "containerMap: removing container" podUID="b5a56ae5-b5db-4e0e-b9f7-2b452d333c86" containerName="extract" Nov 03 22:27:36.497408 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497279 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5a56ae5-b5db-4e0e-b9f7-2b452d333c86" containerName="extract" Nov 03 22:27:36.497408 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497294 5196 cpu_manager.go:414] "containerMap: removing container" podUID="6ece4112-1ac1-45a7-bd87-0361364a08a6" containerName="util" Nov 03 22:27:36.497408 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497299 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ece4112-1ac1-45a7-bd87-0361364a08a6" containerName="util" Nov 03 22:27:36.497408 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497305 5196 cpu_manager.go:414] "containerMap: removing container" podUID="e48ce008-81c3-4b1d-b67c-ea51fd6f0279" containerName="util" Nov 03 22:27:36.497408 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497310 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e48ce008-81c3-4b1d-b67c-ea51fd6f0279" containerName="util" Nov 03 22:27:36.497408 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497317 5196 cpu_manager.go:414] "containerMap: removing container" podUID="9d9f52e4-ae4e-4c9f-a477-fc1f1149791e" containerName="pull" Nov 03 22:27:36.497408 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497321 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d9f52e4-ae4e-4c9f-a477-fc1f1149791e" containerName="pull" Nov 03 22:27:36.497408 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497331 5196 cpu_manager.go:414] "containerMap: removing container" podUID="4e0857a6-bb8a-4563-91c1-4e3ff43d1078" containerName="util" Nov 03 22:27:36.497408 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497335 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e0857a6-bb8a-4563-91c1-4e3ff43d1078" containerName="util" Nov 03 22:27:36.497408 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497340 5196 cpu_manager.go:414] "containerMap: removing container" podUID="b5a56ae5-b5db-4e0e-b9f7-2b452d333c86" containerName="util" Nov 03 22:27:36.497408 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497345 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5a56ae5-b5db-4e0e-b9f7-2b452d333c86" containerName="util" Nov 03 22:27:36.497408 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497349 5196 cpu_manager.go:414] "containerMap: removing container" podUID="e48ce008-81c3-4b1d-b67c-ea51fd6f0279" containerName="pull" Nov 03 22:27:36.497408 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497353 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e48ce008-81c3-4b1d-b67c-ea51fd6f0279" containerName="pull" Nov 03 22:27:36.497408 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497358 5196 cpu_manager.go:414] "containerMap: removing container" podUID="7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c" containerName="pull" Nov 03 22:27:36.497408 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497362 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c" containerName="pull" Nov 03 22:27:36.497408 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497374 5196 cpu_manager.go:414] "containerMap: removing container" podUID="e48ce008-81c3-4b1d-b67c-ea51fd6f0279" containerName="extract" Nov 03 22:27:36.497408 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497378 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e48ce008-81c3-4b1d-b67c-ea51fd6f0279" containerName="extract" Nov 03 22:27:36.497408 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497385 5196 cpu_manager.go:414] "containerMap: removing container" podUID="7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c" containerName="extract" Nov 03 22:27:36.497408 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497390 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c" containerName="extract" Nov 03 22:27:36.497408 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497397 5196 cpu_manager.go:414] "containerMap: removing container" podUID="4e0857a6-bb8a-4563-91c1-4e3ff43d1078" containerName="extract" Nov 03 22:27:36.497408 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497401 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e0857a6-bb8a-4563-91c1-4e3ff43d1078" containerName="extract" Nov 03 22:27:36.497408 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497408 5196 cpu_manager.go:414] "containerMap: removing container" podUID="4e0857a6-bb8a-4563-91c1-4e3ff43d1078" containerName="pull" Nov 03 22:27:36.497408 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497412 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e0857a6-bb8a-4563-91c1-4e3ff43d1078" containerName="pull" Nov 03 22:27:36.497408 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497419 5196 cpu_manager.go:414] "containerMap: removing container" podUID="9d9f52e4-ae4e-4c9f-a477-fc1f1149791e" containerName="extract" Nov 03 22:27:36.498119 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497423 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d9f52e4-ae4e-4c9f-a477-fc1f1149791e" containerName="extract" Nov 03 22:27:36.498119 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497430 5196 cpu_manager.go:414] "containerMap: removing container" podUID="7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c" containerName="util" Nov 03 22:27:36.498119 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497434 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c" containerName="util" Nov 03 22:27:36.498119 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497440 5196 cpu_manager.go:414] "containerMap: removing container" podUID="3041da2a-28ee-437a-a3d0-93c0aad89939" containerName="util" Nov 03 22:27:36.498119 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497444 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3041da2a-28ee-437a-a3d0-93c0aad89939" containerName="util" Nov 03 22:27:36.498119 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497449 5196 cpu_manager.go:414] "containerMap: removing container" podUID="3041da2a-28ee-437a-a3d0-93c0aad89939" containerName="pull" Nov 03 22:27:36.498119 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497454 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3041da2a-28ee-437a-a3d0-93c0aad89939" containerName="pull" Nov 03 22:27:36.498119 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497459 5196 cpu_manager.go:414] "containerMap: removing container" podUID="9d9f52e4-ae4e-4c9f-a477-fc1f1149791e" containerName="util" Nov 03 22:27:36.498119 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497463 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d9f52e4-ae4e-4c9f-a477-fc1f1149791e" containerName="util" Nov 03 22:27:36.498119 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497467 5196 cpu_manager.go:414] "containerMap: removing container" podUID="3041da2a-28ee-437a-a3d0-93c0aad89939" containerName="extract" Nov 03 22:27:36.498119 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497471 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3041da2a-28ee-437a-a3d0-93c0aad89939" containerName="extract" Nov 03 22:27:36.498119 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497476 5196 cpu_manager.go:414] "containerMap: removing container" podUID="6ece4112-1ac1-45a7-bd87-0361364a08a6" containerName="extract" Nov 03 22:27:36.498119 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497479 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ece4112-1ac1-45a7-bd87-0361364a08a6" containerName="extract" Nov 03 22:27:36.498119 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497484 5196 cpu_manager.go:414] "containerMap: removing container" podUID="b5a56ae5-b5db-4e0e-b9f7-2b452d333c86" containerName="pull" Nov 03 22:27:36.498119 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497489 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5a56ae5-b5db-4e0e-b9f7-2b452d333c86" containerName="pull" Nov 03 22:27:36.498119 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497494 5196 cpu_manager.go:414] "containerMap: removing container" podUID="6ece4112-1ac1-45a7-bd87-0361364a08a6" containerName="pull" Nov 03 22:27:36.498119 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497498 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ece4112-1ac1-45a7-bd87-0361364a08a6" containerName="pull" Nov 03 22:27:36.498119 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497536 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d9f52e4-ae4e-4c9f-a477-fc1f1149791e" containerName="extract" Nov 03 22:27:36.498119 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497543 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="3041da2a-28ee-437a-a3d0-93c0aad89939" containerName="extract" Nov 03 22:27:36.498119 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497549 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ece4112-1ac1-45a7-bd87-0361364a08a6" containerName="extract" Nov 03 22:27:36.498119 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497554 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5a56ae5-b5db-4e0e-b9f7-2b452d333c86" containerName="extract" Nov 03 22:27:36.498119 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497561 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="e48ce008-81c3-4b1d-b67c-ea51fd6f0279" containerName="extract" Nov 03 22:27:36.498119 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497566 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f307c0c-e2e6-42f1-92e3-d6c0b2ba876c" containerName="extract" Nov 03 22:27:36.498119 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.497571 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e0857a6-bb8a-4563-91c1-4e3ff43d1078" containerName="extract" Nov 03 22:27:36.529159 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.529133 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dr2qp" Nov 03 22:27:36.582667 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.582635 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 22:27:36.582667 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.582664 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 22:27:36.582878 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.582664 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 03 22:27:36.601036 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.601010 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqp5z\" (UniqueName: \"kubernetes.io/projected/c7b3698d-7883-4a59-aaf6-a022a950bdb0-kube-api-access-nqp5z\") pod \"dns-operator-controller-manager-8557cbdb84-dr2qp\" (UID: \"c7b3698d-7883-4a59-aaf6-a022a950bdb0\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dr2qp" Nov 03 22:27:36.633393 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.633368 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-cb9p9\"" Nov 03 22:27:36.634463 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.634444 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-dr2qp"] Nov 03 22:27:36.701390 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.701357 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqp5z\" (UniqueName: \"kubernetes.io/projected/c7b3698d-7883-4a59-aaf6-a022a950bdb0-kube-api-access-nqp5z\") pod \"dns-operator-controller-manager-8557cbdb84-dr2qp\" (UID: \"c7b3698d-7883-4a59-aaf6-a022a950bdb0\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dr2qp" Nov 03 22:27:36.779089 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.779025 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqp5z\" (UniqueName: \"kubernetes.io/projected/c7b3698d-7883-4a59-aaf6-a022a950bdb0-kube-api-access-nqp5z\") pod \"dns-operator-controller-manager-8557cbdb84-dr2qp\" (UID: \"c7b3698d-7883-4a59-aaf6-a022a950bdb0\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dr2qp" Nov 03 22:27:36.854101 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.854071 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dr2qp" Nov 03 22:27:36.902041 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:36.901981 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:37.180516 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:37.180480 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:37.254317 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:37.254286 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-dr2qp"] Nov 03 22:27:37.257192 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:27:37.257164 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b3698d_7883_4a59_aaf6_a022a950bdb0.slice/crio-e8656cafd53f3b33731cc2e12954f13efb34118e2abce68eb01f00358370baca WatchSource:0}: Error finding container e8656cafd53f3b33731cc2e12954f13efb34118e2abce68eb01f00358370baca: Status 404 returned error can't find the container with id e8656cafd53f3b33731cc2e12954f13efb34118e2abce68eb01f00358370baca Nov 03 22:27:38.050661 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:38.050628 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:38.059383 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:38.059353 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dr2qp" event={"ID":"c7b3698d-7883-4a59-aaf6-a022a950bdb0","Type":"ContainerStarted","Data":"e8656cafd53f3b33731cc2e12954f13efb34118e2abce68eb01f00358370baca"} Nov 03 22:27:38.199333 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:38.199304 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:38.289302 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:38.289270 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-l87w4"] Nov 03 22:27:38.291939 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:38.291924 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-l87w4" Nov 03 22:27:38.345364 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:38.345346 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 22:27:38.346145 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:38.346125 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 22:27:38.346328 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:38.346310 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 22:27:38.346375 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:38.346362 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-6hr8v\"" Nov 03 22:27:38.414328 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:38.414302 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/92e77f1b-508c-4d0d-8a2b-89db5231d059-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b88f79f5c-l87w4\" (UID: \"92e77f1b-508c-4d0d-8a2b-89db5231d059\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-l87w4" Nov 03 22:27:38.414403 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:38.414338 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/92e77f1b-508c-4d0d-8a2b-89db5231d059-metrics-cert\") pod \"opendatahub-operator-controller-manager-6b88f79f5c-l87w4\" (UID: \"92e77f1b-508c-4d0d-8a2b-89db5231d059\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-l87w4" Nov 03 22:27:38.414455 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:38.414414 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/92e77f1b-508c-4d0d-8a2b-89db5231d059-manifests\") pod \"opendatahub-operator-controller-manager-6b88f79f5c-l87w4\" (UID: \"92e77f1b-508c-4d0d-8a2b-89db5231d059\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-l87w4" Nov 03 22:27:38.414455 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:38.414444 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92e77f1b-508c-4d0d-8a2b-89db5231d059-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b88f79f5c-l87w4\" (UID: \"92e77f1b-508c-4d0d-8a2b-89db5231d059\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-l87w4" Nov 03 22:27:38.414513 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:38.414486 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9ftk\" (UniqueName: \"kubernetes.io/projected/92e77f1b-508c-4d0d-8a2b-89db5231d059-kube-api-access-p9ftk\") pod \"opendatahub-operator-controller-manager-6b88f79f5c-l87w4\" (UID: \"92e77f1b-508c-4d0d-8a2b-89db5231d059\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-l87w4" Nov 03 22:27:38.414579 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:38.414562 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92e77f1b-508c-4d0d-8a2b-89db5231d059-tmp\") pod \"opendatahub-operator-controller-manager-6b88f79f5c-l87w4\" (UID: \"92e77f1b-508c-4d0d-8a2b-89db5231d059\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-l87w4" Nov 03 22:27:38.447566 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:38.447546 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 22:27:38.448234 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:38.448217 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 22:27:38.456524 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:38.456501 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:38.514919 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:38.514889 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92e77f1b-508c-4d0d-8a2b-89db5231d059-tmp\") pod \"opendatahub-operator-controller-manager-6b88f79f5c-l87w4\" (UID: \"92e77f1b-508c-4d0d-8a2b-89db5231d059\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-l87w4" Nov 03 22:27:38.515128 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:38.514937 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/92e77f1b-508c-4d0d-8a2b-89db5231d059-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b88f79f5c-l87w4\" (UID: \"92e77f1b-508c-4d0d-8a2b-89db5231d059\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-l87w4" Nov 03 22:27:38.515128 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:38.514977 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/92e77f1b-508c-4d0d-8a2b-89db5231d059-metrics-cert\") pod \"opendatahub-operator-controller-manager-6b88f79f5c-l87w4\" (UID: \"92e77f1b-508c-4d0d-8a2b-89db5231d059\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-l87w4" Nov 03 22:27:38.515128 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:38.515028 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/92e77f1b-508c-4d0d-8a2b-89db5231d059-manifests\") pod \"opendatahub-operator-controller-manager-6b88f79f5c-l87w4\" (UID: \"92e77f1b-508c-4d0d-8a2b-89db5231d059\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-l87w4" Nov 03 22:27:38.515128 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:38.515052 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92e77f1b-508c-4d0d-8a2b-89db5231d059-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b88f79f5c-l87w4\" (UID: \"92e77f1b-508c-4d0d-8a2b-89db5231d059\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-l87w4" Nov 03 22:27:38.515128 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:38.515083 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9ftk\" (UniqueName: \"kubernetes.io/projected/92e77f1b-508c-4d0d-8a2b-89db5231d059-kube-api-access-p9ftk\") pod \"opendatahub-operator-controller-manager-6b88f79f5c-l87w4\" (UID: \"92e77f1b-508c-4d0d-8a2b-89db5231d059\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-l87w4" Nov 03 22:27:38.515440 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:38.515409 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/92e77f1b-508c-4d0d-8a2b-89db5231d059-manifests\") pod \"opendatahub-operator-controller-manager-6b88f79f5c-l87w4\" (UID: \"92e77f1b-508c-4d0d-8a2b-89db5231d059\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-l87w4" Nov 03 22:27:38.517480 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:38.517456 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92e77f1b-508c-4d0d-8a2b-89db5231d059-tmp\") pod \"opendatahub-operator-controller-manager-6b88f79f5c-l87w4\" (UID: \"92e77f1b-508c-4d0d-8a2b-89db5231d059\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-l87w4" Nov 03 22:27:38.517480 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:38.517473 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/92e77f1b-508c-4d0d-8a2b-89db5231d059-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b88f79f5c-l87w4\" (UID: \"92e77f1b-508c-4d0d-8a2b-89db5231d059\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-l87w4" Nov 03 22:27:38.517617 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:38.517525 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92e77f1b-508c-4d0d-8a2b-89db5231d059-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b88f79f5c-l87w4\" (UID: \"92e77f1b-508c-4d0d-8a2b-89db5231d059\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-l87w4" Nov 03 22:27:38.517667 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:38.517651 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/92e77f1b-508c-4d0d-8a2b-89db5231d059-metrics-cert\") pod \"opendatahub-operator-controller-manager-6b88f79f5c-l87w4\" (UID: \"92e77f1b-508c-4d0d-8a2b-89db5231d059\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-l87w4" Nov 03 22:27:38.566917 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:38.566894 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-l87w4"] Nov 03 22:27:38.577361 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:38.577341 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9ftk\" (UniqueName: \"kubernetes.io/projected/92e77f1b-508c-4d0d-8a2b-89db5231d059-kube-api-access-p9ftk\") pod \"opendatahub-operator-controller-manager-6b88f79f5c-l87w4\" (UID: \"92e77f1b-508c-4d0d-8a2b-89db5231d059\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-l87w4" Nov 03 22:27:38.603879 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:38.603818 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-l87w4" Nov 03 22:27:38.607448 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:38.607427 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:38.750811 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:38.750774 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:38.901599 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:38.901509 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:39.042805 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:39.042764 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:39.187779 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:39.187702 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-l87w4"] Nov 03 22:27:39.193048 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:27:39.193016 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92e77f1b_508c_4d0d_8a2b_89db5231d059.slice/crio-39f95093865d2a4169313da3134bcef7ca6d87fb42415dca71d4fb8ccfd01437 WatchSource:0}: Error finding container 39f95093865d2a4169313da3134bcef7ca6d87fb42415dca71d4fb8ccfd01437: Status 404 returned error can't find the container with id 39f95093865d2a4169313da3134bcef7ca6d87fb42415dca71d4fb8ccfd01437 Nov 03 22:27:40.070795 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:40.070761 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-l87w4" event={"ID":"92e77f1b-508c-4d0d-8a2b-89db5231d059","Type":"ContainerStarted","Data":"39f95093865d2a4169313da3134bcef7ca6d87fb42415dca71d4fb8ccfd01437"} Nov 03 22:27:40.348310 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:40.346956 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-xf2bk"] Nov 03 22:27:40.349924 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:40.349444 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xf2bk" Nov 03 22:27:40.431609 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:40.431576 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c8fz\" (UniqueName: \"kubernetes.io/projected/ba199d3c-aeb0-44d0-96f8-48286a2c3b40-kube-api-access-8c8fz\") pod \"limitador-operator-controller-manager-5778bb4449-xf2bk\" (UID: \"ba199d3c-aeb0-44d0-96f8-48286a2c3b40\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xf2bk" Nov 03 22:27:40.450958 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:40.450844 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:40.455898 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:40.455740 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-xf2bk"] Nov 03 22:27:40.455898 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:40.455867 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-f5fhh\"" Nov 03 22:27:40.533038 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:40.532350 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c8fz\" (UniqueName: \"kubernetes.io/projected/ba199d3c-aeb0-44d0-96f8-48286a2c3b40-kube-api-access-8c8fz\") pod \"limitador-operator-controller-manager-5778bb4449-xf2bk\" (UID: \"ba199d3c-aeb0-44d0-96f8-48286a2c3b40\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xf2bk" Nov 03 22:27:40.601852 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:40.601780 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c8fz\" (UniqueName: \"kubernetes.io/projected/ba199d3c-aeb0-44d0-96f8-48286a2c3b40-kube-api-access-8c8fz\") pod \"limitador-operator-controller-manager-5778bb4449-xf2bk\" (UID: \"ba199d3c-aeb0-44d0-96f8-48286a2c3b40\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xf2bk" Nov 03 22:27:40.664287 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:40.664258 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xf2bk" Nov 03 22:27:40.892190 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:40.892101 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:41.066951 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:41.066913 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-xf2bk"] Nov 03 22:27:41.077986 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:41.077906 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dr2qp" event={"ID":"c7b3698d-7883-4a59-aaf6-a022a950bdb0","Type":"ContainerStarted","Data":"d799d298b66960ea6cc07bf0bdca9203d8b42c98cceec6dd7c98ca2b0ff9bb04"} Nov 03 22:27:41.078158 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:41.078086 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dr2qp" Nov 03 22:27:41.096129 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:27:41.096099 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba199d3c_aeb0_44d0_96f8_48286a2c3b40.slice/crio-e0c90f7204e9afd636d8c4b60eeb2eeb1e124c66e2281fb4ab5eeaa3d0f75a81 WatchSource:0}: Error finding container e0c90f7204e9afd636d8c4b60eeb2eeb1e124c66e2281fb4ab5eeaa3d0f75a81: Status 404 returned error can't find the container with id e0c90f7204e9afd636d8c4b60eeb2eeb1e124c66e2281fb4ab5eeaa3d0f75a81 Nov 03 22:27:41.199255 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:41.198452 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dr2qp" podStartSLOduration=1.796095985 podStartE2EDuration="5.198435188s" totalImagesPullingTime="3.402339203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:41.196795502 +0000 UTC m=+290.506354824" watchObservedRunningTime="2026-11-03 22:27:41.198435188 +0000 UTC m=+290.507994481" Nov 03 22:27:41.882841 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:41.882798 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:42.084192 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:42.084154 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xf2bk" event={"ID":"ba199d3c-aeb0-44d0-96f8-48286a2c3b40","Type":"ContainerStarted","Data":"e0c90f7204e9afd636d8c4b60eeb2eeb1e124c66e2281fb4ab5eeaa3d0f75a81"} Nov 03 22:27:43.090967 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:43.090926 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-l87w4" event={"ID":"92e77f1b-508c-4d0d-8a2b-89db5231d059","Type":"ContainerStarted","Data":"efa2e624cfcf5cbe5d1c19e700cb640238d1a7f83337078ffcceaaca54d908d8"} Nov 03 22:27:44.096585 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:44.096549 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xf2bk" event={"ID":"ba199d3c-aeb0-44d0-96f8-48286a2c3b40","Type":"ContainerStarted","Data":"2ec1564522d583f46dd1b8e622a310a7a20fcea4eed322f92bcc3856d533e5c0"} Nov 03 22:27:44.097649 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:44.096713 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xf2bk" Nov 03 22:27:44.098406 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:44.098377 5196 generic.go:356] "Generic (PLEG): container finished" podID="92e77f1b-508c-4d0d-8a2b-89db5231d059" containerID="efa2e624cfcf5cbe5d1c19e700cb640238d1a7f83337078ffcceaaca54d908d8" exitCode=0 Nov 03 22:27:44.098521 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:44.098457 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-l87w4" event={"ID":"92e77f1b-508c-4d0d-8a2b-89db5231d059","Type":"ContainerDied","Data":"efa2e624cfcf5cbe5d1c19e700cb640238d1a7f83337078ffcceaaca54d908d8"} Nov 03 22:27:44.227029 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:44.226987 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xf2bk" podStartSLOduration=1.988216345 podStartE2EDuration="4.226976839s" totalImagesPullingTime="2.238760494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:44.225573314 +0000 UTC m=+293.535132608" watchObservedRunningTime="2026-11-03 22:27:44.226976839 +0000 UTC m=+293.536536132" Nov 03 22:27:44.880740 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:44.880709 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:45.041893 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:45.041826 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:45.105168 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:45.105121 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-l87w4" event={"ID":"92e77f1b-508c-4d0d-8a2b-89db5231d059","Type":"ContainerStarted","Data":"e7a03ed95f683b99ae9897c973c7ff5e0852bebc94014453e4699ce202a63893"} Nov 03 22:27:45.213145 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:45.213042 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:45.251073 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:45.251025 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-l87w4" podStartSLOduration=2.482758925 podStartE2EDuration="7.251009992s" totalImagesPullingTime="4.247611434s" totalInitContainerRuntime="520.639633ms" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:45.24897343 +0000 UTC m=+294.558532740" watchObservedRunningTime="2026-11-03 22:27:45.251009992 +0000 UTC m=+294.560569284" Nov 03 22:27:45.422049 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:45.421997 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:46.075038 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:46.075000 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:46.109982 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:46.109956 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-l87w4" Nov 03 22:27:47.074750 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:47.074717 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:47.637586 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:47.637530 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:47.946280 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:47.946201 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:48.074322 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:48.074292 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-dcjnt"] Nov 03 22:27:48.076431 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:48.076416 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dcjnt" Nov 03 22:27:48.180125 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:48.180097 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:27:48.180125 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:48.180111 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 03 22:27:48.180474 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:48.180460 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-j7cnp\"" Nov 03 22:27:48.181713 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:48.181694 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-dcjnt"] Nov 03 22:27:48.190584 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:48.190567 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76af3ea2-e51c-4fc4-8f0f-2d9062a50d56-tmp\") pod \"jobset-operator-674cd59dc5-dcjnt\" (UID: \"76af3ea2-e51c-4fc4-8f0f-2d9062a50d56\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dcjnt" Nov 03 22:27:48.190653 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:48.190595 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfggp\" (UniqueName: \"kubernetes.io/projected/76af3ea2-e51c-4fc4-8f0f-2d9062a50d56-kube-api-access-dfggp\") pod \"jobset-operator-674cd59dc5-dcjnt\" (UID: \"76af3ea2-e51c-4fc4-8f0f-2d9062a50d56\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dcjnt" Nov 03 22:27:48.291430 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:48.291356 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76af3ea2-e51c-4fc4-8f0f-2d9062a50d56-tmp\") pod \"jobset-operator-674cd59dc5-dcjnt\" (UID: \"76af3ea2-e51c-4fc4-8f0f-2d9062a50d56\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dcjnt" Nov 03 22:27:48.291430 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:48.291386 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfggp\" (UniqueName: \"kubernetes.io/projected/76af3ea2-e51c-4fc4-8f0f-2d9062a50d56-kube-api-access-dfggp\") pod \"jobset-operator-674cd59dc5-dcjnt\" (UID: \"76af3ea2-e51c-4fc4-8f0f-2d9062a50d56\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dcjnt" Nov 03 22:27:48.291750 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:48.291732 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76af3ea2-e51c-4fc4-8f0f-2d9062a50d56-tmp\") pod \"jobset-operator-674cd59dc5-dcjnt\" (UID: \"76af3ea2-e51c-4fc4-8f0f-2d9062a50d56\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dcjnt" Nov 03 22:27:48.356314 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:48.356285 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfggp\" (UniqueName: \"kubernetes.io/projected/76af3ea2-e51c-4fc4-8f0f-2d9062a50d56-kube-api-access-dfggp\") pod \"jobset-operator-674cd59dc5-dcjnt\" (UID: \"76af3ea2-e51c-4fc4-8f0f-2d9062a50d56\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dcjnt" Nov 03 22:27:48.394184 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:48.394156 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dcjnt" Nov 03 22:27:48.754516 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:48.754483 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:48.769697 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:48.769668 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-dcjnt"] Nov 03 22:27:48.772723 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:27:48.772693 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76af3ea2_e51c_4fc4_8f0f_2d9062a50d56.slice/crio-f22cdc98911e514385632811c50f09fa50dde144362b4f78ef8080da86c3bc9a WatchSource:0}: Error finding container f22cdc98911e514385632811c50f09fa50dde144362b4f78ef8080da86c3bc9a: Status 404 returned error can't find the container with id f22cdc98911e514385632811c50f09fa50dde144362b4f78ef8080da86c3bc9a Nov 03 22:27:48.957960 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:48.957927 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:49.127844 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:49.127809 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dcjnt" event={"ID":"76af3ea2-e51c-4fc4-8f0f-2d9062a50d56","Type":"ContainerStarted","Data":"f22cdc98911e514385632811c50f09fa50dde144362b4f78ef8080da86c3bc9a"} Nov 03 22:27:50.278914 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:50.278875 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:50.441852 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:50.441814 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:52.086572 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:52.086541 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-dr2qp" Nov 03 22:27:52.161687 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:52.161655 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:52.308396 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:52.308362 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:52.784037 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:52.784008 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:53.147090 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:53.147051 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dcjnt" event={"ID":"76af3ea2-e51c-4fc4-8f0f-2d9062a50d56","Type":"ContainerStarted","Data":"dc47725cc4e58c8624d45e0d6b3d43ef1b722918820b088a2348e573b31c78a9"} Nov 03 22:27:53.274324 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:53.274268 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dcjnt" podStartSLOduration=1.945389022 podStartE2EDuration="5.274253139s" totalImagesPullingTime="3.328864117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:53.273501202 +0000 UTC m=+302.583060495" watchObservedRunningTime="2026-11-03 22:27:53.274253139 +0000 UTC m=+302.583812431" Nov 03 22:27:54.215261 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:54.215228 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:54.363196 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:54.363163 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:55.107984 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:55.107952 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xf2bk" Nov 03 22:27:55.120881 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:55.120838 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lrlsz"] Nov 03 22:27:55.122815 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:55.122801 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lrlsz" Nov 03 22:27:55.177111 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:55.177080 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lrlsz"] Nov 03 22:27:55.214709 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:55.214684 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-5n4t5\"" Nov 03 22:27:55.214870 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:55.214741 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 03 22:27:55.214870 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:55.214745 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:27:55.249883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:55.249837 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rj9b\" (UniqueName: \"kubernetes.io/projected/a9dc11f3-b091-488f-8aa5-24363cfa8cba-kube-api-access-9rj9b\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lrlsz\" (UID: \"a9dc11f3-b091-488f-8aa5-24363cfa8cba\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lrlsz" Nov 03 22:27:55.250230 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:55.249915 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9dc11f3-b091-488f-8aa5-24363cfa8cba-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lrlsz\" (UID: \"a9dc11f3-b091-488f-8aa5-24363cfa8cba\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lrlsz" Nov 03 22:27:55.350483 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:55.350452 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rj9b\" (UniqueName: \"kubernetes.io/projected/a9dc11f3-b091-488f-8aa5-24363cfa8cba-kube-api-access-9rj9b\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lrlsz\" (UID: \"a9dc11f3-b091-488f-8aa5-24363cfa8cba\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lrlsz" Nov 03 22:27:55.350641 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:55.350511 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9dc11f3-b091-488f-8aa5-24363cfa8cba-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lrlsz\" (UID: \"a9dc11f3-b091-488f-8aa5-24363cfa8cba\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lrlsz" Nov 03 22:27:55.350961 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:55.350938 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9dc11f3-b091-488f-8aa5-24363cfa8cba-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lrlsz\" (UID: \"a9dc11f3-b091-488f-8aa5-24363cfa8cba\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lrlsz" Nov 03 22:27:55.423107 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:55.423037 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rj9b\" (UniqueName: \"kubernetes.io/projected/a9dc11f3-b091-488f-8aa5-24363cfa8cba-kube-api-access-9rj9b\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-lrlsz\" (UID: \"a9dc11f3-b091-488f-8aa5-24363cfa8cba\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lrlsz" Nov 03 22:27:55.432881 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:55.432843 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lrlsz" Nov 03 22:27:55.861590 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:55.861565 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lrlsz"] Nov 03 22:27:55.864456 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:27:55.864424 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9dc11f3_b091_488f_8aa5_24363cfa8cba.slice/crio-e02d6b38991ee6655ecfc385c17bc423376f707dad7437849267e361c8ffcf46 WatchSource:0}: Error finding container e02d6b38991ee6655ecfc385c17bc423376f707dad7437849267e361c8ffcf46: Status 404 returned error can't find the container with id e02d6b38991ee6655ecfc385c17bc423376f707dad7437849267e361c8ffcf46 Nov 03 22:27:55.867182 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:55.867165 5196 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:27:55.918168 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:27:55.918134 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:27:56.160846 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:56.160761 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lrlsz" event={"ID":"a9dc11f3-b091-488f-8aa5-24363cfa8cba","Type":"ContainerStarted","Data":"e02d6b38991ee6655ecfc385c17bc423376f707dad7437849267e361c8ffcf46"} Nov 03 22:27:57.115577 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:57.115549 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6b88f79f5c-l87w4" Nov 03 22:27:57.282968 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:57.282921 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:57.441405 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:57.441325 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:57.678081 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:57.678047 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:27:59.176460 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:59.176422 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lrlsz" event={"ID":"a9dc11f3-b091-488f-8aa5-24363cfa8cba","Type":"ContainerStarted","Data":"b792e12c2fe32a0f50fe25702ac5cc05e550abc4de93a20580502088295044a7"} Nov 03 22:27:59.411989 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:27:59.411944 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-lrlsz" podStartSLOduration=1.9461065450000001 podStartE2EDuration="4.411930541s" totalImagesPullingTime="2.465823996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:59.36022738 +0000 UTC m=+308.669786673" watchObservedRunningTime="2026-11-03 22:27:59.411930541 +0000 UTC m=+308.721489839" Nov 03 22:28:00.049234 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:00.049201 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:00.234077 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:00.234047 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:00.676549 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:00.676520 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-s88s5"] Nov 03 22:28:00.679624 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:00.679604 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s88s5" Nov 03 22:28:00.741599 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:00.741574 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 03 22:28:00.741725 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:00.741706 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 03 22:28:00.793215 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:00.793184 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-s88s5"] Nov 03 22:28:00.793430 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:00.793412 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbz56\" (UniqueName: \"kubernetes.io/projected/b1cfc4a2-22dd-4cd2-abf3-f3e863b2268b-kube-api-access-qbz56\") pod \"cert-manager-webhook-5dbbbf75cc-s88s5\" (UID: \"b1cfc4a2-22dd-4cd2-abf3-f3e863b2268b\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s88s5" Nov 03 22:28:00.793483 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:00.793471 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1cfc4a2-22dd-4cd2-abf3-f3e863b2268b-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-s88s5\" (UID: \"b1cfc4a2-22dd-4cd2-abf3-f3e863b2268b\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s88s5" Nov 03 22:28:00.856969 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:00.856933 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:00.894019 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:00.893993 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbz56\" (UniqueName: \"kubernetes.io/projected/b1cfc4a2-22dd-4cd2-abf3-f3e863b2268b-kube-api-access-qbz56\") pod \"cert-manager-webhook-5dbbbf75cc-s88s5\" (UID: \"b1cfc4a2-22dd-4cd2-abf3-f3e863b2268b\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s88s5" Nov 03 22:28:00.894154 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:00.894056 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1cfc4a2-22dd-4cd2-abf3-f3e863b2268b-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-s88s5\" (UID: \"b1cfc4a2-22dd-4cd2-abf3-f3e863b2268b\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s88s5" Nov 03 22:28:00.959944 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:00.959886 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbz56\" (UniqueName: \"kubernetes.io/projected/b1cfc4a2-22dd-4cd2-abf3-f3e863b2268b-kube-api-access-qbz56\") pod \"cert-manager-webhook-5dbbbf75cc-s88s5\" (UID: \"b1cfc4a2-22dd-4cd2-abf3-f3e863b2268b\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s88s5" Nov 03 22:28:00.959944 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:00.959896 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1cfc4a2-22dd-4cd2-abf3-f3e863b2268b-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-s88s5\" (UID: \"b1cfc4a2-22dd-4cd2-abf3-f3e863b2268b\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s88s5" Nov 03 22:28:00.990672 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:00.990651 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s88s5" Nov 03 22:28:01.030133 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:01.030090 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:01.364063 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:01.364040 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-s88s5"] Nov 03 22:28:01.366602 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:28:01.366578 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1cfc4a2_22dd_4cd2_abf3_f3e863b2268b.slice/crio-e2c9727d448b5b0110971c950957f74186c021381f48d9206d55c9382cc9a69a WatchSource:0}: Error finding container e2c9727d448b5b0110971c950957f74186c021381f48d9206d55c9382cc9a69a: Status 404 returned error can't find the container with id e2c9727d448b5b0110971c950957f74186c021381f48d9206d55c9382cc9a69a Nov 03 22:28:02.191630 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:02.191588 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s88s5" event={"ID":"b1cfc4a2-22dd-4cd2-abf3-f3e863b2268b","Type":"ContainerStarted","Data":"e2c9727d448b5b0110971c950957f74186c021381f48d9206d55c9382cc9a69a"} Nov 03 22:28:02.386875 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:02.386826 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:02.532095 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:02.532027 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:02.556673 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:02.556646 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-lhh88"] Nov 03 22:28:02.560889 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:02.560849 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lhh88" Nov 03 22:28:02.615974 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:02.615951 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 03 22:28:02.619099 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:02.619074 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-plwfc\"" Nov 03 22:28:02.666954 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:02.666928 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 03 22:28:02.670981 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:02.670962 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-lhh88"] Nov 03 22:28:02.678778 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:02.678735 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:02.709425 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:02.709404 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e6a6a86a-e69b-49ec-bc45-a8875edd45e3-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-lhh88\" (UID: \"e6a6a86a-e69b-49ec-bc45-a8875edd45e3\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lhh88" Nov 03 22:28:02.709510 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:02.709488 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e6a6a86a-e69b-49ec-bc45-a8875edd45e3-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-lhh88\" (UID: \"e6a6a86a-e69b-49ec-bc45-a8875edd45e3\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lhh88" Nov 03 22:28:02.709556 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:02.709532 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtxmw\" (UniqueName: \"kubernetes.io/projected/e6a6a86a-e69b-49ec-bc45-a8875edd45e3-kube-api-access-rtxmw\") pod \"kuadrant-console-plugin-66c4fb575d-lhh88\" (UID: \"e6a6a86a-e69b-49ec-bc45-a8875edd45e3\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lhh88" Nov 03 22:28:02.810057 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:02.809996 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e6a6a86a-e69b-49ec-bc45-a8875edd45e3-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-lhh88\" (UID: \"e6a6a86a-e69b-49ec-bc45-a8875edd45e3\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lhh88" Nov 03 22:28:02.810057 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:02.810035 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtxmw\" (UniqueName: \"kubernetes.io/projected/e6a6a86a-e69b-49ec-bc45-a8875edd45e3-kube-api-access-rtxmw\") pod \"kuadrant-console-plugin-66c4fb575d-lhh88\" (UID: \"e6a6a86a-e69b-49ec-bc45-a8875edd45e3\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lhh88" Nov 03 22:28:02.810222 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:02.810071 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e6a6a86a-e69b-49ec-bc45-a8875edd45e3-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-lhh88\" (UID: \"e6a6a86a-e69b-49ec-bc45-a8875edd45e3\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lhh88" Nov 03 22:28:02.811055 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:02.811032 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e6a6a86a-e69b-49ec-bc45-a8875edd45e3-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-lhh88\" (UID: \"e6a6a86a-e69b-49ec-bc45-a8875edd45e3\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lhh88" Nov 03 22:28:02.812895 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:02.812850 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e6a6a86a-e69b-49ec-bc45-a8875edd45e3-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-lhh88\" (UID: \"e6a6a86a-e69b-49ec-bc45-a8875edd45e3\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lhh88" Nov 03 22:28:02.829642 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:02.829618 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:02.871720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:02.871701 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtxmw\" (UniqueName: \"kubernetes.io/projected/e6a6a86a-e69b-49ec-bc45-a8875edd45e3-kube-api-access-rtxmw\") pod \"kuadrant-console-plugin-66c4fb575d-lhh88\" (UID: \"e6a6a86a-e69b-49ec-bc45-a8875edd45e3\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lhh88" Nov 03 22:28:02.872474 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:02.872460 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lhh88" Nov 03 22:28:02.982919 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:02.982890 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:03.245570 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:03.245542 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-lhh88"] Nov 03 22:28:03.248740 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:28:03.248713 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6a6a86a_e69b_49ec_bc45_a8875edd45e3.slice/crio-faf84ef5eb370eae2aed6e3dd7e3a7742d6af9896eeacb17fb2f08c5ac199ec5 WatchSource:0}: Error finding container faf84ef5eb370eae2aed6e3dd7e3a7742d6af9896eeacb17fb2f08c5ac199ec5: Status 404 returned error can't find the container with id faf84ef5eb370eae2aed6e3dd7e3a7742d6af9896eeacb17fb2f08c5ac199ec5 Nov 03 22:28:04.203823 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:04.203783 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lhh88" event={"ID":"e6a6a86a-e69b-49ec-bc45-a8875edd45e3","Type":"ContainerStarted","Data":"faf84ef5eb370eae2aed6e3dd7e3a7742d6af9896eeacb17fb2f08c5ac199ec5"} Nov 03 22:28:05.148189 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:05.148088 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:05.214440 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:05.214342 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s88s5" event={"ID":"b1cfc4a2-22dd-4cd2-abf3-f3e863b2268b","Type":"ContainerStarted","Data":"051c36441ed010c4af106127136c4e45206d1e77544002b70ca040949153dba2"} Nov 03 22:28:05.215140 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:05.215118 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s88s5" Nov 03 22:28:05.334124 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:05.334063 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s88s5" podStartSLOduration=1.752300874 podStartE2EDuration="5.33402261s" totalImagesPullingTime="3.581721736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:05.332030201 +0000 UTC m=+314.641589493" watchObservedRunningTime="2026-11-03 22:28:05.33402261 +0000 UTC m=+314.643581905" Nov 03 22:28:06.247502 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:06.247449 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:06.388748 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:06.388710 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:06.853661 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:06.853620 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:07.002631 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:07.002594 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:07.415383 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:07.415289 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:07.558880 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:07.558825 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:07.944401 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:07.944363 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:09.117491 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:09.117453 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:09.236186 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:09.236155 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lhh88" event={"ID":"e6a6a86a-e69b-49ec-bc45-a8875edd45e3","Type":"ContainerStarted","Data":"b9868570d3c94f78215d1440f1856e75c5b3398fe80f3d96f3be321d836d69e4"} Nov 03 22:28:09.260460 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:09.260433 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:09.355227 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:09.355176 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lhh88" podStartSLOduration=1.986191039 podStartE2EDuration="7.355163091s" totalImagesPullingTime="5.368972052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:09.354016635 +0000 UTC m=+318.663575929" watchObservedRunningTime="2026-11-03 22:28:09.355163091 +0000 UTC m=+318.664722383" Nov 03 22:28:09.427942 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:09.427842 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:09.516591 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:09.516559 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-586b8ddf6d-sgzck"] Nov 03 22:28:09.521188 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:09.521141 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-586b8ddf6d-sgzck" Nov 03 22:28:09.563899 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:09.563843 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/157b1eec-64e7-4c2f-94ad-dfa8a419173b-tmp\") pod \"kube-auth-proxy-586b8ddf6d-sgzck\" (UID: \"157b1eec-64e7-4c2f-94ad-dfa8a419173b\") " pod="openshift-ingress/kube-auth-proxy-586b8ddf6d-sgzck" Nov 03 22:28:09.564087 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:09.563971 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrtm5\" (UniqueName: \"kubernetes.io/projected/157b1eec-64e7-4c2f-94ad-dfa8a419173b-kube-api-access-rrtm5\") pod \"kube-auth-proxy-586b8ddf6d-sgzck\" (UID: \"157b1eec-64e7-4c2f-94ad-dfa8a419173b\") " pod="openshift-ingress/kube-auth-proxy-586b8ddf6d-sgzck" Nov 03 22:28:09.564087 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:09.564014 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/157b1eec-64e7-4c2f-94ad-dfa8a419173b-tls-certs\") pod \"kube-auth-proxy-586b8ddf6d-sgzck\" (UID: \"157b1eec-64e7-4c2f-94ad-dfa8a419173b\") " pod="openshift-ingress/kube-auth-proxy-586b8ddf6d-sgzck" Nov 03 22:28:09.602009 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:09.601955 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 22:28:09.602247 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:09.602234 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 22:28:09.602299 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:09.602265 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-sqnzz\"" Nov 03 22:28:09.614353 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:09.614330 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:09.618955 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:09.618931 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-586b8ddf6d-sgzck"] Nov 03 22:28:09.664753 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:09.664728 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/157b1eec-64e7-4c2f-94ad-dfa8a419173b-tmp\") pod \"kube-auth-proxy-586b8ddf6d-sgzck\" (UID: \"157b1eec-64e7-4c2f-94ad-dfa8a419173b\") " pod="openshift-ingress/kube-auth-proxy-586b8ddf6d-sgzck" Nov 03 22:28:09.664888 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:09.664778 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrtm5\" (UniqueName: \"kubernetes.io/projected/157b1eec-64e7-4c2f-94ad-dfa8a419173b-kube-api-access-rrtm5\") pod \"kube-auth-proxy-586b8ddf6d-sgzck\" (UID: \"157b1eec-64e7-4c2f-94ad-dfa8a419173b\") " pod="openshift-ingress/kube-auth-proxy-586b8ddf6d-sgzck" Nov 03 22:28:09.664888 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:09.664804 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/157b1eec-64e7-4c2f-94ad-dfa8a419173b-tls-certs\") pod \"kube-auth-proxy-586b8ddf6d-sgzck\" (UID: \"157b1eec-64e7-4c2f-94ad-dfa8a419173b\") " pod="openshift-ingress/kube-auth-proxy-586b8ddf6d-sgzck" Nov 03 22:28:09.667122 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:09.667093 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/157b1eec-64e7-4c2f-94ad-dfa8a419173b-tmp\") pod \"kube-auth-proxy-586b8ddf6d-sgzck\" (UID: \"157b1eec-64e7-4c2f-94ad-dfa8a419173b\") " pod="openshift-ingress/kube-auth-proxy-586b8ddf6d-sgzck" Nov 03 22:28:09.667213 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:09.667198 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/157b1eec-64e7-4c2f-94ad-dfa8a419173b-tls-certs\") pod \"kube-auth-proxy-586b8ddf6d-sgzck\" (UID: \"157b1eec-64e7-4c2f-94ad-dfa8a419173b\") " pod="openshift-ingress/kube-auth-proxy-586b8ddf6d-sgzck" Nov 03 22:28:09.734219 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:09.734152 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrtm5\" (UniqueName: \"kubernetes.io/projected/157b1eec-64e7-4c2f-94ad-dfa8a419173b-kube-api-access-rrtm5\") pod \"kube-auth-proxy-586b8ddf6d-sgzck\" (UID: \"157b1eec-64e7-4c2f-94ad-dfa8a419173b\") " pod="openshift-ingress/kube-auth-proxy-586b8ddf6d-sgzck" Nov 03 22:28:09.770436 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:09.770385 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:09.832542 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:09.832519 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-586b8ddf6d-sgzck" Nov 03 22:28:09.931607 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:09.931571 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:10.132371 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:10.132326 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:10.231914 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:10.231889 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-586b8ddf6d-sgzck"] Nov 03 22:28:10.234383 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:28:10.234360 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod157b1eec_64e7_4c2f_94ad_dfa8a419173b.slice/crio-f587dcff459f40a98a5fb47b595553c8770a60e16be49e2f3bae8fe39473f116 WatchSource:0}: Error finding container f587dcff459f40a98a5fb47b595553c8770a60e16be49e2f3bae8fe39473f116: Status 404 returned error can't find the container with id f587dcff459f40a98a5fb47b595553c8770a60e16be49e2f3bae8fe39473f116 Nov 03 22:28:10.241434 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:10.241405 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-586b8ddf6d-sgzck" event={"ID":"157b1eec-64e7-4c2f-94ad-dfa8a419173b","Type":"ContainerStarted","Data":"f587dcff459f40a98a5fb47b595553c8770a60e16be49e2f3bae8fe39473f116"} Nov 03 22:28:10.412052 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:10.411973 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:10.679305 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:10.679237 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:11.461127 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:11.461085 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:11.615537 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:11.615500 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:11.773873 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:11.773681 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:12.227732 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:12.227702 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s88s5" Nov 03 22:28:13.259214 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:13.259173 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-586b8ddf6d-sgzck" event={"ID":"157b1eec-64e7-4c2f-94ad-dfa8a419173b","Type":"ContainerStarted","Data":"fe58094bc7adb806bf469936d2d7b92d3a5b851450146cd4de489b38fe455f5e"} Nov 03 22:28:13.379279 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:13.379222 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-586b8ddf6d-sgzck" podStartSLOduration=1.592650626 podStartE2EDuration="4.379207412s" totalImagesPullingTime="2.786556786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:13.378572008 +0000 UTC m=+322.688131303" watchObservedRunningTime="2026-11-03 22:28:13.379207412 +0000 UTC m=+322.688766756" Nov 03 22:28:13.684657 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:13.684583 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xrzzs"] Nov 03 22:28:13.687787 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:13.687769 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-xrzzs" Nov 03 22:28:13.745550 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:13.745520 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xrzzs"] Nov 03 22:28:13.746902 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:13.746880 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-b2nn5\"" Nov 03 22:28:13.800024 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:13.800002 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl49w\" (UniqueName: \"kubernetes.io/projected/c3dfa765-6b3e-435f-8d41-59eb8a1974ec-kube-api-access-cl49w\") pod \"authorino-56c6c9ddd5-xrzzs\" (UID: \"c3dfa765-6b3e-435f-8d41-59eb8a1974ec\") " pod="kuadrant-system/authorino-56c6c9ddd5-xrzzs" Nov 03 22:28:13.901411 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:13.901388 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl49w\" (UniqueName: \"kubernetes.io/projected/c3dfa765-6b3e-435f-8d41-59eb8a1974ec-kube-api-access-cl49w\") pod \"authorino-56c6c9ddd5-xrzzs\" (UID: \"c3dfa765-6b3e-435f-8d41-59eb8a1974ec\") " pod="kuadrant-system/authorino-56c6c9ddd5-xrzzs" Nov 03 22:28:13.962795 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:13.962728 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl49w\" (UniqueName: \"kubernetes.io/projected/c3dfa765-6b3e-435f-8d41-59eb8a1974ec-kube-api-access-cl49w\") pod \"authorino-56c6c9ddd5-xrzzs\" (UID: \"c3dfa765-6b3e-435f-8d41-59eb8a1974ec\") " pod="kuadrant-system/authorino-56c6c9ddd5-xrzzs" Nov 03 22:28:13.998845 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:13.998820 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-xrzzs" Nov 03 22:28:14.373474 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:14.373145 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xrzzs"] Nov 03 22:28:14.375786 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:28:14.375759 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3dfa765_6b3e_435f_8d41_59eb8a1974ec.slice/crio-3248b0d6ab5afc04e454f882f863b1ae6ed45373e8c38bc691e5e390ede271cf WatchSource:0}: Error finding container 3248b0d6ab5afc04e454f882f863b1ae6ed45373e8c38bc691e5e390ede271cf: Status 404 returned error can't find the container with id 3248b0d6ab5afc04e454f882f863b1ae6ed45373e8c38bc691e5e390ede271cf Nov 03 22:28:15.269747 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:15.269716 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-xrzzs" event={"ID":"c3dfa765-6b3e-435f-8d41-59eb8a1974ec","Type":"ContainerStarted","Data":"3248b0d6ab5afc04e454f882f863b1ae6ed45373e8c38bc691e5e390ede271cf"} Nov 03 22:28:17.280998 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:17.280915 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-xrzzs" event={"ID":"c3dfa765-6b3e-435f-8d41-59eb8a1974ec","Type":"ContainerStarted","Data":"e1fe09d7be256c732c5b56a2f03ab18592dc21fa4788ccc3f304e1dadc100d1f"} Nov 03 22:28:17.396396 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:17.396340 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-xrzzs" podStartSLOduration=1.847798243 podStartE2EDuration="4.396324012s" totalImagesPullingTime="2.548525769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:17.395430557 +0000 UTC m=+326.704989850" watchObservedRunningTime="2026-11-03 22:28:17.396324012 +0000 UTC m=+326.705883305" Nov 03 22:28:17.655962 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:17.655927 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:25.917803 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:28:25.917755 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:28:34.956698 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:34.956665 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:35.120769 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:35.120730 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:35.747509 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:35.747476 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:36.580050 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:36.580021 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:36.726742 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:36.726697 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:36.875969 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:36.875944 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:41.969616 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:41.969570 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:55.918217 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:28:55.918175 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:28:56.326619 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:56.326536 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:56.401511 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:56.401483 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-s69fp"] Nov 03 22:28:56.418566 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:56.418543 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-s69fp" Nov 03 22:28:56.471743 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:56.471695 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:56.522402 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:56.522370 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 22:28:56.522553 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:56.522444 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 22:28:56.522553 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:56.522473 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 22:28:56.522553 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:56.522511 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 22:28:56.522553 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:56.522535 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 22:28:56.522697 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:56.522603 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-w82vk\"" Nov 03 22:28:56.524260 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:56.524244 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-s69fp"] Nov 03 22:28:56.555971 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:56.555948 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6135f6a9-74f7-49d5-961b-3e3d877e835e-webhook-cert\") pod \"maas-controller-579cf7c6d-s69fp\" (UID: \"6135f6a9-74f7-49d5-961b-3e3d877e835e\") " pod="opendatahub/maas-controller-579cf7c6d-s69fp" Nov 03 22:28:56.556068 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:56.556004 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r6cd\" (UniqueName: \"kubernetes.io/projected/6135f6a9-74f7-49d5-961b-3e3d877e835e-kube-api-access-9r6cd\") pod \"maas-controller-579cf7c6d-s69fp\" (UID: \"6135f6a9-74f7-49d5-961b-3e3d877e835e\") " pod="opendatahub/maas-controller-579cf7c6d-s69fp" Nov 03 22:28:56.556068 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:56.556033 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6135f6a9-74f7-49d5-961b-3e3d877e835e-metrics-certs\") pod \"maas-controller-579cf7c6d-s69fp\" (UID: \"6135f6a9-74f7-49d5-961b-3e3d877e835e\") " pod="opendatahub/maas-controller-579cf7c6d-s69fp" Nov 03 22:28:56.618514 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:56.618492 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:56.657113 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:56.657087 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r6cd\" (UniqueName: \"kubernetes.io/projected/6135f6a9-74f7-49d5-961b-3e3d877e835e-kube-api-access-9r6cd\") pod \"maas-controller-579cf7c6d-s69fp\" (UID: \"6135f6a9-74f7-49d5-961b-3e3d877e835e\") " pod="opendatahub/maas-controller-579cf7c6d-s69fp" Nov 03 22:28:56.657218 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:56.657127 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6135f6a9-74f7-49d5-961b-3e3d877e835e-metrics-certs\") pod \"maas-controller-579cf7c6d-s69fp\" (UID: \"6135f6a9-74f7-49d5-961b-3e3d877e835e\") " pod="opendatahub/maas-controller-579cf7c6d-s69fp" Nov 03 22:28:56.657218 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:56.657149 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6135f6a9-74f7-49d5-961b-3e3d877e835e-webhook-cert\") pod \"maas-controller-579cf7c6d-s69fp\" (UID: \"6135f6a9-74f7-49d5-961b-3e3d877e835e\") " pod="opendatahub/maas-controller-579cf7c6d-s69fp" Nov 03 22:28:56.659698 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:56.659673 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6135f6a9-74f7-49d5-961b-3e3d877e835e-metrics-certs\") pod \"maas-controller-579cf7c6d-s69fp\" (UID: \"6135f6a9-74f7-49d5-961b-3e3d877e835e\") " pod="opendatahub/maas-controller-579cf7c6d-s69fp" Nov 03 22:28:56.659790 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:56.659698 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6135f6a9-74f7-49d5-961b-3e3d877e835e-webhook-cert\") pod \"maas-controller-579cf7c6d-s69fp\" (UID: \"6135f6a9-74f7-49d5-961b-3e3d877e835e\") " pod="opendatahub/maas-controller-579cf7c6d-s69fp" Nov 03 22:28:56.719334 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:56.719303 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r6cd\" (UniqueName: \"kubernetes.io/projected/6135f6a9-74f7-49d5-961b-3e3d877e835e-kube-api-access-9r6cd\") pod \"maas-controller-579cf7c6d-s69fp\" (UID: \"6135f6a9-74f7-49d5-961b-3e3d877e835e\") " pod="opendatahub/maas-controller-579cf7c6d-s69fp" Nov 03 22:28:56.731089 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:56.731063 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-s69fp" Nov 03 22:28:56.764755 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:56.764725 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:56.832670 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:56.832640 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8657649675-k4qdp"] Nov 03 22:28:56.834995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:56.834979 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8657649675-k4qdp" Nov 03 22:28:56.900095 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:56.900028 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8657649675-k4qdp"] Nov 03 22:28:56.960504 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:56.960466 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zrn4\" (UniqueName: \"kubernetes.io/projected/69b6bfce-f5ca-4c52-a9b1-5749383d2370-kube-api-access-8zrn4\") pod \"maas-controller-8657649675-k4qdp\" (UID: \"69b6bfce-f5ca-4c52-a9b1-5749383d2370\") " pod="opendatahub/maas-controller-8657649675-k4qdp" Nov 03 22:28:56.960504 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:56.960523 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/69b6bfce-f5ca-4c52-a9b1-5749383d2370-webhook-cert\") pod \"maas-controller-8657649675-k4qdp\" (UID: \"69b6bfce-f5ca-4c52-a9b1-5749383d2370\") " pod="opendatahub/maas-controller-8657649675-k4qdp" Nov 03 22:28:56.960953 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:56.960544 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69b6bfce-f5ca-4c52-a9b1-5749383d2370-metrics-certs\") pod \"maas-controller-8657649675-k4qdp\" (UID: \"69b6bfce-f5ca-4c52-a9b1-5749383d2370\") " pod="opendatahub/maas-controller-8657649675-k4qdp" Nov 03 22:28:57.061326 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:57.061287 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/69b6bfce-f5ca-4c52-a9b1-5749383d2370-webhook-cert\") pod \"maas-controller-8657649675-k4qdp\" (UID: \"69b6bfce-f5ca-4c52-a9b1-5749383d2370\") " pod="opendatahub/maas-controller-8657649675-k4qdp" Nov 03 22:28:57.061326 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:57.061328 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69b6bfce-f5ca-4c52-a9b1-5749383d2370-metrics-certs\") pod \"maas-controller-8657649675-k4qdp\" (UID: \"69b6bfce-f5ca-4c52-a9b1-5749383d2370\") " pod="opendatahub/maas-controller-8657649675-k4qdp" Nov 03 22:28:57.061499 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:57.061388 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zrn4\" (UniqueName: \"kubernetes.io/projected/69b6bfce-f5ca-4c52-a9b1-5749383d2370-kube-api-access-8zrn4\") pod \"maas-controller-8657649675-k4qdp\" (UID: \"69b6bfce-f5ca-4c52-a9b1-5749383d2370\") " pod="opendatahub/maas-controller-8657649675-k4qdp" Nov 03 22:28:57.064026 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:57.063997 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/69b6bfce-f5ca-4c52-a9b1-5749383d2370-webhook-cert\") pod \"maas-controller-8657649675-k4qdp\" (UID: \"69b6bfce-f5ca-4c52-a9b1-5749383d2370\") " pod="opendatahub/maas-controller-8657649675-k4qdp" Nov 03 22:28:57.064133 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:57.063997 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69b6bfce-f5ca-4c52-a9b1-5749383d2370-metrics-certs\") pod \"maas-controller-8657649675-k4qdp\" (UID: \"69b6bfce-f5ca-4c52-a9b1-5749383d2370\") " pod="opendatahub/maas-controller-8657649675-k4qdp" Nov 03 22:28:57.103639 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:57.103619 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-s69fp"] Nov 03 22:28:57.106287 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:28:57.106258 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6135f6a9_74f7_49d5_961b_3e3d877e835e.slice/crio-fb35713038550b93d55ee51d9ea7da66b9db611229016090dd9f8bfab5ae49e5 WatchSource:0}: Error finding container fb35713038550b93d55ee51d9ea7da66b9db611229016090dd9f8bfab5ae49e5: Status 404 returned error can't find the container with id fb35713038550b93d55ee51d9ea7da66b9db611229016090dd9f8bfab5ae49e5 Nov 03 22:28:57.122997 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:57.122972 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zrn4\" (UniqueName: \"kubernetes.io/projected/69b6bfce-f5ca-4c52-a9b1-5749383d2370-kube-api-access-8zrn4\") pod \"maas-controller-8657649675-k4qdp\" (UID: \"69b6bfce-f5ca-4c52-a9b1-5749383d2370\") " pod="opendatahub/maas-controller-8657649675-k4qdp" Nov 03 22:28:57.146822 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:57.146803 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8657649675-k4qdp" Nov 03 22:28:57.462352 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:57.462275 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-s69fp" event={"ID":"6135f6a9-74f7-49d5-961b-3e3d877e835e","Type":"ContainerStarted","Data":"fb35713038550b93d55ee51d9ea7da66b9db611229016090dd9f8bfab5ae49e5"} Nov 03 22:28:57.523333 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:57.523292 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8657649675-k4qdp"] Nov 03 22:28:57.525912 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:28:57.525882 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69b6bfce_f5ca_4c52_a9b1_5749383d2370.slice/crio-899949a651254138fb34d6137ad6e2fc96c4f8f9555bbe91df48a59169659b54 WatchSource:0}: Error finding container 899949a651254138fb34d6137ad6e2fc96c4f8f9555bbe91df48a59169659b54: Status 404 returned error can't find the container with id 899949a651254138fb34d6137ad6e2fc96c4f8f9555bbe91df48a59169659b54 Nov 03 22:28:58.022324 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:58.022272 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:58.273190 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:58.273103 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:28:58.469995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:28:58.469957 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8657649675-k4qdp" event={"ID":"69b6bfce-f5ca-4c52-a9b1-5749383d2370","Type":"ContainerStarted","Data":"899949a651254138fb34d6137ad6e2fc96c4f8f9555bbe91df48a59169659b54"} Nov 03 22:29:00.483763 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:00.483668 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-s69fp" event={"ID":"6135f6a9-74f7-49d5-961b-3e3d877e835e","Type":"ContainerStarted","Data":"35d2b957e3b3646ecb87a11cb9a0280a79f2be895e91abef443e90c8b5cab4df"} Nov 03 22:29:00.484589 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:00.484563 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-s69fp" Nov 03 22:29:00.595678 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:00.595634 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:00.607315 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:00.607270 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-s69fp" podStartSLOduration=1.5405051950000002 podStartE2EDuration="4.607256277s" totalImagesPullingTime="3.066751082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:00.606340471 +0000 UTC m=+369.915899779" watchObservedRunningTime="2026-11-03 22:29:00.607256277 +0000 UTC m=+369.916815569" Nov 03 22:29:00.791263 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:00.791178 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:01.493214 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:01.493103 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8657649675-k4qdp" event={"ID":"69b6bfce-f5ca-4c52-a9b1-5749383d2370","Type":"ContainerStarted","Data":"ea84c679a1684e677cf3fe9c886e526d3439eedbea1ef2da0a75fb7cf18b960c"} Nov 03 22:29:01.493214 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:01.493172 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8657649675-k4qdp" Nov 03 22:29:01.612849 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:01.612785 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-8657649675-k4qdp" podStartSLOduration=2.521558804 podStartE2EDuration="5.612769438s" totalImagesPullingTime="3.091210634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:01.612283523 +0000 UTC m=+370.921842854" watchObservedRunningTime="2026-11-03 22:29:01.612769438 +0000 UTC m=+370.922328732" Nov 03 22:29:02.997098 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:02.997060 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:03.647121 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:03.647085 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b4xp2"] Nov 03 22:29:03.649448 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:03.649424 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b4xp2" Nov 03 22:29:03.702738 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:03.702704 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 03 22:29:03.717143 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:03.717118 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b4xp2\" (UID: \"f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b4xp2" Nov 03 22:29:03.717268 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:03.717155 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjzqf\" (UniqueName: \"kubernetes.io/projected/f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0-kube-api-access-kjzqf\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b4xp2\" (UID: \"f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b4xp2" Nov 03 22:29:03.717268 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:03.717239 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b4xp2\" (UID: \"f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b4xp2" Nov 03 22:29:03.717350 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:03.717293 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b4xp2\" (UID: \"f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b4xp2" Nov 03 22:29:03.753402 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:03.753374 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-m95jv\"" Nov 03 22:29:03.753536 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:03.753463 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 03 22:29:03.753536 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:03.753490 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 03 22:29:03.755080 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:03.755059 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b4xp2"] Nov 03 22:29:03.802100 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:03.802051 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:03.817786 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:03.817763 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b4xp2\" (UID: \"f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b4xp2" Nov 03 22:29:03.817908 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:03.817795 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjzqf\" (UniqueName: \"kubernetes.io/projected/f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0-kube-api-access-kjzqf\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b4xp2\" (UID: \"f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b4xp2" Nov 03 22:29:03.817908 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:03.817834 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b4xp2\" (UID: \"f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b4xp2" Nov 03 22:29:03.817908 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:29:03.817876 5196 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Nov 03 22:29:03.818046 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:03.817934 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b4xp2\" (UID: \"f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b4xp2" Nov 03 22:29:03.818046 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:29:03.817950 5196 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0-proxy-source podName:f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0 nodeName:}" failed. No retries permitted until 2026-11-03 22:29:04.31793111 +0000 UTC m=+373.627490399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-b4xp2" (UID: "f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0") : configmap "usage-tenancy-proxy-source" not found Nov 03 22:29:03.818178 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:29:03.818159 5196 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Nov 03 22:29:03.818241 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:29:03.818227 5196 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0-tls-cert podName:f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0 nodeName:}" failed. No retries permitted until 2026-11-03 22:29:04.318208612 +0000 UTC m=+373.627767900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-b4xp2" (UID: "f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0") : secret "usage-logs-tenancy-proxy-tls" not found Nov 03 22:29:03.818482 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:03.818461 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b4xp2\" (UID: \"f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b4xp2" Nov 03 22:29:03.882923 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:03.882899 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjzqf\" (UniqueName: \"kubernetes.io/projected/f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0-kube-api-access-kjzqf\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b4xp2\" (UID: \"f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b4xp2" Nov 03 22:29:04.002165 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:04.002073 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:04.320735 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:04.320658 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b4xp2\" (UID: \"f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b4xp2" Nov 03 22:29:04.320735 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:04.320711 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b4xp2\" (UID: \"f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b4xp2" Nov 03 22:29:04.321479 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:04.321447 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b4xp2\" (UID: \"f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b4xp2" Nov 03 22:29:04.323169 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:04.323148 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b4xp2\" (UID: \"f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b4xp2" Nov 03 22:29:04.560954 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:04.560927 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b4xp2" Nov 03 22:29:04.932810 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:04.932783 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b4xp2"] Nov 03 22:29:04.935054 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:29:04.935029 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d8baf9_a8dc_4101_82b6_a6ab8c7adbf0.slice/crio-e5a01d051da0404b7cab039ca19068a303a0995e7670d9baf15ee3107276a1aa WatchSource:0}: Error finding container e5a01d051da0404b7cab039ca19068a303a0995e7670d9baf15ee3107276a1aa: Status 404 returned error can't find the container with id e5a01d051da0404b7cab039ca19068a303a0995e7670d9baf15ee3107276a1aa Nov 03 22:29:05.512847 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:05.512814 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b4xp2" event={"ID":"f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0","Type":"ContainerStarted","Data":"e5a01d051da0404b7cab039ca19068a303a0995e7670d9baf15ee3107276a1aa"} Nov 03 22:29:12.500110 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:12.500074 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-s69fp" Nov 03 22:29:12.500636 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:12.500378 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8657649675-k4qdp" Nov 03 22:29:12.936685 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:12.936644 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-s69fp"] Nov 03 22:29:12.937012 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:12.936908 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-s69fp" podUID="6135f6a9-74f7-49d5-961b-3e3d877e835e" containerName="manager" containerID="cri-o://35d2b957e3b3646ecb87a11cb9a0280a79f2be895e91abef443e90c8b5cab4df" gracePeriod=30 Nov 03 22:29:13.203727 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:13.203666 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-s69fp" Nov 03 22:29:13.297370 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:13.297331 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6135f6a9-74f7-49d5-961b-3e3d877e835e-kube-api-access-9r6cd\" (UniqueName: \"kubernetes.io/projected/6135f6a9-74f7-49d5-961b-3e3d877e835e-kube-api-access-9r6cd\") pod \"6135f6a9-74f7-49d5-961b-3e3d877e835e\" (UID: \"6135f6a9-74f7-49d5-961b-3e3d877e835e\") " Nov 03 22:29:13.297554 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:13.297441 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6135f6a9-74f7-49d5-961b-3e3d877e835e-webhook-cert\" (UniqueName: \"kubernetes.io/secret/6135f6a9-74f7-49d5-961b-3e3d877e835e-webhook-cert\") pod \"6135f6a9-74f7-49d5-961b-3e3d877e835e\" (UID: \"6135f6a9-74f7-49d5-961b-3e3d877e835e\") " Nov 03 22:29:13.297554 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:13.297475 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6135f6a9-74f7-49d5-961b-3e3d877e835e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6135f6a9-74f7-49d5-961b-3e3d877e835e-metrics-certs\") pod \"6135f6a9-74f7-49d5-961b-3e3d877e835e\" (UID: \"6135f6a9-74f7-49d5-961b-3e3d877e835e\") " Nov 03 22:29:13.300166 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:13.300129 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6135f6a9-74f7-49d5-961b-3e3d877e835e-kube-api-access-9r6cd" pod "6135f6a9-74f7-49d5-961b-3e3d877e835e" (UID: "6135f6a9-74f7-49d5-961b-3e3d877e835e"). InnerVolumeSpecName "kube-api-access-9r6cd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:29:13.300166 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:13.300146 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6135f6a9-74f7-49d5-961b-3e3d877e835e-metrics-certs" pod "6135f6a9-74f7-49d5-961b-3e3d877e835e" (UID: "6135f6a9-74f7-49d5-961b-3e3d877e835e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:29:13.300339 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:13.300230 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6135f6a9-74f7-49d5-961b-3e3d877e835e-webhook-cert" pod "6135f6a9-74f7-49d5-961b-3e3d877e835e" (UID: "6135f6a9-74f7-49d5-961b-3e3d877e835e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:29:13.398390 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:13.398353 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r6cd\" (UniqueName: \"kubernetes.io/projected/6135f6a9-74f7-49d5-961b-3e3d877e835e-kube-api-access-9r6cd\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:29:13.398390 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:13.398386 5196 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6135f6a9-74f7-49d5-961b-3e3d877e835e-webhook-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:29:13.398611 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:13.398402 5196 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6135f6a9-74f7-49d5-961b-3e3d877e835e-metrics-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:29:13.546035 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:13.545948 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:13.558525 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:13.558496 5196 generic.go:356] "Generic (PLEG): container finished" podID="6135f6a9-74f7-49d5-961b-3e3d877e835e" containerID="35d2b957e3b3646ecb87a11cb9a0280a79f2be895e91abef443e90c8b5cab4df" exitCode=0 Nov 03 22:29:13.558682 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:13.558555 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-s69fp" Nov 03 22:29:13.558682 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:13.558583 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-s69fp" event={"ID":"6135f6a9-74f7-49d5-961b-3e3d877e835e","Type":"ContainerDied","Data":"35d2b957e3b3646ecb87a11cb9a0280a79f2be895e91abef443e90c8b5cab4df"} Nov 03 22:29:13.558682 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:13.558648 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-s69fp" event={"ID":"6135f6a9-74f7-49d5-961b-3e3d877e835e","Type":"ContainerDied","Data":"fb35713038550b93d55ee51d9ea7da66b9db611229016090dd9f8bfab5ae49e5"} Nov 03 22:29:13.558851 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:13.558685 5196 scope.go:122] "RemoveContainer" containerID="35d2b957e3b3646ecb87a11cb9a0280a79f2be895e91abef443e90c8b5cab4df" Nov 03 22:29:13.569287 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:13.569271 5196 scope.go:122] "RemoveContainer" containerID="35d2b957e3b3646ecb87a11cb9a0280a79f2be895e91abef443e90c8b5cab4df" Nov 03 22:29:13.569593 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:29:13.569571 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35d2b957e3b3646ecb87a11cb9a0280a79f2be895e91abef443e90c8b5cab4df\": container with ID starting with 35d2b957e3b3646ecb87a11cb9a0280a79f2be895e91abef443e90c8b5cab4df not found: ID does not exist" containerID="35d2b957e3b3646ecb87a11cb9a0280a79f2be895e91abef443e90c8b5cab4df" Nov 03 22:29:13.569704 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:13.569598 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35d2b957e3b3646ecb87a11cb9a0280a79f2be895e91abef443e90c8b5cab4df"} err="failed to get container status \"35d2b957e3b3646ecb87a11cb9a0280a79f2be895e91abef443e90c8b5cab4df\": rpc error: code = NotFound desc = could not find container \"35d2b957e3b3646ecb87a11cb9a0280a79f2be895e91abef443e90c8b5cab4df\": container with ID starting with 35d2b957e3b3646ecb87a11cb9a0280a79f2be895e91abef443e90c8b5cab4df not found: ID does not exist" Nov 03 22:29:13.736519 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:13.736486 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-s69fp"] Nov 03 22:29:13.787207 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:13.787170 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-s69fp"] Nov 03 22:29:14.814879 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:14.814830 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6135f6a9-74f7-49d5-961b-3e3d877e835e" path="/var/lib/kubelet/pods/6135f6a9-74f7-49d5-961b-3e3d877e835e/volumes" Nov 03 22:29:17.581053 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:17.580965 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b4xp2" event={"ID":"f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0","Type":"ContainerStarted","Data":"c1490ec5b048478422e4d2f2442bfcf1405a7a2b23bfac6d3bd883f4185d1403"} Nov 03 22:29:17.581450 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:17.581120 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b4xp2" Nov 03 22:29:17.701951 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:17.701904 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b4xp2" podStartSLOduration=2.4976751200000002 podStartE2EDuration="14.701890679s" totalImagesPullingTime="12.204215559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:17.700447356 +0000 UTC m=+387.010006651" watchObservedRunningTime="2026-11-03 22:29:17.701890679 +0000 UTC m=+387.011449971" Nov 03 22:29:25.917579 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:29:25.917534 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:29:27.839246 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:27.839217 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:27.987056 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:27.987022 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:28.138490 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:28.138410 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:28.286701 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:28.286667 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:28.432139 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:28.432058 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:28.593278 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:28.593240 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:28.594605 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:28.594583 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b4xp2" Nov 03 22:29:28.706242 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:28.706165 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-s2rsn"] Nov 03 22:29:28.706485 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:28.706473 5196 cpu_manager.go:414] "containerMap: removing container" podUID="6135f6a9-74f7-49d5-961b-3e3d877e835e" containerName="manager" Nov 03 22:29:28.706529 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:28.706486 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6135f6a9-74f7-49d5-961b-3e3d877e835e" containerName="manager" Nov 03 22:29:28.706567 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:28.706554 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="6135f6a9-74f7-49d5-961b-3e3d877e835e" containerName="manager" Nov 03 22:29:28.709700 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:28.709684 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-s2rsn" Nov 03 22:29:28.751366 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:28.751340 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:28.763144 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:28.763118 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 03 22:29:28.763144 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:28.763128 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-n79js\"" Nov 03 22:29:28.768899 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:28.768849 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-s2rsn"] Nov 03 22:29:28.823290 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:28.823265 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-8gvc7"] Nov 03 22:29:28.826548 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:28.826531 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8gvc7" Nov 03 22:29:28.834097 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:28.834072 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89fdaa67-48e4-45c9-aca2-5bf2d3125d38-cert\") pod \"kserve-controller-manager-776cc66d47-s2rsn\" (UID: \"89fdaa67-48e4-45c9-aca2-5bf2d3125d38\") " pod="opendatahub/kserve-controller-manager-776cc66d47-s2rsn" Nov 03 22:29:28.834247 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:28.834148 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mq7t\" (UniqueName: \"kubernetes.io/projected/89fdaa67-48e4-45c9-aca2-5bf2d3125d38-kube-api-access-8mq7t\") pod \"kserve-controller-manager-776cc66d47-s2rsn\" (UID: \"89fdaa67-48e4-45c9-aca2-5bf2d3125d38\") " pod="opendatahub/kserve-controller-manager-776cc66d47-s2rsn" Nov 03 22:29:28.879737 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:28.879713 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-27985\"" Nov 03 22:29:28.880109 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:28.879837 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 03 22:29:28.888441 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:28.888419 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-8gvc7"] Nov 03 22:29:28.934910 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:28.934849 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89fdaa67-48e4-45c9-aca2-5bf2d3125d38-cert\") pod \"kserve-controller-manager-776cc66d47-s2rsn\" (UID: \"89fdaa67-48e4-45c9-aca2-5bf2d3125d38\") " pod="opendatahub/kserve-controller-manager-776cc66d47-s2rsn" Nov 03 22:29:28.935042 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:28.934961 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9fed221-31d5-489f-a844-c02ccc1c9853-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-8gvc7\" (UID: \"a9fed221-31d5-489f-a844-c02ccc1c9853\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8gvc7" Nov 03 22:29:28.935042 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:28.934995 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mq7t\" (UniqueName: \"kubernetes.io/projected/89fdaa67-48e4-45c9-aca2-5bf2d3125d38-kube-api-access-8mq7t\") pod \"kserve-controller-manager-776cc66d47-s2rsn\" (UID: \"89fdaa67-48e4-45c9-aca2-5bf2d3125d38\") " pod="opendatahub/kserve-controller-manager-776cc66d47-s2rsn" Nov 03 22:29:28.935137 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:28.935043 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9v86\" (UniqueName: \"kubernetes.io/projected/a9fed221-31d5-489f-a844-c02ccc1c9853-kube-api-access-h9v86\") pod \"llmisvc-controller-manager-869d5dcc5d-8gvc7\" (UID: \"a9fed221-31d5-489f-a844-c02ccc1c9853\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8gvc7" Nov 03 22:29:28.937358 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:28.937337 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89fdaa67-48e4-45c9-aca2-5bf2d3125d38-cert\") pod \"kserve-controller-manager-776cc66d47-s2rsn\" (UID: \"89fdaa67-48e4-45c9-aca2-5bf2d3125d38\") " pod="opendatahub/kserve-controller-manager-776cc66d47-s2rsn" Nov 03 22:29:28.997687 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:28.997619 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mq7t\" (UniqueName: \"kubernetes.io/projected/89fdaa67-48e4-45c9-aca2-5bf2d3125d38-kube-api-access-8mq7t\") pod \"kserve-controller-manager-776cc66d47-s2rsn\" (UID: \"89fdaa67-48e4-45c9-aca2-5bf2d3125d38\") " pod="opendatahub/kserve-controller-manager-776cc66d47-s2rsn" Nov 03 22:29:29.001217 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:29.001187 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:29.022173 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:29.022152 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-s2rsn" Nov 03 22:29:29.036196 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:29.036176 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9fed221-31d5-489f-a844-c02ccc1c9853-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-8gvc7\" (UID: \"a9fed221-31d5-489f-a844-c02ccc1c9853\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8gvc7" Nov 03 22:29:29.036292 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:29.036220 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9v86\" (UniqueName: \"kubernetes.io/projected/a9fed221-31d5-489f-a844-c02ccc1c9853-kube-api-access-h9v86\") pod \"llmisvc-controller-manager-869d5dcc5d-8gvc7\" (UID: \"a9fed221-31d5-489f-a844-c02ccc1c9853\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8gvc7" Nov 03 22:29:29.038522 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:29.038503 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9fed221-31d5-489f-a844-c02ccc1c9853-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-8gvc7\" (UID: \"a9fed221-31d5-489f-a844-c02ccc1c9853\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8gvc7" Nov 03 22:29:29.098329 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:29.098304 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9v86\" (UniqueName: \"kubernetes.io/projected/a9fed221-31d5-489f-a844-c02ccc1c9853-kube-api-access-h9v86\") pod \"llmisvc-controller-manager-869d5dcc5d-8gvc7\" (UID: \"a9fed221-31d5-489f-a844-c02ccc1c9853\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8gvc7" Nov 03 22:29:29.141416 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:29.141391 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8gvc7" Nov 03 22:29:29.414147 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:29.414120 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-s2rsn"] Nov 03 22:29:29.416517 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:29:29.416489 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89fdaa67_48e4_45c9_aca2_5bf2d3125d38.slice/crio-1b391be8a757578d5594657f860c266c6b8d117a2480dc236ee480a995574ce4 WatchSource:0}: Error finding container 1b391be8a757578d5594657f860c266c6b8d117a2480dc236ee480a995574ce4: Status 404 returned error can't find the container with id 1b391be8a757578d5594657f860c266c6b8d117a2480dc236ee480a995574ce4 Nov 03 22:29:29.514340 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:29.514312 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-8gvc7"] Nov 03 22:29:29.516193 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:29:29.516168 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda9fed221_31d5_489f_a844_c02ccc1c9853.slice/crio-33a741a2182765e3dc121f1017ff3c949d67928d369acecebd2b360177c7aadd WatchSource:0}: Error finding container 33a741a2182765e3dc121f1017ff3c949d67928d369acecebd2b360177c7aadd: Status 404 returned error can't find the container with id 33a741a2182765e3dc121f1017ff3c949d67928d369acecebd2b360177c7aadd Nov 03 22:29:29.638279 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:29.638244 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8gvc7" event={"ID":"a9fed221-31d5-489f-a844-c02ccc1c9853","Type":"ContainerStarted","Data":"33a741a2182765e3dc121f1017ff3c949d67928d369acecebd2b360177c7aadd"} Nov 03 22:29:29.639370 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:29.639348 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-s2rsn" event={"ID":"89fdaa67-48e4-45c9-aca2-5bf2d3125d38","Type":"ContainerStarted","Data":"1b391be8a757578d5594657f860c266c6b8d117a2480dc236ee480a995574ce4"} Nov 03 22:29:30.237529 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:30.237494 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:30.545941 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:30.545164 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:30.700909 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:30.690637 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:31.790940 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:31.790908 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:31.941974 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:31.941935 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:32.657261 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:32.657217 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8gvc7" event={"ID":"a9fed221-31d5-489f-a844-c02ccc1c9853","Type":"ContainerStarted","Data":"10b96119ee04403d096dc7f28c43efb967bb8da0fc344d73ffc420d7789966d8"} Nov 03 22:29:32.657437 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:32.657407 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8gvc7" Nov 03 22:29:32.658871 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:32.658830 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-s2rsn" event={"ID":"89fdaa67-48e4-45c9-aca2-5bf2d3125d38","Type":"ContainerStarted","Data":"8b46b07f409c36d78fd30ec7905c74cda307730e02087520ce07ed28a9eaefb5"} Nov 03 22:29:32.659008 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:32.658984 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-s2rsn" Nov 03 22:29:32.892893 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:32.892825 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-s2rsn" podStartSLOduration=2.46840419 podStartE2EDuration="4.892810188s" totalImagesPullingTime="2.424405998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:32.891804977 +0000 UTC m=+402.201364271" watchObservedRunningTime="2026-11-03 22:29:32.892810188 +0000 UTC m=+402.202369481" Nov 03 22:29:32.893273 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:32.893227 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8gvc7" podStartSLOduration=2.6382022000000003 podStartE2EDuration="4.893219465s" totalImagesPullingTime="2.255017265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:32.774558255 +0000 UTC m=+402.084117549" watchObservedRunningTime="2026-11-03 22:29:32.893219465 +0000 UTC m=+402.202778759" Nov 03 22:29:33.765649 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:33.765608 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:33.931450 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:33.931419 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:34.132364 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:34.132328 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:34.278045 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:34.278004 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:34.865267 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:34.865232 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-tsmf7"] Nov 03 22:29:34.872272 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:34.872256 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tsmf7" Nov 03 22:29:34.978146 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:34.978114 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:29:34.978539 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:34.978517 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 22:29:34.978539 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:34.978520 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:29:34.978661 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:34.978582 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sw68g\"" Nov 03 22:29:34.978661 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:34.978605 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 22:29:34.980281 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:34.980262 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-tsmf7"] Nov 03 22:29:34.993715 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:34.993684 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd7ad0fe-0939-478c-aeaa-00869b958f3f-maas-api-tls\") pod \"maas-api-6d669c7fcd-tsmf7\" (UID: \"bd7ad0fe-0939-478c-aeaa-00869b958f3f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tsmf7" Nov 03 22:29:34.993837 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:34.993726 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd7ad0fe-0939-478c-aeaa-00869b958f3f-metrics-certs\") pod \"maas-api-6d669c7fcd-tsmf7\" (UID: \"bd7ad0fe-0939-478c-aeaa-00869b958f3f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tsmf7" Nov 03 22:29:34.993837 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:34.993754 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh8gr\" (UniqueName: \"kubernetes.io/projected/bd7ad0fe-0939-478c-aeaa-00869b958f3f-kube-api-access-rh8gr\") pod \"maas-api-6d669c7fcd-tsmf7\" (UID: \"bd7ad0fe-0939-478c-aeaa-00869b958f3f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tsmf7" Nov 03 22:29:34.993949 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:34.993890 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd7ad0fe-0939-478c-aeaa-00869b958f3f-openshift-service-ca\") pod \"maas-api-6d669c7fcd-tsmf7\" (UID: \"bd7ad0fe-0939-478c-aeaa-00869b958f3f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tsmf7" Nov 03 22:29:35.094878 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:35.094818 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd7ad0fe-0939-478c-aeaa-00869b958f3f-maas-api-tls\") pod \"maas-api-6d669c7fcd-tsmf7\" (UID: \"bd7ad0fe-0939-478c-aeaa-00869b958f3f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tsmf7" Nov 03 22:29:35.095042 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:35.094900 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd7ad0fe-0939-478c-aeaa-00869b958f3f-metrics-certs\") pod \"maas-api-6d669c7fcd-tsmf7\" (UID: \"bd7ad0fe-0939-478c-aeaa-00869b958f3f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tsmf7" Nov 03 22:29:35.095042 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:35.094943 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh8gr\" (UniqueName: \"kubernetes.io/projected/bd7ad0fe-0939-478c-aeaa-00869b958f3f-kube-api-access-rh8gr\") pod \"maas-api-6d669c7fcd-tsmf7\" (UID: \"bd7ad0fe-0939-478c-aeaa-00869b958f3f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tsmf7" Nov 03 22:29:35.095042 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:35.094992 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd7ad0fe-0939-478c-aeaa-00869b958f3f-openshift-service-ca\") pod \"maas-api-6d669c7fcd-tsmf7\" (UID: \"bd7ad0fe-0939-478c-aeaa-00869b958f3f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tsmf7" Nov 03 22:29:35.095669 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:35.095643 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd7ad0fe-0939-478c-aeaa-00869b958f3f-openshift-service-ca\") pod \"maas-api-6d669c7fcd-tsmf7\" (UID: \"bd7ad0fe-0939-478c-aeaa-00869b958f3f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tsmf7" Nov 03 22:29:35.097493 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:35.097474 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd7ad0fe-0939-478c-aeaa-00869b958f3f-maas-api-tls\") pod \"maas-api-6d669c7fcd-tsmf7\" (UID: \"bd7ad0fe-0939-478c-aeaa-00869b958f3f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tsmf7" Nov 03 22:29:35.097591 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:35.097518 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd7ad0fe-0939-478c-aeaa-00869b958f3f-metrics-certs\") pod \"maas-api-6d669c7fcd-tsmf7\" (UID: \"bd7ad0fe-0939-478c-aeaa-00869b958f3f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tsmf7" Nov 03 22:29:35.157540 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:35.157456 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh8gr\" (UniqueName: \"kubernetes.io/projected/bd7ad0fe-0939-478c-aeaa-00869b958f3f-kube-api-access-rh8gr\") pod \"maas-api-6d669c7fcd-tsmf7\" (UID: \"bd7ad0fe-0939-478c-aeaa-00869b958f3f\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tsmf7" Nov 03 22:29:35.184499 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:35.184472 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tsmf7" Nov 03 22:29:35.309190 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:35.309159 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:35.451809 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:35.451723 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:35.569263 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:35.569239 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-tsmf7"] Nov 03 22:29:35.571631 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:29:35.571596 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd7ad0fe_0939_478c_aeaa_00869b958f3f.slice/crio-90ce1b27738cc5cb6ea32121bc455e78c8035248906e05d1d5211a4fb9b4a7d8 WatchSource:0}: Error finding container 90ce1b27738cc5cb6ea32121bc455e78c8035248906e05d1d5211a4fb9b4a7d8: Status 404 returned error can't find the container with id 90ce1b27738cc5cb6ea32121bc455e78c8035248906e05d1d5211a4fb9b4a7d8 Nov 03 22:29:35.596429 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:35.596387 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:35.675344 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:35.675311 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tsmf7" event={"ID":"bd7ad0fe-0939-478c-aeaa-00869b958f3f","Type":"ContainerStarted","Data":"90ce1b27738cc5cb6ea32121bc455e78c8035248906e05d1d5211a4fb9b4a7d8"} Nov 03 22:29:35.737943 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:35.737874 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:35.880254 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:35.880220 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:36.022884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:36.022785 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:36.171873 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:36.170224 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:36.330406 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:36.330306 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:36.474497 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:36.474465 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:37.687526 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:37.687482 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tsmf7" event={"ID":"bd7ad0fe-0939-478c-aeaa-00869b958f3f","Type":"ContainerStarted","Data":"2f8a82dc8ecdb281d1e0c9c37884a01c0eb48249048ec83f7140530c5d1cee48"} Nov 03 22:29:37.687988 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:37.687629 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tsmf7" Nov 03 22:29:37.808345 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:37.808290 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tsmf7" podStartSLOduration=2.043553873 podStartE2EDuration="3.808276008s" totalImagesPullingTime="1.764722135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:37.807041301 +0000 UTC m=+407.116600594" watchObservedRunningTime="2026-11-03 22:29:37.808276008 +0000 UTC m=+407.117835301" Nov 03 22:29:41.524918 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:41.524886 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:41.672983 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:41.672937 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:43.716088 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:43.716058 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tsmf7" Nov 03 22:29:44.235869 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:44.235830 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:44.544890 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:44.544794 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:29:45.411824 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:45.410646 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f4f5fc5-qvkmq"] Nov 03 22:29:45.474501 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:45.474476 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f4f5fc5-qvkmq"] Nov 03 22:29:45.474652 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:45.474576 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f4f5fc5-qvkmq" Nov 03 22:29:45.527728 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:45.527694 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 03 22:29:45.586217 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:45.586182 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/65bb3dd1-64a9-4c80-bdf4-b5b68fee9128-tls-cert\") pod \"authorino-6f4f5fc5-qvkmq\" (UID: \"65bb3dd1-64a9-4c80-bdf4-b5b68fee9128\") " pod="kuadrant-system/authorino-6f4f5fc5-qvkmq" Nov 03 22:29:45.586359 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:45.586230 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbk6m\" (UniqueName: \"kubernetes.io/projected/65bb3dd1-64a9-4c80-bdf4-b5b68fee9128-kube-api-access-bbk6m\") pod \"authorino-6f4f5fc5-qvkmq\" (UID: \"65bb3dd1-64a9-4c80-bdf4-b5b68fee9128\") " pod="kuadrant-system/authorino-6f4f5fc5-qvkmq" Nov 03 22:29:45.687530 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:45.687447 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbk6m\" (UniqueName: \"kubernetes.io/projected/65bb3dd1-64a9-4c80-bdf4-b5b68fee9128-kube-api-access-bbk6m\") pod \"authorino-6f4f5fc5-qvkmq\" (UID: \"65bb3dd1-64a9-4c80-bdf4-b5b68fee9128\") " pod="kuadrant-system/authorino-6f4f5fc5-qvkmq" Nov 03 22:29:45.687530 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:45.687518 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/65bb3dd1-64a9-4c80-bdf4-b5b68fee9128-tls-cert\") pod \"authorino-6f4f5fc5-qvkmq\" (UID: \"65bb3dd1-64a9-4c80-bdf4-b5b68fee9128\") " pod="kuadrant-system/authorino-6f4f5fc5-qvkmq" Nov 03 22:29:45.691892 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:45.690012 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/65bb3dd1-64a9-4c80-bdf4-b5b68fee9128-tls-cert\") pod \"authorino-6f4f5fc5-qvkmq\" (UID: \"65bb3dd1-64a9-4c80-bdf4-b5b68fee9128\") " pod="kuadrant-system/authorino-6f4f5fc5-qvkmq" Nov 03 22:29:45.749346 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:45.749319 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbk6m\" (UniqueName: \"kubernetes.io/projected/65bb3dd1-64a9-4c80-bdf4-b5b68fee9128-kube-api-access-bbk6m\") pod \"authorino-6f4f5fc5-qvkmq\" (UID: \"65bb3dd1-64a9-4c80-bdf4-b5b68fee9128\") " pod="kuadrant-system/authorino-6f4f5fc5-qvkmq" Nov 03 22:29:45.788896 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:45.788872 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f4f5fc5-qvkmq" Nov 03 22:29:46.214600 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:46.214576 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f4f5fc5-qvkmq"] Nov 03 22:29:46.217977 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:29:46.217949 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65bb3dd1_64a9_4c80_bdf4_b5b68fee9128.slice/crio-e92d1d866f3fc605ddd37537aa90d87c798d523b3e1604411cf0fb230629c822 WatchSource:0}: Error finding container e92d1d866f3fc605ddd37537aa90d87c798d523b3e1604411cf0fb230629c822: Status 404 returned error can't find the container with id e92d1d866f3fc605ddd37537aa90d87c798d523b3e1604411cf0fb230629c822 Nov 03 22:29:46.748599 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:46.748558 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f4f5fc5-qvkmq" event={"ID":"65bb3dd1-64a9-4c80-bdf4-b5b68fee9128","Type":"ContainerStarted","Data":"e92d1d866f3fc605ddd37537aa90d87c798d523b3e1604411cf0fb230629c822"} Nov 03 22:29:47.756234 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:47.756195 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f4f5fc5-qvkmq" event={"ID":"65bb3dd1-64a9-4c80-bdf4-b5b68fee9128","Type":"ContainerStarted","Data":"7047b5ed81a11ab5abb3da2b402d2c40efa5f852c77abbfcd6c615748316e0f0"} Nov 03 22:29:47.876600 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:47.876532 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f4f5fc5-qvkmq" podStartSLOduration=1.523288595 podStartE2EDuration="2.876511362s" totalImagesPullingTime="1.353222767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:47.874114289 +0000 UTC m=+417.183673581" watchObservedRunningTime="2026-11-03 22:29:47.876511362 +0000 UTC m=+417.186070674" Nov 03 22:29:47.903724 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:47.903686 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xrzzs"] Nov 03 22:29:47.903939 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:47.903911 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-xrzzs" podUID="c3dfa765-6b3e-435f-8d41-59eb8a1974ec" containerName="authorino" containerID="cri-o://e1fe09d7be256c732c5b56a2f03ab18592dc21fa4788ccc3f304e1dadc100d1f" gracePeriod=30 Nov 03 22:29:48.180521 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:48.180494 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-xrzzs" Nov 03 22:29:48.313591 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:48.313558 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3dfa765-6b3e-435f-8d41-59eb8a1974ec-kube-api-access-cl49w\" (UniqueName: \"kubernetes.io/projected/c3dfa765-6b3e-435f-8d41-59eb8a1974ec-kube-api-access-cl49w\") pod \"c3dfa765-6b3e-435f-8d41-59eb8a1974ec\" (UID: \"c3dfa765-6b3e-435f-8d41-59eb8a1974ec\") " Nov 03 22:29:48.316262 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:48.316229 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3dfa765-6b3e-435f-8d41-59eb8a1974ec-kube-api-access-cl49w" pod "c3dfa765-6b3e-435f-8d41-59eb8a1974ec" (UID: "c3dfa765-6b3e-435f-8d41-59eb8a1974ec"). InnerVolumeSpecName "kube-api-access-cl49w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:29:48.414971 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:48.414895 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl49w\" (UniqueName: \"kubernetes.io/projected/c3dfa765-6b3e-435f-8d41-59eb8a1974ec-kube-api-access-cl49w\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:29:48.762950 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:48.762848 5196 generic.go:356] "Generic (PLEG): container finished" podID="c3dfa765-6b3e-435f-8d41-59eb8a1974ec" containerID="e1fe09d7be256c732c5b56a2f03ab18592dc21fa4788ccc3f304e1dadc100d1f" exitCode=0 Nov 03 22:29:48.762950 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:48.762936 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-xrzzs" event={"ID":"c3dfa765-6b3e-435f-8d41-59eb8a1974ec","Type":"ContainerDied","Data":"e1fe09d7be256c732c5b56a2f03ab18592dc21fa4788ccc3f304e1dadc100d1f"} Nov 03 22:29:48.763546 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:48.762992 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-xrzzs" event={"ID":"c3dfa765-6b3e-435f-8d41-59eb8a1974ec","Type":"ContainerDied","Data":"3248b0d6ab5afc04e454f882f863b1ae6ed45373e8c38bc691e5e390ede271cf"} Nov 03 22:29:48.763546 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:48.763025 5196 scope.go:122] "RemoveContainer" containerID="e1fe09d7be256c732c5b56a2f03ab18592dc21fa4788ccc3f304e1dadc100d1f" Nov 03 22:29:48.763546 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:48.763036 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-xrzzs" Nov 03 22:29:48.775937 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:48.775900 5196 scope.go:122] "RemoveContainer" containerID="e1fe09d7be256c732c5b56a2f03ab18592dc21fa4788ccc3f304e1dadc100d1f" Nov 03 22:29:48.777898 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:29:48.777873 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1fe09d7be256c732c5b56a2f03ab18592dc21fa4788ccc3f304e1dadc100d1f\": container with ID starting with e1fe09d7be256c732c5b56a2f03ab18592dc21fa4788ccc3f304e1dadc100d1f not found: ID does not exist" containerID="e1fe09d7be256c732c5b56a2f03ab18592dc21fa4788ccc3f304e1dadc100d1f" Nov 03 22:29:48.777998 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:48.777905 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1fe09d7be256c732c5b56a2f03ab18592dc21fa4788ccc3f304e1dadc100d1f"} err="failed to get container status \"e1fe09d7be256c732c5b56a2f03ab18592dc21fa4788ccc3f304e1dadc100d1f\": rpc error: code = NotFound desc = could not find container \"e1fe09d7be256c732c5b56a2f03ab18592dc21fa4788ccc3f304e1dadc100d1f\": container with ID starting with e1fe09d7be256c732c5b56a2f03ab18592dc21fa4788ccc3f304e1dadc100d1f not found: ID does not exist" Nov 03 22:29:48.928778 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:48.928747 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xrzzs"] Nov 03 22:29:48.930885 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:48.930845 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xrzzs"] Nov 03 22:29:49.921253 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:49.770924 5196 generic.go:356] "Generic (PLEG): container finished" podID="aa3310c5-bd35-4bef-b838-1722c7143cf5" containerID="94185fe4b0d8ffd8765aacdeece9a13a1b59a1924ca7c60218f604135935965c" exitCode=0 Nov 03 22:29:49.921253 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:49.770957 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" event={"ID":"aa3310c5-bd35-4bef-b838-1722c7143cf5","Type":"ContainerDied","Data":"94185fe4b0d8ffd8765aacdeece9a13a1b59a1924ca7c60218f604135935965c"} Nov 03 22:29:49.921253 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:49.771292 5196 scope.go:122] "RemoveContainer" containerID="94185fe4b0d8ffd8765aacdeece9a13a1b59a1924ca7c60218f604135935965c" Nov 03 22:29:49.921253 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:49.772441 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:29:49.921253 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:49.772508 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:29:49.921253 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:49.772543 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:29:50.777523 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:50.777487 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" event={"ID":"aa3310c5-bd35-4bef-b838-1722c7143cf5","Type":"ContainerStarted","Data":"779c2ca89aa4a9f4f3fa65ee3dc70cf070ed23f5d9088c47c1d008705cf57287"} Nov 03 22:29:50.818426 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:50.818397 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3dfa765-6b3e-435f-8d41-59eb8a1974ec" path="/var/lib/kubelet/pods/c3dfa765-6b3e-435f-8d41-59eb8a1974ec/volumes" Nov 03 22:29:50.936109 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:50.936081 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:29:50.936494 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:50.936254 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-nqmgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body= Nov 03 22:29:50.936494 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:50.936310 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" podUID="aa3310c5-bd35-4bef-b838-1722c7143cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" Nov 03 22:29:51.936361 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:51.936323 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-nqmgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body= Nov 03 22:29:51.936725 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:51.936388 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" podUID="aa3310c5-bd35-4bef-b838-1722c7143cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" Nov 03 22:29:52.936174 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:52.936140 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-nqmgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body= Nov 03 22:29:52.936347 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:52.936192 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" podUID="aa3310c5-bd35-4bef-b838-1722c7143cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" Nov 03 22:29:53.935940 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:53.935908 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-nqmgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body= Nov 03 22:29:53.936335 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:53.935961 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" podUID="aa3310c5-bd35-4bef-b838-1722c7143cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" Nov 03 22:29:54.624195 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:54.624160 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-tsmf7"] Nov 03 22:29:54.624394 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:54.624369 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tsmf7" podUID="bd7ad0fe-0939-478c-aeaa-00869b958f3f" containerName="maas-api" containerID="cri-o://2f8a82dc8ecdb281d1e0c9c37884a01c0eb48249048ec83f7140530c5d1cee48" gracePeriod=30 Nov 03 22:29:54.824881 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:54.824818 5196 generic.go:356] "Generic (PLEG): container finished" podID="bd7ad0fe-0939-478c-aeaa-00869b958f3f" containerID="2f8a82dc8ecdb281d1e0c9c37884a01c0eb48249048ec83f7140530c5d1cee48" exitCode=0 Nov 03 22:29:54.825038 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:54.824896 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tsmf7" event={"ID":"bd7ad0fe-0939-478c-aeaa-00869b958f3f","Type":"ContainerDied","Data":"2f8a82dc8ecdb281d1e0c9c37884a01c0eb48249048ec83f7140530c5d1cee48"} Nov 03 22:29:54.922743 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:54.922719 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tsmf7" Nov 03 22:29:54.935998 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:54.935962 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-nqmgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body= Nov 03 22:29:54.936418 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:54.936017 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" podUID="aa3310c5-bd35-4bef-b838-1722c7143cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" Nov 03 22:29:54.971420 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:54.971393 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd7ad0fe-0939-478c-aeaa-00869b958f3f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd7ad0fe-0939-478c-aeaa-00869b958f3f-openshift-service-ca\") pod \"bd7ad0fe-0939-478c-aeaa-00869b958f3f\" (UID: \"bd7ad0fe-0939-478c-aeaa-00869b958f3f\") " Nov 03 22:29:54.971543 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:54.971447 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd7ad0fe-0939-478c-aeaa-00869b958f3f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd7ad0fe-0939-478c-aeaa-00869b958f3f-metrics-certs\") pod \"bd7ad0fe-0939-478c-aeaa-00869b958f3f\" (UID: \"bd7ad0fe-0939-478c-aeaa-00869b958f3f\") " Nov 03 22:29:54.971543 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:54.971468 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd7ad0fe-0939-478c-aeaa-00869b958f3f-kube-api-access-rh8gr\" (UniqueName: \"kubernetes.io/projected/bd7ad0fe-0939-478c-aeaa-00869b958f3f-kube-api-access-rh8gr\") pod \"bd7ad0fe-0939-478c-aeaa-00869b958f3f\" (UID: \"bd7ad0fe-0939-478c-aeaa-00869b958f3f\") " Nov 03 22:29:54.971543 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:54.971513 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd7ad0fe-0939-478c-aeaa-00869b958f3f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd7ad0fe-0939-478c-aeaa-00869b958f3f-maas-api-tls\") pod \"bd7ad0fe-0939-478c-aeaa-00869b958f3f\" (UID: \"bd7ad0fe-0939-478c-aeaa-00869b958f3f\") " Nov 03 22:29:54.971774 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:54.971751 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd7ad0fe-0939-478c-aeaa-00869b958f3f-openshift-service-ca" pod "bd7ad0fe-0939-478c-aeaa-00869b958f3f" (UID: "bd7ad0fe-0939-478c-aeaa-00869b958f3f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:29:54.973810 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:54.973778 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd7ad0fe-0939-478c-aeaa-00869b958f3f-maas-api-tls" pod "bd7ad0fe-0939-478c-aeaa-00869b958f3f" (UID: "bd7ad0fe-0939-478c-aeaa-00869b958f3f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:29:54.973933 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:54.973833 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd7ad0fe-0939-478c-aeaa-00869b958f3f-kube-api-access-rh8gr" pod "bd7ad0fe-0939-478c-aeaa-00869b958f3f" (UID: "bd7ad0fe-0939-478c-aeaa-00869b958f3f"). InnerVolumeSpecName "kube-api-access-rh8gr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:29:54.973933 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:54.973835 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd7ad0fe-0939-478c-aeaa-00869b958f3f-metrics-certs" pod "bd7ad0fe-0939-478c-aeaa-00869b958f3f" (UID: "bd7ad0fe-0939-478c-aeaa-00869b958f3f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:29:55.072298 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:55.072255 5196 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd7ad0fe-0939-478c-aeaa-00869b958f3f-openshift-service-ca\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:29:55.072298 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:55.072290 5196 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd7ad0fe-0939-478c-aeaa-00869b958f3f-metrics-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:29:55.072298 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:55.072301 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh8gr\" (UniqueName: \"kubernetes.io/projected/bd7ad0fe-0939-478c-aeaa-00869b958f3f-kube-api-access-rh8gr\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:29:55.072559 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:55.072310 5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd7ad0fe-0939-478c-aeaa-00869b958f3f-maas-api-tls\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:29:55.831255 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:55.831226 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tsmf7" Nov 03 22:29:55.831404 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:55.831261 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tsmf7" event={"ID":"bd7ad0fe-0939-478c-aeaa-00869b958f3f","Type":"ContainerDied","Data":"90ce1b27738cc5cb6ea32121bc455e78c8035248906e05d1d5211a4fb9b4a7d8"} Nov 03 22:29:55.831404 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:55.831325 5196 scope.go:122] "RemoveContainer" containerID="2f8a82dc8ecdb281d1e0c9c37884a01c0eb48249048ec83f7140530c5d1cee48" Nov 03 22:29:55.918572 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:29:55.918528 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:29:55.939582 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:55.939554 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:29:55.939954 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:55.939890 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:29:55.940466 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:55.940452 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:29:55.998367 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:55.998339 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-tsmf7"] Nov 03 22:29:56.048685 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:56.048663 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-tsmf7"] Nov 03 22:29:56.813972 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:56.813942 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd7ad0fe-0939-478c-aeaa-00869b958f3f" path="/var/lib/kubelet/pods/bd7ad0fe-0939-478c-aeaa-00869b958f3f/volumes" Nov 03 22:29:58.368525 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:29:58.368484 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:00.165108 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:00.165073 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895750-9cx48"] Nov 03 22:30:00.165512 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:00.165409 5196 cpu_manager.go:414] "containerMap: removing container" podUID="c3dfa765-6b3e-435f-8d41-59eb8a1974ec" containerName="authorino" Nov 03 22:30:00.165512 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:00.165419 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3dfa765-6b3e-435f-8d41-59eb8a1974ec" containerName="authorino" Nov 03 22:30:00.165512 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:00.165450 5196 cpu_manager.go:414] "containerMap: removing container" podUID="bd7ad0fe-0939-478c-aeaa-00869b958f3f" containerName="maas-api" Nov 03 22:30:00.165512 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:00.165455 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd7ad0fe-0939-478c-aeaa-00869b958f3f" containerName="maas-api" Nov 03 22:30:00.165512 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:00.165505 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3dfa765-6b3e-435f-8d41-59eb8a1974ec" containerName="authorino" Nov 03 22:30:00.165676 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:00.165516 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd7ad0fe-0939-478c-aeaa-00869b958f3f" containerName="maas-api" Nov 03 22:30:00.168227 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:00.168212 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-9cx48" Nov 03 22:30:00.221924 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:00.221897 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:30:00.222068 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:00.221942 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:30:00.272572 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:00.272542 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:30:00.274348 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:00.274321 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895750-9cx48"] Nov 03 22:30:00.319485 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:00.319446 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/652aaed8-4a57-4779-80fc-db3a6206045b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895750-9cx48\" (UID: \"652aaed8-4a57-4779-80fc-db3a6206045b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-9cx48" Nov 03 22:30:00.319485 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:00.319486 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp7ss\" (UniqueName: \"kubernetes.io/projected/652aaed8-4a57-4779-80fc-db3a6206045b-kube-api-access-hp7ss\") pod \"maas-api-key-cleanup-29895750-9cx48\" (UID: \"652aaed8-4a57-4779-80fc-db3a6206045b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-9cx48" Nov 03 22:30:00.420273 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:00.420195 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/652aaed8-4a57-4779-80fc-db3a6206045b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895750-9cx48\" (UID: \"652aaed8-4a57-4779-80fc-db3a6206045b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-9cx48" Nov 03 22:30:00.420273 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:00.420226 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp7ss\" (UniqueName: \"kubernetes.io/projected/652aaed8-4a57-4779-80fc-db3a6206045b-kube-api-access-hp7ss\") pod \"maas-api-key-cleanup-29895750-9cx48\" (UID: \"652aaed8-4a57-4779-80fc-db3a6206045b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-9cx48" Nov 03 22:30:00.420803 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:00.420784 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/652aaed8-4a57-4779-80fc-db3a6206045b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895750-9cx48\" (UID: \"652aaed8-4a57-4779-80fc-db3a6206045b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-9cx48" Nov 03 22:30:00.482305 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:00.482275 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp7ss\" (UniqueName: \"kubernetes.io/projected/652aaed8-4a57-4779-80fc-db3a6206045b-kube-api-access-hp7ss\") pod \"maas-api-key-cleanup-29895750-9cx48\" (UID: \"652aaed8-4a57-4779-80fc-db3a6206045b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-9cx48" Nov 03 22:30:00.779775 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:00.779692 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-9cx48" Nov 03 22:30:01.151525 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:01.151504 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895750-9cx48"] Nov 03 22:30:01.153909 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:30:01.153849 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod652aaed8_4a57_4779_80fc_db3a6206045b.slice/crio-89ddd1878ecfbe602f881f2a9bcc208437a1ce21ced96e1666c4eb907cbf965c WatchSource:0}: Error finding container 89ddd1878ecfbe602f881f2a9bcc208437a1ce21ced96e1666c4eb907cbf965c: Status 404 returned error can't find the container with id 89ddd1878ecfbe602f881f2a9bcc208437a1ce21ced96e1666c4eb907cbf965c Nov 03 22:30:01.861604 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:01.861571 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-9cx48" event={"ID":"652aaed8-4a57-4779-80fc-db3a6206045b","Type":"ContainerStarted","Data":"89ddd1878ecfbe602f881f2a9bcc208437a1ce21ced96e1666c4eb907cbf965c"} Nov 03 22:30:03.330564 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:03.330533 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:30:03.666669 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:03.666636 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8gvc7" Nov 03 22:30:03.669564 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:03.669545 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-s2rsn" Nov 03 22:30:05.883574 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:05.883541 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-9cx48" event={"ID":"652aaed8-4a57-4779-80fc-db3a6206045b","Type":"ContainerStarted","Data":"a680c7afcded6b31401f545d8fa2a2efbb4b08afdf987b9acc9f3cdae02800ba"} Nov 03 22:30:06.001655 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:06.001558 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-9cx48" podStartSLOduration=1.762601802 podStartE2EDuration="6.001541953s" totalImagesPullingTime="4.238940151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:06.000129653 +0000 UTC m=+435.309688970" watchObservedRunningTime="2026-11-03 22:30:06.001541953 +0000 UTC m=+435.311101272" Nov 03 22:30:06.889048 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:06.889017 5196 generic.go:356] "Generic (PLEG): container finished" podID="652aaed8-4a57-4779-80fc-db3a6206045b" containerID="a680c7afcded6b31401f545d8fa2a2efbb4b08afdf987b9acc9f3cdae02800ba" exitCode=60 Nov 03 22:30:06.889522 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:06.889107 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-9cx48" event={"ID":"652aaed8-4a57-4779-80fc-db3a6206045b","Type":"ContainerDied","Data":"a680c7afcded6b31401f545d8fa2a2efbb4b08afdf987b9acc9f3cdae02800ba"} Nov 03 22:30:06.889522 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:06.889347 5196 scope.go:122] "RemoveContainer" containerID="a680c7afcded6b31401f545d8fa2a2efbb4b08afdf987b9acc9f3cdae02800ba" Nov 03 22:30:07.895891 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:07.895838 5196 generic.go:356] "Generic (PLEG): container finished" podID="652aaed8-4a57-4779-80fc-db3a6206045b" containerID="891aeb07582ef3a901cc5e719c9b870a62de1ef5501eb55d5ea5b2c8d8a9400a" exitCode=60 Nov 03 22:30:07.896358 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:07.895895 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-9cx48" event={"ID":"652aaed8-4a57-4779-80fc-db3a6206045b","Type":"ContainerDied","Data":"891aeb07582ef3a901cc5e719c9b870a62de1ef5501eb55d5ea5b2c8d8a9400a"} Nov 03 22:30:07.896358 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:07.895958 5196 scope.go:122] "RemoveContainer" containerID="a680c7afcded6b31401f545d8fa2a2efbb4b08afdf987b9acc9f3cdae02800ba" Nov 03 22:30:07.896358 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:07.896197 5196 scope.go:122] "RemoveContainer" containerID="891aeb07582ef3a901cc5e719c9b870a62de1ef5501eb55d5ea5b2c8d8a9400a" Nov 03 22:30:07.896467 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:30:07.896443 5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895750-9cx48_odh-ai-gateway-infra(652aaed8-4a57-4779-80fc-db3a6206045b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-9cx48" podUID="652aaed8-4a57-4779-80fc-db3a6206045b" Nov 03 22:30:08.902183 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:08.902152 5196 scope.go:122] "RemoveContainer" containerID="891aeb07582ef3a901cc5e719c9b870a62de1ef5501eb55d5ea5b2c8d8a9400a" Nov 03 22:30:08.902551 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:30:08.902344 5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895750-9cx48_odh-ai-gateway-infra(652aaed8-4a57-4779-80fc-db3a6206045b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-9cx48" podUID="652aaed8-4a57-4779-80fc-db3a6206045b" Nov 03 22:30:19.808136 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:19.808106 5196 scope.go:122] "RemoveContainer" containerID="891aeb07582ef3a901cc5e719c9b870a62de1ef5501eb55d5ea5b2c8d8a9400a" Nov 03 22:30:20.960230 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:20.960199 5196 generic.go:356] "Generic (PLEG): container finished" podID="652aaed8-4a57-4779-80fc-db3a6206045b" containerID="119cced84cfb6fb277f7b174cde80d9d9b743f7a93826093cb0cd436dfe3ab1e" exitCode=60 Nov 03 22:30:20.960712 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:20.960283 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-9cx48" event={"ID":"652aaed8-4a57-4779-80fc-db3a6206045b","Type":"ContainerDied","Data":"119cced84cfb6fb277f7b174cde80d9d9b743f7a93826093cb0cd436dfe3ab1e"} Nov 03 22:30:20.960712 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:20.960354 5196 scope.go:122] "RemoveContainer" containerID="891aeb07582ef3a901cc5e719c9b870a62de1ef5501eb55d5ea5b2c8d8a9400a" Nov 03 22:30:20.960712 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:20.960676 5196 scope.go:122] "RemoveContainer" containerID="119cced84cfb6fb277f7b174cde80d9d9b743f7a93826093cb0cd436dfe3ab1e" Nov 03 22:30:20.960949 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:30:20.960929 5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895750-9cx48_odh-ai-gateway-infra(652aaed8-4a57-4779-80fc-db3a6206045b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-9cx48" podUID="652aaed8-4a57-4779-80fc-db3a6206045b" Nov 03 22:30:22.085754 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:22.085725 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895750-9cx48"] Nov 03 22:30:22.216586 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:22.216560 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-9cx48" Nov 03 22:30:22.314024 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:22.313993 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/652aaed8-4a57-4779-80fc-db3a6206045b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/652aaed8-4a57-4779-80fc-db3a6206045b-maas-api-service-ca\") pod \"652aaed8-4a57-4779-80fc-db3a6206045b\" (UID: \"652aaed8-4a57-4779-80fc-db3a6206045b\") " Nov 03 22:30:22.314200 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:22.314119 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/652aaed8-4a57-4779-80fc-db3a6206045b-kube-api-access-hp7ss\" (UniqueName: \"kubernetes.io/projected/652aaed8-4a57-4779-80fc-db3a6206045b-kube-api-access-hp7ss\") pod \"652aaed8-4a57-4779-80fc-db3a6206045b\" (UID: \"652aaed8-4a57-4779-80fc-db3a6206045b\") " Nov 03 22:30:22.314379 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:22.314354 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/652aaed8-4a57-4779-80fc-db3a6206045b-maas-api-service-ca" pod "652aaed8-4a57-4779-80fc-db3a6206045b" (UID: "652aaed8-4a57-4779-80fc-db3a6206045b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:30:22.316314 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:22.316290 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/652aaed8-4a57-4779-80fc-db3a6206045b-kube-api-access-hp7ss" pod "652aaed8-4a57-4779-80fc-db3a6206045b" (UID: "652aaed8-4a57-4779-80fc-db3a6206045b"). InnerVolumeSpecName "kube-api-access-hp7ss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:22.415037 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:22.415009 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp7ss\" (UniqueName: \"kubernetes.io/projected/652aaed8-4a57-4779-80fc-db3a6206045b-kube-api-access-hp7ss\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:30:22.415037 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:22.415033 5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/652aaed8-4a57-4779-80fc-db3a6206045b-maas-api-service-ca\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:30:22.972714 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:22.972688 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-9cx48" Nov 03 22:30:22.972714 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:22.972696 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-9cx48" event={"ID":"652aaed8-4a57-4779-80fc-db3a6206045b","Type":"ContainerDied","Data":"89ddd1878ecfbe602f881f2a9bcc208437a1ce21ced96e1666c4eb907cbf965c"} Nov 03 22:30:22.972952 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:22.972757 5196 scope.go:122] "RemoveContainer" containerID="119cced84cfb6fb277f7b174cde80d9d9b743f7a93826093cb0cd436dfe3ab1e" Nov 03 22:30:23.139493 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:23.139463 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895750-9cx48"] Nov 03 22:30:23.143739 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:23.143719 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895750-9cx48"] Nov 03 22:30:24.813210 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:24.813175 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="652aaed8-4a57-4779-80fc-db3a6206045b" path="/var/lib/kubelet/pods/652aaed8-4a57-4779-80fc-db3a6206045b/volumes" Nov 03 22:30:25.917677 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:30:25.917639 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:30:49.329032 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:49.329001 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-hhp7p"] Nov 03 22:30:49.329506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:49.329338 5196 cpu_manager.go:414] "containerMap: removing container" podUID="652aaed8-4a57-4779-80fc-db3a6206045b" containerName="cleanup" Nov 03 22:30:49.329506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:49.329351 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="652aaed8-4a57-4779-80fc-db3a6206045b" containerName="cleanup" Nov 03 22:30:49.329506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:49.329371 5196 cpu_manager.go:414] "containerMap: removing container" podUID="652aaed8-4a57-4779-80fc-db3a6206045b" containerName="cleanup" Nov 03 22:30:49.329506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:49.329376 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="652aaed8-4a57-4779-80fc-db3a6206045b" containerName="cleanup" Nov 03 22:30:49.329506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:49.329447 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="652aaed8-4a57-4779-80fc-db3a6206045b" containerName="cleanup" Nov 03 22:30:49.329506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:49.329456 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="652aaed8-4a57-4779-80fc-db3a6206045b" containerName="cleanup" Nov 03 22:30:49.329506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:49.329463 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="652aaed8-4a57-4779-80fc-db3a6206045b" containerName="cleanup" Nov 03 22:30:49.335733 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:49.335718 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-hhp7p" Nov 03 22:30:49.391153 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:49.391115 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-hhp7p"] Nov 03 22:30:49.450955 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:49.448293 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7307cd39-a4a5-45d6-aeee-434e9c640246-cert\") pod \"llmisvc-controller-manager-6466c554d8-hhp7p\" (UID: \"7307cd39-a4a5-45d6-aeee-434e9c640246\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-hhp7p" Nov 03 22:30:49.450955 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:49.448355 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4r6j\" (UniqueName: \"kubernetes.io/projected/7307cd39-a4a5-45d6-aeee-434e9c640246-kube-api-access-v4r6j\") pod \"llmisvc-controller-manager-6466c554d8-hhp7p\" (UID: \"7307cd39-a4a5-45d6-aeee-434e9c640246\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-hhp7p" Nov 03 22:30:49.549611 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:49.549574 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7307cd39-a4a5-45d6-aeee-434e9c640246-cert\") pod \"llmisvc-controller-manager-6466c554d8-hhp7p\" (UID: \"7307cd39-a4a5-45d6-aeee-434e9c640246\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-hhp7p" Nov 03 22:30:49.549785 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:49.549626 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4r6j\" (UniqueName: \"kubernetes.io/projected/7307cd39-a4a5-45d6-aeee-434e9c640246-kube-api-access-v4r6j\") pod \"llmisvc-controller-manager-6466c554d8-hhp7p\" (UID: \"7307cd39-a4a5-45d6-aeee-434e9c640246\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-hhp7p" Nov 03 22:30:49.552178 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:49.552148 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7307cd39-a4a5-45d6-aeee-434e9c640246-cert\") pod \"llmisvc-controller-manager-6466c554d8-hhp7p\" (UID: \"7307cd39-a4a5-45d6-aeee-434e9c640246\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-hhp7p" Nov 03 22:30:49.611469 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:49.611440 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4r6j\" (UniqueName: \"kubernetes.io/projected/7307cd39-a4a5-45d6-aeee-434e9c640246-kube-api-access-v4r6j\") pod \"llmisvc-controller-manager-6466c554d8-hhp7p\" (UID: \"7307cd39-a4a5-45d6-aeee-434e9c640246\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-hhp7p" Nov 03 22:30:49.648315 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:49.648290 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-hhp7p" Nov 03 22:30:50.023783 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:50.023757 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-hhp7p"] Nov 03 22:30:50.026371 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:30:50.026340 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7307cd39_a4a5_45d6_aeee_434e9c640246.slice/crio-20ef96fadfe175dceaf81eb7190ac3deae144ef4edaf22331b7c274884fdf254 WatchSource:0}: Error finding container 20ef96fadfe175dceaf81eb7190ac3deae144ef4edaf22331b7c274884fdf254: Status 404 returned error can't find the container with id 20ef96fadfe175dceaf81eb7190ac3deae144ef4edaf22331b7c274884fdf254 Nov 03 22:30:50.098916 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:50.098887 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-hhp7p" event={"ID":"7307cd39-a4a5-45d6-aeee-434e9c640246","Type":"ContainerStarted","Data":"20ef96fadfe175dceaf81eb7190ac3deae144ef4edaf22331b7c274884fdf254"} Nov 03 22:30:51.104688 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:51.104656 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-hhp7p" event={"ID":"7307cd39-a4a5-45d6-aeee-434e9c640246","Type":"ContainerStarted","Data":"d4dc2e03253376207ee4569cd971bfd7f8de96c40b39526cfb3a136a78e4782f"} Nov 03 22:30:51.105099 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:51.104727 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-hhp7p" Nov 03 22:30:51.223464 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:51.223414 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-hhp7p" podStartSLOduration=1.444427008 podStartE2EDuration="2.223398761s" totalImagesPullingTime="778.971753ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:51.222137588 +0000 UTC m=+480.531696881" watchObservedRunningTime="2026-11-03 22:30:51.223398761 +0000 UTC m=+480.532958090" Nov 03 22:30:55.917431 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:30:55.917388 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:30:58.750127 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:30:58.750100 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:03.683813 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:31:03.683773 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:08.057790 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:31:08.057757 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:22.112096 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:31:22.112063 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-hhp7p" Nov 03 22:31:22.282088 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:31:22.282055 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-8gvc7"] Nov 03 22:31:22.282354 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:31:22.282329 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8gvc7" podUID="a9fed221-31d5-489f-a844-c02ccc1c9853" containerName="manager" containerID="cri-o://10b96119ee04403d096dc7f28c43efb967bb8da0fc344d73ffc420d7789966d8" gracePeriod=30 Nov 03 22:31:22.535379 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:31:22.535355 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8gvc7" Nov 03 22:31:22.627757 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:31:22.627726 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9fed221-31d5-489f-a844-c02ccc1c9853-cert\" (UniqueName: \"kubernetes.io/secret/a9fed221-31d5-489f-a844-c02ccc1c9853-cert\") pod \"a9fed221-31d5-489f-a844-c02ccc1c9853\" (UID: \"a9fed221-31d5-489f-a844-c02ccc1c9853\") " Nov 03 22:31:22.627945 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:31:22.627768 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9fed221-31d5-489f-a844-c02ccc1c9853-kube-api-access-h9v86\" (UniqueName: \"kubernetes.io/projected/a9fed221-31d5-489f-a844-c02ccc1c9853-kube-api-access-h9v86\") pod \"a9fed221-31d5-489f-a844-c02ccc1c9853\" (UID: \"a9fed221-31d5-489f-a844-c02ccc1c9853\") " Nov 03 22:31:22.630091 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:31:22.630056 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9fed221-31d5-489f-a844-c02ccc1c9853-cert" pod "a9fed221-31d5-489f-a844-c02ccc1c9853" (UID: "a9fed221-31d5-489f-a844-c02ccc1c9853"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:31:22.630214 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:31:22.630160 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9fed221-31d5-489f-a844-c02ccc1c9853-kube-api-access-h9v86" pod "a9fed221-31d5-489f-a844-c02ccc1c9853" (UID: "a9fed221-31d5-489f-a844-c02ccc1c9853"). InnerVolumeSpecName "kube-api-access-h9v86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:31:22.728746 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:31:22.728676 5196 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9fed221-31d5-489f-a844-c02ccc1c9853-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:31:22.728746 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:31:22.728702 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9v86\" (UniqueName: \"kubernetes.io/projected/a9fed221-31d5-489f-a844-c02ccc1c9853-kube-api-access-h9v86\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:31:23.104656 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:31:23.104628 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:23.251458 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:31:23.251428 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:23.256290 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:31:23.256263 5196 generic.go:356] "Generic (PLEG): container finished" podID="a9fed221-31d5-489f-a844-c02ccc1c9853" containerID="10b96119ee04403d096dc7f28c43efb967bb8da0fc344d73ffc420d7789966d8" exitCode=0 Nov 03 22:31:23.256433 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:31:23.256314 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8gvc7" Nov 03 22:31:23.256433 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:31:23.256354 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8gvc7" event={"ID":"a9fed221-31d5-489f-a844-c02ccc1c9853","Type":"ContainerDied","Data":"10b96119ee04403d096dc7f28c43efb967bb8da0fc344d73ffc420d7789966d8"} Nov 03 22:31:23.256433 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:31:23.256411 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-8gvc7" event={"ID":"a9fed221-31d5-489f-a844-c02ccc1c9853","Type":"ContainerDied","Data":"33a741a2182765e3dc121f1017ff3c949d67928d369acecebd2b360177c7aadd"} Nov 03 22:31:23.256433 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:31:23.256432 5196 scope.go:122] "RemoveContainer" containerID="10b96119ee04403d096dc7f28c43efb967bb8da0fc344d73ffc420d7789966d8" Nov 03 22:31:23.264874 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:31:23.264835 5196 scope.go:122] "RemoveContainer" containerID="10b96119ee04403d096dc7f28c43efb967bb8da0fc344d73ffc420d7789966d8" Nov 03 22:31:23.265127 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:31:23.265107 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10b96119ee04403d096dc7f28c43efb967bb8da0fc344d73ffc420d7789966d8\": container with ID starting with 10b96119ee04403d096dc7f28c43efb967bb8da0fc344d73ffc420d7789966d8 not found: ID does not exist" containerID="10b96119ee04403d096dc7f28c43efb967bb8da0fc344d73ffc420d7789966d8" Nov 03 22:31:23.265189 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:31:23.265138 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10b96119ee04403d096dc7f28c43efb967bb8da0fc344d73ffc420d7789966d8"} err="failed to get container status \"10b96119ee04403d096dc7f28c43efb967bb8da0fc344d73ffc420d7789966d8\": rpc error: code = NotFound desc = could not find container \"10b96119ee04403d096dc7f28c43efb967bb8da0fc344d73ffc420d7789966d8\": container with ID starting with 10b96119ee04403d096dc7f28c43efb967bb8da0fc344d73ffc420d7789966d8 not found: ID does not exist" Nov 03 22:31:23.429095 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:31:23.429020 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-8gvc7"] Nov 03 22:31:23.479705 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:31:23.479674 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-8gvc7"] Nov 03 22:31:24.813241 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:31:24.813205 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9fed221-31d5-489f-a844-c02ccc1c9853" path="/var/lib/kubelet/pods/a9fed221-31d5-489f-a844-c02ccc1c9853/volumes" Nov 03 22:31:25.918058 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:31:25.918019 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:31:29.056744 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:31:29.056705 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:30.291136 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:31:30.291102 5196 generic.go:356] "Generic (PLEG): container finished" podID="69b6bfce-f5ca-4c52-a9b1-5749383d2370" containerID="ea84c679a1684e677cf3fe9c886e526d3439eedbea1ef2da0a75fb7cf18b960c" exitCode=1 Nov 03 22:31:30.291519 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:31:30.291140 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8657649675-k4qdp" event={"ID":"69b6bfce-f5ca-4c52-a9b1-5749383d2370","Type":"ContainerDied","Data":"ea84c679a1684e677cf3fe9c886e526d3439eedbea1ef2da0a75fb7cf18b960c"} Nov 03 22:31:30.291519 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:31:30.291492 5196 scope.go:122] "RemoveContainer" containerID="ea84c679a1684e677cf3fe9c886e526d3439eedbea1ef2da0a75fb7cf18b960c" Nov 03 22:31:32.302641 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:31:32.302605 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8657649675-k4qdp" event={"ID":"69b6bfce-f5ca-4c52-a9b1-5749383d2370","Type":"ContainerStarted","Data":"17a4819a1111b76dfc2ec3cec7039813a057fa0b5f3df4b3edfd03d67bd6dbde"} Nov 03 22:31:32.303049 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:31:32.302844 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8657649675-k4qdp" Nov 03 22:31:42.403176 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:31:42.403146 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:43.309961 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:31:43.309934 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8657649675-k4qdp" Nov 03 22:31:46.929985 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:31:46.929951 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:55.087731 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:31:55.087696 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:55.917371 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:31:55.917336 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:31:58.580324 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:31:58.580281 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:59.056895 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:31:59.056828 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:31:59.308314 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:31:59.308215 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:32:02.347156 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:02.347112 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:32:02.836679 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:02.836594 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn"] Nov 03 22:32:02.836984 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:02.836967 5196 cpu_manager.go:414] "containerMap: removing container" podUID="a9fed221-31d5-489f-a844-c02ccc1c9853" containerName="manager" Nov 03 22:32:02.837032 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:02.836986 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9fed221-31d5-489f-a844-c02ccc1c9853" containerName="manager" Nov 03 22:32:02.837032 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:02.836995 5196 cpu_manager.go:414] "containerMap: removing container" podUID="652aaed8-4a57-4779-80fc-db3a6206045b" containerName="cleanup" Nov 03 22:32:02.837032 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:02.837000 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="652aaed8-4a57-4779-80fc-db3a6206045b" containerName="cleanup" Nov 03 22:32:02.837127 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:02.837081 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9fed221-31d5-489f-a844-c02ccc1c9853" containerName="manager" Nov 03 22:32:02.840059 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:02.840044 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" Nov 03 22:32:02.875042 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:02.875013 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26ad9218-689e-4989-8fb5-4f311773e668-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vs7zn\" (UID: \"26ad9218-689e-4989-8fb5-4f311773e668\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" Nov 03 22:32:02.875169 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:02.875054 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ad9218-689e-4989-8fb5-4f311773e668-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vs7zn\" (UID: \"26ad9218-689e-4989-8fb5-4f311773e668\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" Nov 03 22:32:02.875169 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:02.875073 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26ad9218-689e-4989-8fb5-4f311773e668-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vs7zn\" (UID: \"26ad9218-689e-4989-8fb5-4f311773e668\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" Nov 03 22:32:02.875169 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:02.875157 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26ad9218-689e-4989-8fb5-4f311773e668-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vs7zn\" (UID: \"26ad9218-689e-4989-8fb5-4f311773e668\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" Nov 03 22:32:02.875276 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:02.875184 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26ad9218-689e-4989-8fb5-4f311773e668-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vs7zn\" (UID: \"26ad9218-689e-4989-8fb5-4f311773e668\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" Nov 03 22:32:02.875276 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:02.875213 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ad9218-689e-4989-8fb5-4f311773e668-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vs7zn\" (UID: \"26ad9218-689e-4989-8fb5-4f311773e668\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" Nov 03 22:32:02.875341 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:02.875267 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26ad9218-689e-4989-8fb5-4f311773e668-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vs7zn\" (UID: \"26ad9218-689e-4989-8fb5-4f311773e668\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" Nov 03 22:32:02.875341 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:02.875292 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv6xm\" (UniqueName: \"kubernetes.io/projected/26ad9218-689e-4989-8fb5-4f311773e668-kube-api-access-cv6xm\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vs7zn\" (UID: \"26ad9218-689e-4989-8fb5-4f311773e668\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" Nov 03 22:32:02.875399 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:02.875358 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26ad9218-689e-4989-8fb5-4f311773e668-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vs7zn\" (UID: \"26ad9218-689e-4989-8fb5-4f311773e668\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" Nov 03 22:32:02.975805 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:02.975777 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ad9218-689e-4989-8fb5-4f311773e668-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vs7zn\" (UID: \"26ad9218-689e-4989-8fb5-4f311773e668\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" Nov 03 22:32:02.976004 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:02.975811 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26ad9218-689e-4989-8fb5-4f311773e668-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vs7zn\" (UID: \"26ad9218-689e-4989-8fb5-4f311773e668\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" Nov 03 22:32:02.976004 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:02.975835 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26ad9218-689e-4989-8fb5-4f311773e668-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vs7zn\" (UID: \"26ad9218-689e-4989-8fb5-4f311773e668\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" Nov 03 22:32:02.976004 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:02.975881 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26ad9218-689e-4989-8fb5-4f311773e668-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vs7zn\" (UID: \"26ad9218-689e-4989-8fb5-4f311773e668\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" Nov 03 22:32:02.976004 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:02.975926 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ad9218-689e-4989-8fb5-4f311773e668-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vs7zn\" (UID: \"26ad9218-689e-4989-8fb5-4f311773e668\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" Nov 03 22:32:02.976004 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:02.975957 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26ad9218-689e-4989-8fb5-4f311773e668-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vs7zn\" (UID: \"26ad9218-689e-4989-8fb5-4f311773e668\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" Nov 03 22:32:02.976318 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:02.976036 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv6xm\" (UniqueName: \"kubernetes.io/projected/26ad9218-689e-4989-8fb5-4f311773e668-kube-api-access-cv6xm\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vs7zn\" (UID: \"26ad9218-689e-4989-8fb5-4f311773e668\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" Nov 03 22:32:02.976318 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:02.976104 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26ad9218-689e-4989-8fb5-4f311773e668-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vs7zn\" (UID: \"26ad9218-689e-4989-8fb5-4f311773e668\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" Nov 03 22:32:02.976318 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:02.976179 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26ad9218-689e-4989-8fb5-4f311773e668-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vs7zn\" (UID: \"26ad9218-689e-4989-8fb5-4f311773e668\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" Nov 03 22:32:02.976318 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:02.976225 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26ad9218-689e-4989-8fb5-4f311773e668-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vs7zn\" (UID: \"26ad9218-689e-4989-8fb5-4f311773e668\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" Nov 03 22:32:02.976318 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:02.976287 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26ad9218-689e-4989-8fb5-4f311773e668-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vs7zn\" (UID: \"26ad9218-689e-4989-8fb5-4f311773e668\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" Nov 03 22:32:02.976573 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:02.976465 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ad9218-689e-4989-8fb5-4f311773e668-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vs7zn\" (UID: \"26ad9218-689e-4989-8fb5-4f311773e668\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" Nov 03 22:32:02.976573 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:02.976559 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ad9218-689e-4989-8fb5-4f311773e668-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vs7zn\" (UID: \"26ad9218-689e-4989-8fb5-4f311773e668\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" Nov 03 22:32:02.976952 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:02.976931 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26ad9218-689e-4989-8fb5-4f311773e668-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vs7zn\" (UID: \"26ad9218-689e-4989-8fb5-4f311773e668\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" Nov 03 22:32:02.978579 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:02.978556 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26ad9218-689e-4989-8fb5-4f311773e668-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vs7zn\" (UID: \"26ad9218-689e-4989-8fb5-4f311773e668\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" Nov 03 22:32:02.978757 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:02.978739 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26ad9218-689e-4989-8fb5-4f311773e668-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vs7zn\" (UID: \"26ad9218-689e-4989-8fb5-4f311773e668\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" Nov 03 22:32:03.038362 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:03.038336 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26ad9218-689e-4989-8fb5-4f311773e668-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vs7zn\" (UID: \"26ad9218-689e-4989-8fb5-4f311773e668\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" Nov 03 22:32:03.038562 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:03.038538 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv6xm\" (UniqueName: \"kubernetes.io/projected/26ad9218-689e-4989-8fb5-4f311773e668-kube-api-access-cv6xm\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vs7zn\" (UID: \"26ad9218-689e-4989-8fb5-4f311773e668\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" Nov 03 22:32:03.057092 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:03.057057 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn"] Nov 03 22:32:03.152915 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:03.152884 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" Nov 03 22:32:03.580223 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:03.580058 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn"] Nov 03 22:32:03.582356 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:32:03.582334 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ad9218_689e_4989_8fb5_4f311773e668.slice/crio-e5b89ac48c34eb3ad468801275c3d93ff0a0da7cc022ebd59fa87b3ef5f458f3 WatchSource:0}: Error finding container e5b89ac48c34eb3ad468801275c3d93ff0a0da7cc022ebd59fa87b3ef5f458f3: Status 404 returned error can't find the container with id e5b89ac48c34eb3ad468801275c3d93ff0a0da7cc022ebd59fa87b3ef5f458f3 Nov 03 22:32:03.584461 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:03.584431 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:32:03.584524 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:03.584499 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:32:03.584563 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:03.584525 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:32:04.061281 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:04.061242 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:32:04.468742 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:04.468697 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" event={"ID":"26ad9218-689e-4989-8fb5-4f311773e668","Type":"ContainerStarted","Data":"bac6b676c2ff28f05520593ed372758812778294eaacb203f01e7afd94597904"} Nov 03 22:32:04.468935 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:04.468765 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" event={"ID":"26ad9218-689e-4989-8fb5-4f311773e668","Type":"ContainerStarted","Data":"e5b89ac48c34eb3ad468801275c3d93ff0a0da7cc022ebd59fa87b3ef5f458f3"} Nov 03 22:32:04.598047 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:04.597792 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" podStartSLOduration=2.597769639 podStartE2EDuration="2.597769639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:04.595319995 +0000 UTC m=+553.904879289" watchObservedRunningTime="2026-11-03 22:32:04.597769639 +0000 UTC m=+553.907328933" Nov 03 22:32:05.153275 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:05.153230 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" Nov 03 22:32:05.155189 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:05.155046 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vs7zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 22:32:05.155189 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:05.155097 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" podUID="26ad9218-689e-4989-8fb5-4f311773e668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 22:32:05.369154 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:05.369119 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:32:06.154145 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:06.154105 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vs7zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 22:32:06.154654 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:06.154196 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" podUID="26ad9218-689e-4989-8fb5-4f311773e668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 22:32:07.153974 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:07.153940 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vs7zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 22:32:07.154148 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:07.153994 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" podUID="26ad9218-689e-4989-8fb5-4f311773e668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 22:32:08.153878 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:08.153813 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vs7zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 22:32:08.154344 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:08.153903 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" podUID="26ad9218-689e-4989-8fb5-4f311773e668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 22:32:09.116315 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:09.116270 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:32:09.153737 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:09.153701 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vs7zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 22:32:09.153917 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:09.153763 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" podUID="26ad9218-689e-4989-8fb5-4f311773e668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 22:32:11.194049 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:11.194010 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vs7zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:32:11.194573 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:11.194097 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" podUID="26ad9218-689e-4989-8fb5-4f311773e668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:32:11.727582 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:11.727545 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" Nov 03 22:32:11.727927 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:11.727889 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" Nov 03 22:32:11.729195 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:11.729176 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" Nov 03 22:32:13.455984 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:13.455941 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:32:15.823253 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:15.823212 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:32:17.955452 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:17.955417 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x"] Nov 03 22:32:17.959248 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:17.959228 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" Nov 03 22:32:18.075095 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:18.075058 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85a61150-46ba-406c-b384-99565f653f15-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vj28x\" (UID: \"85a61150-46ba-406c-b384-99565f653f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" Nov 03 22:32:18.075256 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:18.075112 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a61150-46ba-406c-b384-99565f653f15-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vj28x\" (UID: \"85a61150-46ba-406c-b384-99565f653f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" Nov 03 22:32:18.075256 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:18.075221 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85a61150-46ba-406c-b384-99565f653f15-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vj28x\" (UID: \"85a61150-46ba-406c-b384-99565f653f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" Nov 03 22:32:18.075348 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:18.075272 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85a61150-46ba-406c-b384-99565f653f15-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vj28x\" (UID: \"85a61150-46ba-406c-b384-99565f653f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" Nov 03 22:32:18.075398 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:18.075373 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85a61150-46ba-406c-b384-99565f653f15-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vj28x\" (UID: \"85a61150-46ba-406c-b384-99565f653f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" Nov 03 22:32:18.075447 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:18.075420 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85a61150-46ba-406c-b384-99565f653f15-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vj28x\" (UID: \"85a61150-46ba-406c-b384-99565f653f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" Nov 03 22:32:18.075549 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:18.075533 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9qkm\" (UniqueName: \"kubernetes.io/projected/85a61150-46ba-406c-b384-99565f653f15-kube-api-access-n9qkm\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vj28x\" (UID: \"85a61150-46ba-406c-b384-99565f653f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" Nov 03 22:32:18.075613 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:18.075593 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a61150-46ba-406c-b384-99565f653f15-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vj28x\" (UID: \"85a61150-46ba-406c-b384-99565f653f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" Nov 03 22:32:18.075697 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:18.075677 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85a61150-46ba-406c-b384-99565f653f15-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vj28x\" (UID: \"85a61150-46ba-406c-b384-99565f653f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" Nov 03 22:32:18.165988 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:18.165611 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x"] Nov 03 22:32:18.176296 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:18.176269 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a61150-46ba-406c-b384-99565f653f15-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vj28x\" (UID: \"85a61150-46ba-406c-b384-99565f653f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" Nov 03 22:32:18.176477 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:18.176305 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85a61150-46ba-406c-b384-99565f653f15-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vj28x\" (UID: \"85a61150-46ba-406c-b384-99565f653f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" Nov 03 22:32:18.176477 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:18.176324 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85a61150-46ba-406c-b384-99565f653f15-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vj28x\" (UID: \"85a61150-46ba-406c-b384-99565f653f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" Nov 03 22:32:18.176616 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:18.176594 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85a61150-46ba-406c-b384-99565f653f15-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vj28x\" (UID: \"85a61150-46ba-406c-b384-99565f653f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" Nov 03 22:32:18.176678 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:18.176644 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85a61150-46ba-406c-b384-99565f653f15-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vj28x\" (UID: \"85a61150-46ba-406c-b384-99565f653f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" Nov 03 22:32:18.176738 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:18.176694 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9qkm\" (UniqueName: \"kubernetes.io/projected/85a61150-46ba-406c-b384-99565f653f15-kube-api-access-n9qkm\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vj28x\" (UID: \"85a61150-46ba-406c-b384-99565f653f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" Nov 03 22:32:18.176738 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:18.176727 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a61150-46ba-406c-b384-99565f653f15-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vj28x\" (UID: \"85a61150-46ba-406c-b384-99565f653f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" Nov 03 22:32:18.176884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:18.176737 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85a61150-46ba-406c-b384-99565f653f15-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vj28x\" (UID: \"85a61150-46ba-406c-b384-99565f653f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" Nov 03 22:32:18.176884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:18.176759 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a61150-46ba-406c-b384-99565f653f15-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vj28x\" (UID: \"85a61150-46ba-406c-b384-99565f653f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" Nov 03 22:32:18.176884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:18.176795 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85a61150-46ba-406c-b384-99565f653f15-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vj28x\" (UID: \"85a61150-46ba-406c-b384-99565f653f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" Nov 03 22:32:18.176884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:18.176844 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85a61150-46ba-406c-b384-99565f653f15-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vj28x\" (UID: \"85a61150-46ba-406c-b384-99565f653f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" Nov 03 22:32:18.177130 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:18.177110 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a61150-46ba-406c-b384-99565f653f15-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vj28x\" (UID: \"85a61150-46ba-406c-b384-99565f653f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" Nov 03 22:32:18.177238 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:18.177216 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85a61150-46ba-406c-b384-99565f653f15-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vj28x\" (UID: \"85a61150-46ba-406c-b384-99565f653f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" Nov 03 22:32:18.177767 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:18.177746 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85a61150-46ba-406c-b384-99565f653f15-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vj28x\" (UID: \"85a61150-46ba-406c-b384-99565f653f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" Nov 03 22:32:18.179078 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:18.179049 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85a61150-46ba-406c-b384-99565f653f15-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vj28x\" (UID: \"85a61150-46ba-406c-b384-99565f653f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" Nov 03 22:32:18.179431 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:18.179416 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85a61150-46ba-406c-b384-99565f653f15-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vj28x\" (UID: \"85a61150-46ba-406c-b384-99565f653f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" Nov 03 22:32:18.238164 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:18.238112 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85a61150-46ba-406c-b384-99565f653f15-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vj28x\" (UID: \"85a61150-46ba-406c-b384-99565f653f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" Nov 03 22:32:18.238295 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:18.238273 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9qkm\" (UniqueName: \"kubernetes.io/projected/85a61150-46ba-406c-b384-99565f653f15-kube-api-access-n9qkm\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vj28x\" (UID: \"85a61150-46ba-406c-b384-99565f653f15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" Nov 03 22:32:18.272187 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:18.272165 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" Nov 03 22:32:18.700104 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:18.699972 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x"] Nov 03 22:32:18.702645 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:32:18.702614 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85a61150_46ba_406c_b384_99565f653f15.slice/crio-a4281f489c0c462c60335ea43c56b41caa0ae0174243b46a103a034ad7a96434 WatchSource:0}: Error finding container a4281f489c0c462c60335ea43c56b41caa0ae0174243b46a103a034ad7a96434: Status 404 returned error can't find the container with id a4281f489c0c462c60335ea43c56b41caa0ae0174243b46a103a034ad7a96434 Nov 03 22:32:18.707811 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:18.707773 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:32:18.707950 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:18.707849 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:32:18.707950 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:18.707906 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:32:19.556172 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:19.556132 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" event={"ID":"85a61150-46ba-406c-b384-99565f653f15","Type":"ContainerStarted","Data":"5af2747d92233beb4c13a39aa86448cace1e72168fd51edc1673ff7de72c9005"} Nov 03 22:32:19.556590 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:19.556186 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" event={"ID":"85a61150-46ba-406c-b384-99565f653f15","Type":"ContainerStarted","Data":"a4281f489c0c462c60335ea43c56b41caa0ae0174243b46a103a034ad7a96434"} Nov 03 22:32:19.779386 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:19.779330 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" podStartSLOduration=2.779314338 podStartE2EDuration="2.779314338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:19.728139891 +0000 UTC m=+569.037699196" watchObservedRunningTime="2026-11-03 22:32:19.779314338 +0000 UTC m=+569.088873869" Nov 03 22:32:20.272588 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:20.272552 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" Nov 03 22:32:20.273886 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:20.273841 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vj28x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 03 22:32:20.274054 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:20.273911 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" podUID="85a61150-46ba-406c-b384-99565f653f15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 03 22:32:21.273142 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:21.273097 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vj28x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 03 22:32:21.273514 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:21.273174 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" podUID="85a61150-46ba-406c-b384-99565f653f15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 03 22:32:22.272952 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:22.272913 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vj28x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 03 22:32:22.273124 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:22.272982 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" podUID="85a61150-46ba-406c-b384-99565f653f15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 03 22:32:23.272521 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:23.272482 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vj28x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 03 22:32:23.272931 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:23.272542 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" podUID="85a61150-46ba-406c-b384-99565f653f15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 03 22:32:24.273187 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:24.273153 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vj28x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 03 22:32:24.273582 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:24.273207 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" podUID="85a61150-46ba-406c-b384-99565f653f15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 03 22:32:25.273145 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:25.273106 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vj28x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 03 22:32:25.273374 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:25.273173 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" podUID="85a61150-46ba-406c-b384-99565f653f15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 03 22:32:25.919403 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:32:25.919364 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:32:27.272635 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:27.272587 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vj28x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:32:27.273067 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:27.272650 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" podUID="85a61150-46ba-406c-b384-99565f653f15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:32:28.314011 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:28.313973 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vj28x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:32:28.314537 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:28.314028 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" podUID="85a61150-46ba-406c-b384-99565f653f15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:32:28.649717 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:28.649684 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" Nov 03 22:32:28.650201 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:28.650178 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" Nov 03 22:32:28.651074 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:28.651059 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" Nov 03 22:32:32.559260 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:32.559222 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:32:32.707838 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:32.707798 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:32:43.640523 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:43.640493 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b98df9fc6-mz7zf"] Nov 03 22:32:43.644230 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:43.644208 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b98df9fc6-mz7zf" Nov 03 22:32:43.705315 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:43.705285 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b98df9fc6-mz7zf"] Nov 03 22:32:43.743228 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:43.743200 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6f268a42-a565-44bc-a2f9-318cfc884c5a-tls-cert\") pod \"authorino-b98df9fc6-mz7zf\" (UID: \"6f268a42-a565-44bc-a2f9-318cfc884c5a\") " pod="kuadrant-system/authorino-b98df9fc6-mz7zf" Nov 03 22:32:43.743381 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:43.743241 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlqkh\" (UniqueName: \"kubernetes.io/projected/6f268a42-a565-44bc-a2f9-318cfc884c5a-kube-api-access-mlqkh\") pod \"authorino-b98df9fc6-mz7zf\" (UID: \"6f268a42-a565-44bc-a2f9-318cfc884c5a\") " pod="kuadrant-system/authorino-b98df9fc6-mz7zf" Nov 03 22:32:43.835071 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:43.835040 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:32:43.844132 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:43.844107 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6f268a42-a565-44bc-a2f9-318cfc884c5a-tls-cert\") pod \"authorino-b98df9fc6-mz7zf\" (UID: \"6f268a42-a565-44bc-a2f9-318cfc884c5a\") " pod="kuadrant-system/authorino-b98df9fc6-mz7zf" Nov 03 22:32:43.844255 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:43.844141 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlqkh\" (UniqueName: \"kubernetes.io/projected/6f268a42-a565-44bc-a2f9-318cfc884c5a-kube-api-access-mlqkh\") pod \"authorino-b98df9fc6-mz7zf\" (UID: \"6f268a42-a565-44bc-a2f9-318cfc884c5a\") " pod="kuadrant-system/authorino-b98df9fc6-mz7zf" Nov 03 22:32:43.846820 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:43.846789 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6f268a42-a565-44bc-a2f9-318cfc884c5a-tls-cert\") pod \"authorino-b98df9fc6-mz7zf\" (UID: \"6f268a42-a565-44bc-a2f9-318cfc884c5a\") " pod="kuadrant-system/authorino-b98df9fc6-mz7zf" Nov 03 22:32:43.909615 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:43.909543 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlqkh\" (UniqueName: \"kubernetes.io/projected/6f268a42-a565-44bc-a2f9-318cfc884c5a-kube-api-access-mlqkh\") pod \"authorino-b98df9fc6-mz7zf\" (UID: \"6f268a42-a565-44bc-a2f9-318cfc884c5a\") " pod="kuadrant-system/authorino-b98df9fc6-mz7zf" Nov 03 22:32:43.956642 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:43.956615 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b98df9fc6-mz7zf" Nov 03 22:32:44.031219 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:44.031167 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:32:44.328174 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:44.328145 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b98df9fc6-mz7zf"] Nov 03 22:32:44.330799 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:32:44.330769 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f268a42_a565_44bc_a2f9_318cfc884c5a.slice/crio-b8b38767718e3dff3f34a89bc995af552c42f7c313e746a8d67f3686ffe7d696 WatchSource:0}: Error finding container b8b38767718e3dff3f34a89bc995af552c42f7c313e746a8d67f3686ffe7d696: Status 404 returned error can't find the container with id b8b38767718e3dff3f34a89bc995af552c42f7c313e746a8d67f3686ffe7d696 Nov 03 22:32:44.690051 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:44.690021 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b98df9fc6-mz7zf" event={"ID":"6f268a42-a565-44bc-a2f9-318cfc884c5a","Type":"ContainerStarted","Data":"b8b38767718e3dff3f34a89bc995af552c42f7c313e746a8d67f3686ffe7d696"} Nov 03 22:32:46.703152 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:46.703113 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b98df9fc6-mz7zf" event={"ID":"6f268a42-a565-44bc-a2f9-318cfc884c5a","Type":"ContainerStarted","Data":"93a551033bfe50b3407b08f4dc10674f807ab13f12ed1709ead6dc142f6f2111"} Nov 03 22:32:46.824899 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:46.824831 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-b98df9fc6-mz7zf" podStartSLOduration=2.481853913 podStartE2EDuration="3.824815871s" totalImagesPullingTime="1.342961958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:46.823756978 +0000 UTC m=+596.133316285" watchObservedRunningTime="2026-11-03 22:32:46.824815871 +0000 UTC m=+596.134375163" Nov 03 22:32:46.877106 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:46.877070 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f4f5fc5-qvkmq"] Nov 03 22:32:46.878835 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:46.878789 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f4f5fc5-qvkmq" podUID="65bb3dd1-64a9-4c80-bdf4-b5b68fee9128" containerName="authorino" containerID="cri-o://7047b5ed81a11ab5abb3da2b402d2c40efa5f852c77abbfcd6c615748316e0f0" gracePeriod=30 Nov 03 22:32:47.146755 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:47.146713 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f4f5fc5-qvkmq" Nov 03 22:32:47.282675 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:47.282592 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65bb3dd1-64a9-4c80-bdf4-b5b68fee9128-tls-cert\" (UniqueName: \"kubernetes.io/secret/65bb3dd1-64a9-4c80-bdf4-b5b68fee9128-tls-cert\") pod \"65bb3dd1-64a9-4c80-bdf4-b5b68fee9128\" (UID: \"65bb3dd1-64a9-4c80-bdf4-b5b68fee9128\") " Nov 03 22:32:47.282675 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:47.282648 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65bb3dd1-64a9-4c80-bdf4-b5b68fee9128-kube-api-access-bbk6m\" (UniqueName: \"kubernetes.io/projected/65bb3dd1-64a9-4c80-bdf4-b5b68fee9128-kube-api-access-bbk6m\") pod \"65bb3dd1-64a9-4c80-bdf4-b5b68fee9128\" (UID: \"65bb3dd1-64a9-4c80-bdf4-b5b68fee9128\") " Nov 03 22:32:47.284840 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:47.284813 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65bb3dd1-64a9-4c80-bdf4-b5b68fee9128-kube-api-access-bbk6m" pod "65bb3dd1-64a9-4c80-bdf4-b5b68fee9128" (UID: "65bb3dd1-64a9-4c80-bdf4-b5b68fee9128"). InnerVolumeSpecName "kube-api-access-bbk6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:32:47.292563 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:47.292540 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65bb3dd1-64a9-4c80-bdf4-b5b68fee9128-tls-cert" pod "65bb3dd1-64a9-4c80-bdf4-b5b68fee9128" (UID: "65bb3dd1-64a9-4c80-bdf4-b5b68fee9128"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:32:47.384224 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:47.384189 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbk6m\" (UniqueName: \"kubernetes.io/projected/65bb3dd1-64a9-4c80-bdf4-b5b68fee9128-kube-api-access-bbk6m\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:32:47.384224 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:47.384221 5196 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/65bb3dd1-64a9-4c80-bdf4-b5b68fee9128-tls-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:32:47.710607 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:47.710572 5196 generic.go:356] "Generic (PLEG): container finished" podID="65bb3dd1-64a9-4c80-bdf4-b5b68fee9128" containerID="7047b5ed81a11ab5abb3da2b402d2c40efa5f852c77abbfcd6c615748316e0f0" exitCode=0 Nov 03 22:32:47.711150 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:47.710621 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f4f5fc5-qvkmq" Nov 03 22:32:47.711150 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:47.710658 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f4f5fc5-qvkmq" event={"ID":"65bb3dd1-64a9-4c80-bdf4-b5b68fee9128","Type":"ContainerDied","Data":"7047b5ed81a11ab5abb3da2b402d2c40efa5f852c77abbfcd6c615748316e0f0"} Nov 03 22:32:47.711150 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:47.710708 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f4f5fc5-qvkmq" event={"ID":"65bb3dd1-64a9-4c80-bdf4-b5b68fee9128","Type":"ContainerDied","Data":"e92d1d866f3fc605ddd37537aa90d87c798d523b3e1604411cf0fb230629c822"} Nov 03 22:32:47.711150 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:47.710732 5196 scope.go:122] "RemoveContainer" containerID="7047b5ed81a11ab5abb3da2b402d2c40efa5f852c77abbfcd6c615748316e0f0" Nov 03 22:32:47.721378 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:47.721363 5196 scope.go:122] "RemoveContainer" containerID="7047b5ed81a11ab5abb3da2b402d2c40efa5f852c77abbfcd6c615748316e0f0" Nov 03 22:32:47.721653 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:32:47.721636 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7047b5ed81a11ab5abb3da2b402d2c40efa5f852c77abbfcd6c615748316e0f0\": container with ID starting with 7047b5ed81a11ab5abb3da2b402d2c40efa5f852c77abbfcd6c615748316e0f0 not found: ID does not exist" containerID="7047b5ed81a11ab5abb3da2b402d2c40efa5f852c77abbfcd6c615748316e0f0" Nov 03 22:32:47.721696 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:47.721663 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7047b5ed81a11ab5abb3da2b402d2c40efa5f852c77abbfcd6c615748316e0f0"} err="failed to get container status \"7047b5ed81a11ab5abb3da2b402d2c40efa5f852c77abbfcd6c615748316e0f0\": rpc error: code = NotFound desc = could not find container \"7047b5ed81a11ab5abb3da2b402d2c40efa5f852c77abbfcd6c615748316e0f0\": container with ID starting with 7047b5ed81a11ab5abb3da2b402d2c40efa5f852c77abbfcd6c615748316e0f0 not found: ID does not exist" Nov 03 22:32:47.876502 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:47.876468 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f4f5fc5-qvkmq"] Nov 03 22:32:47.880685 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:47.880657 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f4f5fc5-qvkmq"] Nov 03 22:32:48.813641 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:48.813611 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65bb3dd1-64a9-4c80-bdf4-b5b68fee9128" path="/var/lib/kubelet/pods/65bb3dd1-64a9-4c80-bdf4-b5b68fee9128/volumes" Nov 03 22:32:51.479730 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:51.479698 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:32:51.654182 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:51.654152 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:32:55.917605 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:32:55.917569 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:32:59.451157 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:32:59.451121 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:04.413730 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:04.413692 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:05.201082 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:05.201043 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:14.145874 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:14.145790 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:14.639970 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:14.639939 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:16.200944 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:16.200911 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:16.353110 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:16.353074 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:17.721399 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:17.721334 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:17.878505 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:17.878464 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:25.917329 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:33:25.917293 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:33:43.100120 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:43.100084 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:43.245328 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:43.245267 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:43.725690 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:43.725654 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:43.878977 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:43.878942 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:44.029028 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:44.028944 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:44.222430 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:44.222398 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:44.371477 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:44.371439 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:44.684218 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:44.684113 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:44.843467 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:44.843436 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:44.943286 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:44.943205 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd"] Nov 03 22:33:44.943601 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:44.943584 5196 cpu_manager.go:414] "containerMap: removing container" podUID="65bb3dd1-64a9-4c80-bdf4-b5b68fee9128" containerName="authorino" Nov 03 22:33:44.943601 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:44.943600 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65bb3dd1-64a9-4c80-bdf4-b5b68fee9128" containerName="authorino" Nov 03 22:33:44.943770 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:44.943706 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="65bb3dd1-64a9-4c80-bdf4-b5b68fee9128" containerName="authorino" Nov 03 22:33:44.946827 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:44.946812 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 22:33:44.981972 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:44.981933 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa304409-0e6c-4543-a6ae-3db66214e80d-credential-socket\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd\" (UID: \"aa304409-0e6c-4543-a6ae-3db66214e80d\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 22:33:44.982287 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:44.982256 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa304409-0e6c-4543-a6ae-3db66214e80d-workload-certs\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd\" (UID: \"aa304409-0e6c-4543-a6ae-3db66214e80d\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 22:33:44.982287 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:44.982298 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa304409-0e6c-4543-a6ae-3db66214e80d-istiod-ca-cert\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd\" (UID: \"aa304409-0e6c-4543-a6ae-3db66214e80d\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 22:33:44.982456 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:44.982353 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa304409-0e6c-4543-a6ae-3db66214e80d-istio-data\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd\" (UID: \"aa304409-0e6c-4543-a6ae-3db66214e80d\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 22:33:44.982505 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:44.982464 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mg5k\" (UniqueName: \"kubernetes.io/projected/aa304409-0e6c-4543-a6ae-3db66214e80d-kube-api-access-4mg5k\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd\" (UID: \"aa304409-0e6c-4543-a6ae-3db66214e80d\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 22:33:44.982542 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:44.982525 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa304409-0e6c-4543-a6ae-3db66214e80d-istio-envoy\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd\" (UID: \"aa304409-0e6c-4543-a6ae-3db66214e80d\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 22:33:44.982586 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:44.982547 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa304409-0e6c-4543-a6ae-3db66214e80d-istio-podinfo\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd\" (UID: \"aa304409-0e6c-4543-a6ae-3db66214e80d\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 22:33:44.982586 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:44.982571 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa304409-0e6c-4543-a6ae-3db66214e80d-istio-token\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd\" (UID: \"aa304409-0e6c-4543-a6ae-3db66214e80d\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 22:33:44.982666 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:44.982608 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa304409-0e6c-4543-a6ae-3db66214e80d-workload-socket\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd\" (UID: \"aa304409-0e6c-4543-a6ae-3db66214e80d\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 22:33:44.986218 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:44.986191 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:45.000226 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:45.000204 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-164c36-openshift-default-dockercfg-tlv7g\"" Nov 03 22:33:45.053521 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:45.053487 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd"] Nov 03 22:33:45.083336 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:45.083306 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa304409-0e6c-4543-a6ae-3db66214e80d-credential-socket\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd\" (UID: \"aa304409-0e6c-4543-a6ae-3db66214e80d\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 22:33:45.083490 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:45.083340 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa304409-0e6c-4543-a6ae-3db66214e80d-workload-certs\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd\" (UID: \"aa304409-0e6c-4543-a6ae-3db66214e80d\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 22:33:45.083490 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:45.083359 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa304409-0e6c-4543-a6ae-3db66214e80d-istiod-ca-cert\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd\" (UID: \"aa304409-0e6c-4543-a6ae-3db66214e80d\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 22:33:45.083490 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:45.083382 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa304409-0e6c-4543-a6ae-3db66214e80d-istio-data\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd\" (UID: \"aa304409-0e6c-4543-a6ae-3db66214e80d\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 22:33:45.083490 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:45.083443 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mg5k\" (UniqueName: \"kubernetes.io/projected/aa304409-0e6c-4543-a6ae-3db66214e80d-kube-api-access-4mg5k\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd\" (UID: \"aa304409-0e6c-4543-a6ae-3db66214e80d\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 22:33:45.083490 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:45.083482 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa304409-0e6c-4543-a6ae-3db66214e80d-istio-envoy\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd\" (UID: \"aa304409-0e6c-4543-a6ae-3db66214e80d\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 22:33:45.083810 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:45.083606 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa304409-0e6c-4543-a6ae-3db66214e80d-istio-podinfo\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd\" (UID: \"aa304409-0e6c-4543-a6ae-3db66214e80d\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 22:33:45.083810 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:45.083637 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa304409-0e6c-4543-a6ae-3db66214e80d-istio-token\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd\" (UID: \"aa304409-0e6c-4543-a6ae-3db66214e80d\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 22:33:45.083810 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:45.083672 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa304409-0e6c-4543-a6ae-3db66214e80d-workload-socket\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd\" (UID: \"aa304409-0e6c-4543-a6ae-3db66214e80d\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 22:33:45.083810 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:45.083798 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa304409-0e6c-4543-a6ae-3db66214e80d-credential-socket\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd\" (UID: \"aa304409-0e6c-4543-a6ae-3db66214e80d\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 22:33:45.084052 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:45.083879 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa304409-0e6c-4543-a6ae-3db66214e80d-workload-certs\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd\" (UID: \"aa304409-0e6c-4543-a6ae-3db66214e80d\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 22:33:45.084052 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:45.083977 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa304409-0e6c-4543-a6ae-3db66214e80d-istio-data\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd\" (UID: \"aa304409-0e6c-4543-a6ae-3db66214e80d\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 22:33:45.084130 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:45.084065 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa304409-0e6c-4543-a6ae-3db66214e80d-workload-socket\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd\" (UID: \"aa304409-0e6c-4543-a6ae-3db66214e80d\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 22:33:45.084207 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:45.084182 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa304409-0e6c-4543-a6ae-3db66214e80d-istiod-ca-cert\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd\" (UID: \"aa304409-0e6c-4543-a6ae-3db66214e80d\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 22:33:45.086261 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:45.086236 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa304409-0e6c-4543-a6ae-3db66214e80d-istio-envoy\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd\" (UID: \"aa304409-0e6c-4543-a6ae-3db66214e80d\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 22:33:45.086356 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:45.086328 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa304409-0e6c-4543-a6ae-3db66214e80d-istio-podinfo\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd\" (UID: \"aa304409-0e6c-4543-a6ae-3db66214e80d\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 22:33:45.127584 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:45.127553 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:45.148331 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:45.148304 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa304409-0e6c-4543-a6ae-3db66214e80d-istio-token\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd\" (UID: \"aa304409-0e6c-4543-a6ae-3db66214e80d\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 22:33:45.148432 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:45.148374 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mg5k\" (UniqueName: \"kubernetes.io/projected/aa304409-0e6c-4543-a6ae-3db66214e80d-kube-api-access-4mg5k\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd\" (UID: \"aa304409-0e6c-4543-a6ae-3db66214e80d\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 22:33:45.259579 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:45.259501 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 22:33:45.366353 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:45.366324 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:45.522245 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:45.522169 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:45.686564 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:45.686524 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:45.704190 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:45.704166 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd"] Nov 03 22:33:45.705736 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:33:45.705696 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa304409_0e6c_4543_a6ae_3db66214e80d.slice/crio-b2c29d11276b95c797b51b10c53742eb4fa69c1dbb4a18c4bc7b57dc58729e81 WatchSource:0}: Error finding container b2c29d11276b95c797b51b10c53742eb4fa69c1dbb4a18c4bc7b57dc58729e81: Status 404 returned error can't find the container with id b2c29d11276b95c797b51b10c53742eb4fa69c1dbb4a18c4bc7b57dc58729e81 Nov 03 22:33:45.708016 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:45.707982 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:33:45.708090 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:45.708053 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:33:45.708090 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:45.708082 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:33:45.901260 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:45.901227 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:46.008773 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:46.006872 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" event={"ID":"aa304409-0e6c-4543-a6ae-3db66214e80d","Type":"ContainerStarted","Data":"78d4402e5d9a4616c7d5fbb4fd1456c482bf0a981e8c4f2d101b043b89a7f391"} Nov 03 22:33:46.008773 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:46.006951 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" event={"ID":"aa304409-0e6c-4543-a6ae-3db66214e80d","Type":"ContainerStarted","Data":"b2c29d11276b95c797b51b10c53742eb4fa69c1dbb4a18c4bc7b57dc58729e81"} Nov 03 22:33:46.131552 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:46.131497 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" podStartSLOduration=2.131482897 podStartE2EDuration="2.131482897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:46.129768716 +0000 UTC m=+655.439328035" watchObservedRunningTime="2026-11-03 22:33:46.131482897 +0000 UTC m=+655.441042189" Nov 03 22:33:46.260147 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:46.260118 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 22:33:46.264882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:46.264830 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 22:33:47.018178 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:47.018140 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 22:33:47.019296 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:47.019274 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 22:33:48.403589 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:48.403547 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:48.570479 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:48.570444 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:48.728104 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:48.728020 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:49.185810 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:49.185769 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:49.357597 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:49.357560 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:49.501561 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:49.501472 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:49.973674 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:49.973640 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:50.208513 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:50.208481 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:50.353412 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:50.353378 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:50.655335 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:50.655243 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:50.826278 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:50.826240 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:50.965372 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:50.965280 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:52.295708 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:52.295671 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:52.468976 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:52.468929 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:52.611596 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:52.611559 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:52.845202 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:52.845172 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:53.014740 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:53.014637 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:54.138543 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:54.138510 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:54.367294 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:54.367253 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:54.513750 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:54.513667 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:54.783494 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:54.783406 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m"] Nov 03 22:33:54.787081 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:54.787056 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m" Nov 03 22:33:54.841807 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:54.841780 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-32449d-serving-cert\"" Nov 03 22:33:54.841965 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:54.841875 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 22:33:54.869952 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:54.869928 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltfm6\" (UniqueName: \"kubernetes.io/projected/89748893-72b3-46f6-9419-c8fd5fe4cd73-kube-api-access-ltfm6\") pod \"maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m\" (UID: \"89748893-72b3-46f6-9419-c8fd5fe4cd73\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m" Nov 03 22:33:54.870087 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:54.869964 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89748893-72b3-46f6-9419-c8fd5fe4cd73-metrics-certs\") pod \"maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m\" (UID: \"89748893-72b3-46f6-9419-c8fd5fe4cd73\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m" Nov 03 22:33:54.870087 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:54.869988 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89748893-72b3-46f6-9419-c8fd5fe4cd73-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m\" (UID: \"89748893-72b3-46f6-9419-c8fd5fe4cd73\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m" Nov 03 22:33:54.870206 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:54.870154 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89748893-72b3-46f6-9419-c8fd5fe4cd73-maas-api-tls\") pod \"maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m\" (UID: \"89748893-72b3-46f6-9419-c8fd5fe4cd73\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m" Nov 03 22:33:54.892391 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:54.892370 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:33:54.892695 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:54.892680 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:33:54.892915 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:54.892897 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sw68g\"" Nov 03 22:33:54.894463 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:54.894448 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m"] Nov 03 22:33:54.958423 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:54.958398 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:54.971547 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:54.971519 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltfm6\" (UniqueName: \"kubernetes.io/projected/89748893-72b3-46f6-9419-c8fd5fe4cd73-kube-api-access-ltfm6\") pod \"maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m\" (UID: \"89748893-72b3-46f6-9419-c8fd5fe4cd73\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m" Nov 03 22:33:54.971687 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:54.971560 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89748893-72b3-46f6-9419-c8fd5fe4cd73-metrics-certs\") pod \"maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m\" (UID: \"89748893-72b3-46f6-9419-c8fd5fe4cd73\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m" Nov 03 22:33:54.971750 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:54.971682 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89748893-72b3-46f6-9419-c8fd5fe4cd73-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m\" (UID: \"89748893-72b3-46f6-9419-c8fd5fe4cd73\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m" Nov 03 22:33:54.971750 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:54.971725 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89748893-72b3-46f6-9419-c8fd5fe4cd73-maas-api-tls\") pod \"maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m\" (UID: \"89748893-72b3-46f6-9419-c8fd5fe4cd73\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m" Nov 03 22:33:54.971894 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:33:54.971877 5196 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-32449d-serving-cert: secret "maas-api-e2e-worker-w4-32449d-serving-cert" not found Nov 03 22:33:54.971961 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:33:54.971941 5196 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89748893-72b3-46f6-9419-c8fd5fe4cd73-maas-api-tls podName:89748893-72b3-46f6-9419-c8fd5fe4cd73 nodeName:}" failed. No retries permitted until 2026-11-03 22:33:55.47192498 +0000 UTC m=+664.781484252 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/89748893-72b3-46f6-9419-c8fd5fe4cd73-maas-api-tls") pod "maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m" (UID: "89748893-72b3-46f6-9419-c8fd5fe4cd73") : secret "maas-api-e2e-worker-w4-32449d-serving-cert" not found Nov 03 22:33:54.972283 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:54.972267 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89748893-72b3-46f6-9419-c8fd5fe4cd73-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m\" (UID: \"89748893-72b3-46f6-9419-c8fd5fe4cd73\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m" Nov 03 22:33:54.974129 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:54.974109 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89748893-72b3-46f6-9419-c8fd5fe4cd73-metrics-certs\") pod \"maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m\" (UID: \"89748893-72b3-46f6-9419-c8fd5fe4cd73\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m" Nov 03 22:33:55.034076 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:55.034020 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltfm6\" (UniqueName: \"kubernetes.io/projected/89748893-72b3-46f6-9419-c8fd5fe4cd73-kube-api-access-ltfm6\") pod \"maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m\" (UID: \"89748893-72b3-46f6-9419-c8fd5fe4cd73\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m" Nov 03 22:33:55.143553 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:55.143506 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:55.287185 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:55.287082 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:55.476640 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:55.476603 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89748893-72b3-46f6-9419-c8fd5fe4cd73-maas-api-tls\") pod \"maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m\" (UID: \"89748893-72b3-46f6-9419-c8fd5fe4cd73\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m" Nov 03 22:33:55.479205 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:55.479178 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89748893-72b3-46f6-9419-c8fd5fe4cd73-maas-api-tls\") pod \"maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m\" (UID: \"89748893-72b3-46f6-9419-c8fd5fe4cd73\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m" Nov 03 22:33:55.699756 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:55.699725 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m" Nov 03 22:33:55.918494 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:33:55.918459 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:33:56.080806 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:56.080778 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m"] Nov 03 22:33:56.083051 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:33:56.083028 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89748893_72b3_46f6_9419_c8fd5fe4cd73.slice/crio-61ab4cf7e669129c8ac6ad5a43b7a7fb27df7bb168026be7d796851ea83ab7f2 WatchSource:0}: Error finding container 61ab4cf7e669129c8ac6ad5a43b7a7fb27df7bb168026be7d796851ea83ab7f2: Status 404 returned error can't find the container with id 61ab4cf7e669129c8ac6ad5a43b7a7fb27df7bb168026be7d796851ea83ab7f2 Nov 03 22:33:56.084329 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:56.084310 5196 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:33:57.067335 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:57.067308 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m" event={"ID":"89748893-72b3-46f6-9419-c8fd5fe4cd73","Type":"ContainerStarted","Data":"788c68cedef14248acd1037df4c9e18416c1e99832cf67bf494886c4d8bd8e0a"} Nov 03 22:33:57.067665 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:57.067359 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m" event={"ID":"89748893-72b3-46f6-9419-c8fd5fe4cd73","Type":"ContainerStarted","Data":"61ab4cf7e669129c8ac6ad5a43b7a7fb27df7bb168026be7d796851ea83ab7f2"} Nov 03 22:33:57.067665 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:57.067413 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m" Nov 03 22:33:57.195845 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:57.195796 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m" podStartSLOduration=2.320884047 podStartE2EDuration="3.195780306s" totalImagesPullingTime="874.896259ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:57.19440548 +0000 UTC m=+666.503964784" watchObservedRunningTime="2026-11-03 22:33:57.195780306 +0000 UTC m=+666.505339598" Nov 03 22:33:57.772750 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:57.772723 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:57.943602 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:57.943567 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:59.762411 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:59.762378 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:33:59.913949 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:59.913913 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl"] Nov 03 22:33:59.917520 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:59.917498 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl" Nov 03 22:33:59.970742 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:33:59.970714 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-7afb1c-serving-cert\"" Nov 03 22:34:00.017227 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:00.017143 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f7ce2e3-e5ee-45b8-99e6-985011216f8b-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl\" (UID: \"8f7ce2e3-e5ee-45b8-99e6-985011216f8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl" Nov 03 22:34:00.017227 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:00.017180 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbd46\" (UniqueName: \"kubernetes.io/projected/8f7ce2e3-e5ee-45b8-99e6-985011216f8b-kube-api-access-qbd46\") pod \"maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl\" (UID: \"8f7ce2e3-e5ee-45b8-99e6-985011216f8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl" Nov 03 22:34:00.017387 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:00.017300 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f7ce2e3-e5ee-45b8-99e6-985011216f8b-metrics-certs\") pod \"maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl\" (UID: \"8f7ce2e3-e5ee-45b8-99e6-985011216f8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl" Nov 03 22:34:00.017387 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:00.017330 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f7ce2e3-e5ee-45b8-99e6-985011216f8b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl\" (UID: \"8f7ce2e3-e5ee-45b8-99e6-985011216f8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl" Nov 03 22:34:00.023805 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:00.023781 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl"] Nov 03 22:34:00.101623 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:00.101589 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:00.118772 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:00.118742 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f7ce2e3-e5ee-45b8-99e6-985011216f8b-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl\" (UID: \"8f7ce2e3-e5ee-45b8-99e6-985011216f8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl" Nov 03 22:34:00.118772 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:00.118774 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbd46\" (UniqueName: \"kubernetes.io/projected/8f7ce2e3-e5ee-45b8-99e6-985011216f8b-kube-api-access-qbd46\") pod \"maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl\" (UID: \"8f7ce2e3-e5ee-45b8-99e6-985011216f8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl" Nov 03 22:34:00.118959 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:00.118840 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f7ce2e3-e5ee-45b8-99e6-985011216f8b-metrics-certs\") pod \"maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl\" (UID: \"8f7ce2e3-e5ee-45b8-99e6-985011216f8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl" Nov 03 22:34:00.118959 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:00.118902 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f7ce2e3-e5ee-45b8-99e6-985011216f8b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl\" (UID: \"8f7ce2e3-e5ee-45b8-99e6-985011216f8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl" Nov 03 22:34:00.119093 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:34:00.119072 5196 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-7afb1c-serving-cert: secret "maas-api-e2e-worker-w1-7afb1c-serving-cert" not found Nov 03 22:34:00.119178 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:34:00.119166 5196 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f7ce2e3-e5ee-45b8-99e6-985011216f8b-maas-api-tls podName:8f7ce2e3-e5ee-45b8-99e6-985011216f8b nodeName:}" failed. No retries permitted until 2026-11-03 22:34:00.619151255 +0000 UTC m=+669.928710527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8f7ce2e3-e5ee-45b8-99e6-985011216f8b-maas-api-tls") pod "maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl" (UID: "8f7ce2e3-e5ee-45b8-99e6-985011216f8b") : secret "maas-api-e2e-worker-w1-7afb1c-serving-cert" not found Nov 03 22:34:00.119409 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:00.119389 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f7ce2e3-e5ee-45b8-99e6-985011216f8b-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl\" (UID: \"8f7ce2e3-e5ee-45b8-99e6-985011216f8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl" Nov 03 22:34:00.121433 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:00.121414 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f7ce2e3-e5ee-45b8-99e6-985011216f8b-metrics-certs\") pod \"maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl\" (UID: \"8f7ce2e3-e5ee-45b8-99e6-985011216f8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl" Nov 03 22:34:00.182292 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:00.182248 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbd46\" (UniqueName: \"kubernetes.io/projected/8f7ce2e3-e5ee-45b8-99e6-985011216f8b-kube-api-access-qbd46\") pod \"maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl\" (UID: \"8f7ce2e3-e5ee-45b8-99e6-985011216f8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl" Nov 03 22:34:00.247042 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:00.247003 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:00.390314 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:00.390284 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:00.623229 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:00.623194 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f7ce2e3-e5ee-45b8-99e6-985011216f8b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl\" (UID: \"8f7ce2e3-e5ee-45b8-99e6-985011216f8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl" Nov 03 22:34:00.625791 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:00.625760 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f7ce2e3-e5ee-45b8-99e6-985011216f8b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl\" (UID: \"8f7ce2e3-e5ee-45b8-99e6-985011216f8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl" Nov 03 22:34:00.628572 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:00.628523 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:00.829998 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:00.829921 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl" Nov 03 22:34:01.205169 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:01.205149 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl"] Nov 03 22:34:01.209122 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:34:01.209083 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f7ce2e3_e5ee_45b8_99e6_985011216f8b.slice/crio-b4c50aef55a605c1fda99c19919b914bbd1c38dcbfc2aff1c78e393b834af0d5 WatchSource:0}: Error finding container b4c50aef55a605c1fda99c19919b914bbd1c38dcbfc2aff1c78e393b834af0d5: Status 404 returned error can't find the container with id b4c50aef55a605c1fda99c19919b914bbd1c38dcbfc2aff1c78e393b834af0d5 Nov 03 22:34:01.551929 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:01.551818 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:01.722343 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:01.722304 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:01.866213 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:01.866178 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:02.095379 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:02.095351 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl" event={"ID":"8f7ce2e3-e5ee-45b8-99e6-985011216f8b","Type":"ContainerStarted","Data":"b4c50aef55a605c1fda99c19919b914bbd1c38dcbfc2aff1c78e393b834af0d5"} Nov 03 22:34:03.078943 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:03.078911 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m" Nov 03 22:34:03.105449 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:03.105406 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl" event={"ID":"8f7ce2e3-e5ee-45b8-99e6-985011216f8b","Type":"ContainerStarted","Data":"53cec113a2bf82ab6da87eee3e35e57187b114db27fd18f7c1ab7b3db51b7a99"} Nov 03 22:34:03.106154 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:03.106106 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl" Nov 03 22:34:03.324892 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:03.324811 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl" podStartSLOduration=3.493839725 podStartE2EDuration="4.324791968s" totalImagesPullingTime="830.952243ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:34:03.322140249 +0000 UTC m=+672.631699543" watchObservedRunningTime="2026-11-03 22:34:03.324791968 +0000 UTC m=+672.634351262" Nov 03 22:34:04.756424 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:04.756390 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:05.202969 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:05.202918 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:05.383108 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:05.383077 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:05.549297 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:05.549209 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:05.735392 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:05.735357 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:05.903705 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:05.903672 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:07.758222 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:07.758194 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:07.940527 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:07.940491 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:10.121805 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:10.121773 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl" Nov 03 22:34:10.445941 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:10.445850 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm"] Nov 03 22:34:10.449429 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:10.449411 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm" Nov 03 22:34:10.502656 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:10.502634 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-164c36-serving-cert\"" Nov 03 22:34:10.510124 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:10.510100 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm"] Nov 03 22:34:10.615313 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:10.615283 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3692067f-4a4d-4bd0-bc74-8957aaf8531c-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm\" (UID: \"3692067f-4a4d-4bd0-bc74-8957aaf8531c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm" Nov 03 22:34:10.615313 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:10.615315 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3692067f-4a4d-4bd0-bc74-8957aaf8531c-metrics-certs\") pod \"maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm\" (UID: \"3692067f-4a4d-4bd0-bc74-8957aaf8531c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm" Nov 03 22:34:10.615495 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:10.615350 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxzrk\" (UniqueName: \"kubernetes.io/projected/3692067f-4a4d-4bd0-bc74-8957aaf8531c-kube-api-access-xxzrk\") pod \"maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm\" (UID: \"3692067f-4a4d-4bd0-bc74-8957aaf8531c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm" Nov 03 22:34:10.615495 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:10.615462 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3692067f-4a4d-4bd0-bc74-8957aaf8531c-maas-api-tls\") pod \"maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm\" (UID: \"3692067f-4a4d-4bd0-bc74-8957aaf8531c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm" Nov 03 22:34:10.632474 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:10.632450 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:10.716260 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:10.716179 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3692067f-4a4d-4bd0-bc74-8957aaf8531c-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm\" (UID: \"3692067f-4a4d-4bd0-bc74-8957aaf8531c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm" Nov 03 22:34:10.716260 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:10.716223 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3692067f-4a4d-4bd0-bc74-8957aaf8531c-metrics-certs\") pod \"maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm\" (UID: \"3692067f-4a4d-4bd0-bc74-8957aaf8531c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm" Nov 03 22:34:10.716466 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:10.716278 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxzrk\" (UniqueName: \"kubernetes.io/projected/3692067f-4a4d-4bd0-bc74-8957aaf8531c-kube-api-access-xxzrk\") pod \"maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm\" (UID: \"3692067f-4a4d-4bd0-bc74-8957aaf8531c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm" Nov 03 22:34:10.716466 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:10.716312 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3692067f-4a4d-4bd0-bc74-8957aaf8531c-maas-api-tls\") pod \"maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm\" (UID: \"3692067f-4a4d-4bd0-bc74-8957aaf8531c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm" Nov 03 22:34:10.716549 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:34:10.716496 5196 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-164c36-serving-cert: secret "maas-api-e2e-worker-w2-164c36-serving-cert" not found Nov 03 22:34:10.716607 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:34:10.716594 5196 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3692067f-4a4d-4bd0-bc74-8957aaf8531c-maas-api-tls podName:3692067f-4a4d-4bd0-bc74-8957aaf8531c nodeName:}" failed. No retries permitted until 2026-11-03 22:34:11.216570154 +0000 UTC m=+680.526129443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3692067f-4a4d-4bd0-bc74-8957aaf8531c-maas-api-tls") pod "maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm" (UID: "3692067f-4a4d-4bd0-bc74-8957aaf8531c") : secret "maas-api-e2e-worker-w2-164c36-serving-cert" not found Nov 03 22:34:10.716915 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:10.716894 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3692067f-4a4d-4bd0-bc74-8957aaf8531c-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm\" (UID: \"3692067f-4a4d-4bd0-bc74-8957aaf8531c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm" Nov 03 22:34:10.719050 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:10.719028 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3692067f-4a4d-4bd0-bc74-8957aaf8531c-metrics-certs\") pod \"maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm\" (UID: \"3692067f-4a4d-4bd0-bc74-8957aaf8531c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm" Nov 03 22:34:10.778768 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:10.778742 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxzrk\" (UniqueName: \"kubernetes.io/projected/3692067f-4a4d-4bd0-bc74-8957aaf8531c-kube-api-access-xxzrk\") pod \"maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm\" (UID: \"3692067f-4a4d-4bd0-bc74-8957aaf8531c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm" Nov 03 22:34:10.827670 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:10.827645 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:10.969618 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:10.969528 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:11.221242 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:11.221157 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3692067f-4a4d-4bd0-bc74-8957aaf8531c-maas-api-tls\") pod \"maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm\" (UID: \"3692067f-4a4d-4bd0-bc74-8957aaf8531c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm" Nov 03 22:34:11.223657 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:11.223637 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3692067f-4a4d-4bd0-bc74-8957aaf8531c-maas-api-tls\") pod \"maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm\" (UID: \"3692067f-4a4d-4bd0-bc74-8957aaf8531c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm" Nov 03 22:34:11.362378 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:11.362348 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm" Nov 03 22:34:11.736513 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:11.736490 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm"] Nov 03 22:34:11.738661 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:34:11.738633 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3692067f_4a4d_4bd0_bc74_8957aaf8531c.slice/crio-a293ed3cc42d1f1b8acb7f872f9c0a1a9058bbe2e7653ab38fc49bd02abc9b59 WatchSource:0}: Error finding container a293ed3cc42d1f1b8acb7f872f9c0a1a9058bbe2e7653ab38fc49bd02abc9b59: Status 404 returned error can't find the container with id a293ed3cc42d1f1b8acb7f872f9c0a1a9058bbe2e7653ab38fc49bd02abc9b59 Nov 03 22:34:12.158785 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:12.158750 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm" event={"ID":"3692067f-4a4d-4bd0-bc74-8957aaf8531c","Type":"ContainerStarted","Data":"a293ed3cc42d1f1b8acb7f872f9c0a1a9058bbe2e7653ab38fc49bd02abc9b59"} Nov 03 22:34:12.555021 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:12.554934 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:12.738111 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:12.738081 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:13.165624 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:13.165591 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm" event={"ID":"3692067f-4a4d-4bd0-bc74-8957aaf8531c","Type":"ContainerStarted","Data":"782dd45c8c888507deaaabb5b572ef215ea0d5c1f7e75ba56ceafe1226840954"} Nov 03 22:34:13.165820 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:13.165656 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm" Nov 03 22:34:13.286127 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:13.286072 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm" podStartSLOduration=2.380393367 podStartE2EDuration="3.286059384s" totalImagesPullingTime="905.666017ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:34:13.284169984 +0000 UTC m=+682.593729280" watchObservedRunningTime="2026-11-03 22:34:13.286059384 +0000 UTC m=+682.595618678" Nov 03 22:34:15.150622 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:15.150586 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:15.337547 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:15.337507 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:15.971611 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:15.971580 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:19.176146 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:19.176121 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm" Nov 03 22:34:20.991190 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:20.991151 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:21.190366 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:21.190330 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:21.337275 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:21.337184 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:25.937007 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:34:25.936959 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:34:26.157670 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:26.157608 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65"] Nov 03 22:34:26.161448 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:26.161431 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65" Nov 03 22:34:26.217466 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:26.217217 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-1d31f8-serving-cert\"" Nov 03 22:34:26.262241 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:26.262212 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de6a6c43-96f4-435e-9552-292d2112ed70-maas-api-tls\") pod \"maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65\" (UID: \"de6a6c43-96f4-435e-9552-292d2112ed70\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65" Nov 03 22:34:26.262241 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:26.262242 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de6a6c43-96f4-435e-9552-292d2112ed70-metrics-certs\") pod \"maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65\" (UID: \"de6a6c43-96f4-435e-9552-292d2112ed70\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65" Nov 03 22:34:26.262447 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:26.262268 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de6a6c43-96f4-435e-9552-292d2112ed70-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65\" (UID: \"de6a6c43-96f4-435e-9552-292d2112ed70\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65" Nov 03 22:34:26.262447 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:26.262342 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb9pd\" (UniqueName: \"kubernetes.io/projected/de6a6c43-96f4-435e-9552-292d2112ed70-kube-api-access-qb9pd\") pod \"maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65\" (UID: \"de6a6c43-96f4-435e-9552-292d2112ed70\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65" Nov 03 22:34:26.269989 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:26.269966 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65"] Nov 03 22:34:26.363789 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:26.363756 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de6a6c43-96f4-435e-9552-292d2112ed70-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65\" (UID: \"de6a6c43-96f4-435e-9552-292d2112ed70\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65" Nov 03 22:34:26.363964 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:26.363796 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb9pd\" (UniqueName: \"kubernetes.io/projected/de6a6c43-96f4-435e-9552-292d2112ed70-kube-api-access-qb9pd\") pod \"maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65\" (UID: \"de6a6c43-96f4-435e-9552-292d2112ed70\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65" Nov 03 22:34:26.363964 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:26.363946 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de6a6c43-96f4-435e-9552-292d2112ed70-maas-api-tls\") pod \"maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65\" (UID: \"de6a6c43-96f4-435e-9552-292d2112ed70\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65" Nov 03 22:34:26.364052 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:26.363971 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de6a6c43-96f4-435e-9552-292d2112ed70-metrics-certs\") pod \"maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65\" (UID: \"de6a6c43-96f4-435e-9552-292d2112ed70\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65" Nov 03 22:34:26.364104 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:34:26.364087 5196 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-1d31f8-serving-cert: secret "maas-api-e2e-worker-w3-1d31f8-serving-cert" not found Nov 03 22:34:26.364182 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:34:26.364172 5196 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de6a6c43-96f4-435e-9552-292d2112ed70-maas-api-tls podName:de6a6c43-96f4-435e-9552-292d2112ed70 nodeName:}" failed. No retries permitted until 2026-11-03 22:34:26.864151146 +0000 UTC m=+696.173710417 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/de6a6c43-96f4-435e-9552-292d2112ed70-maas-api-tls") pod "maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65" (UID: "de6a6c43-96f4-435e-9552-292d2112ed70") : secret "maas-api-e2e-worker-w3-1d31f8-serving-cert" not found Nov 03 22:34:26.364598 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:26.364579 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de6a6c43-96f4-435e-9552-292d2112ed70-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65\" (UID: \"de6a6c43-96f4-435e-9552-292d2112ed70\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65" Nov 03 22:34:26.366384 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:26.366366 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de6a6c43-96f4-435e-9552-292d2112ed70-metrics-certs\") pod \"maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65\" (UID: \"de6a6c43-96f4-435e-9552-292d2112ed70\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65" Nov 03 22:34:26.386121 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:26.386099 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:26.425944 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:26.425917 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb9pd\" (UniqueName: \"kubernetes.io/projected/de6a6c43-96f4-435e-9552-292d2112ed70-kube-api-access-qb9pd\") pod \"maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65\" (UID: \"de6a6c43-96f4-435e-9552-292d2112ed70\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65" Nov 03 22:34:26.548794 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:26.548707 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:26.701392 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:26.701354 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:26.869056 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:26.869027 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de6a6c43-96f4-435e-9552-292d2112ed70-maas-api-tls\") pod \"maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65\" (UID: \"de6a6c43-96f4-435e-9552-292d2112ed70\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65" Nov 03 22:34:26.871578 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:26.871532 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de6a6c43-96f4-435e-9552-292d2112ed70-maas-api-tls\") pod \"maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65\" (UID: \"de6a6c43-96f4-435e-9552-292d2112ed70\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65" Nov 03 22:34:27.073957 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:27.073920 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65" Nov 03 22:34:27.477721 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:27.477698 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65"] Nov 03 22:34:27.479830 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:34:27.479803 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde6a6c43_96f4_435e_9552_292d2112ed70.slice/crio-8160dd2856e1aac282c17df520454ad274a7980678d1b9ae1a041ef04f0bb0e0 WatchSource:0}: Error finding container 8160dd2856e1aac282c17df520454ad274a7980678d1b9ae1a041ef04f0bb0e0: Status 404 returned error can't find the container with id 8160dd2856e1aac282c17df520454ad274a7980678d1b9ae1a041ef04f0bb0e0 Nov 03 22:34:28.260890 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:28.260828 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65" event={"ID":"de6a6c43-96f4-435e-9552-292d2112ed70","Type":"ContainerStarted","Data":"8160dd2856e1aac282c17df520454ad274a7980678d1b9ae1a041ef04f0bb0e0"} Nov 03 22:34:29.275881 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:29.274962 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65" event={"ID":"de6a6c43-96f4-435e-9552-292d2112ed70","Type":"ContainerStarted","Data":"87d840da773fc1763e9e586a9d8de75c2a16f73f7269335e0aeb2bebfa3ab531"} Nov 03 22:34:29.275881 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:29.275240 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65" Nov 03 22:34:29.436492 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:29.436415 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65" podStartSLOduration=2.542860777 podStartE2EDuration="3.436398374s" totalImagesPullingTime="893.537597ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:34:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:34:29.434954717 +0000 UTC m=+698.744514009" watchObservedRunningTime="2026-11-03 22:34:29.436398374 +0000 UTC m=+698.745957667" Nov 03 22:34:30.435740 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:30.435707 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh"] Nov 03 22:34:30.439352 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:30.439336 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" Nov 03 22:34:30.508064 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:30.508036 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-workload-certs\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh\" (UID: \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" Nov 03 22:34:30.508251 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:30.508075 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-credential-socket\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh\" (UID: \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" Nov 03 22:34:30.508251 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:30.508098 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-istiod-ca-cert\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh\" (UID: \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" Nov 03 22:34:30.508251 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:30.508190 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-istio-data\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh\" (UID: \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" Nov 03 22:34:30.508444 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:30.508318 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-workload-socket\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh\" (UID: \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" Nov 03 22:34:30.508444 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:30.508362 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-istio-podinfo\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh\" (UID: \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" Nov 03 22:34:30.508561 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:30.508446 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-istio-envoy\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh\" (UID: \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" Nov 03 22:34:30.508561 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:30.508502 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-istio-token\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh\" (UID: \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" Nov 03 22:34:30.508561 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:30.508528 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scr6d\" (UniqueName: \"kubernetes.io/projected/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-kube-api-access-scr6d\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh\" (UID: \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" Nov 03 22:34:30.528414 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:30.528393 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh"] Nov 03 22:34:30.609744 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:30.609714 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scr6d\" (UniqueName: \"kubernetes.io/projected/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-kube-api-access-scr6d\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh\" (UID: \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" Nov 03 22:34:30.609937 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:30.609769 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-workload-certs\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh\" (UID: \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" Nov 03 22:34:30.609937 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:30.609800 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-credential-socket\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh\" (UID: \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" Nov 03 22:34:30.610051 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:30.609983 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-istiod-ca-cert\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh\" (UID: \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" Nov 03 22:34:30.610051 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:30.610005 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-istio-data\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh\" (UID: \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" Nov 03 22:34:30.610051 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:30.610023 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-workload-socket\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh\" (UID: \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" Nov 03 22:34:30.610255 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:30.610232 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-istio-podinfo\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh\" (UID: \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" Nov 03 22:34:30.610322 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:30.610283 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-istio-envoy\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh\" (UID: \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" Nov 03 22:34:30.610376 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:30.610334 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-workload-certs\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh\" (UID: \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" Nov 03 22:34:30.610376 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:30.610365 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-istio-token\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh\" (UID: \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" Nov 03 22:34:30.610376 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:30.610344 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-credential-socket\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh\" (UID: \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" Nov 03 22:34:30.610530 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:30.610366 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-workload-socket\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh\" (UID: \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" Nov 03 22:34:30.610778 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:30.610753 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-istiod-ca-cert\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh\" (UID: \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" Nov 03 22:34:30.610939 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:30.610852 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-istio-data\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh\" (UID: \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" Nov 03 22:34:30.612747 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:30.612722 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-istio-envoy\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh\" (UID: \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" Nov 03 22:34:30.612879 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:30.612837 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-istio-podinfo\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh\" (UID: \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" Nov 03 22:34:30.671725 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:30.671691 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-istio-token\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh\" (UID: \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" Nov 03 22:34:30.671922 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:30.671878 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scr6d\" (UniqueName: \"kubernetes.io/projected/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-kube-api-access-scr6d\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh\" (UID: \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" Nov 03 22:34:30.752605 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:30.752523 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" Nov 03 22:34:31.192767 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:31.192741 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh"] Nov 03 22:34:31.194312 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:34:31.194281 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c75a6b1_9f57_4b0b_bafb_06e1dd7957bb.slice/crio-e5db1f0dab1ace1d151471a628a5e0bc9f381a831a5a20473119adf7e5d143b2 WatchSource:0}: Error finding container e5db1f0dab1ace1d151471a628a5e0bc9f381a831a5a20473119adf7e5d143b2: Status 404 returned error can't find the container with id e5db1f0dab1ace1d151471a628a5e0bc9f381a831a5a20473119adf7e5d143b2 Nov 03 22:34:31.197215 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:31.197179 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:34:31.197290 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:31.197248 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:34:31.197290 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:31.197277 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:34:31.289095 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:31.289016 5196 generic.go:356] "Generic (PLEG): container finished" podID="aa304409-0e6c-4543-a6ae-3db66214e80d" containerID="78d4402e5d9a4616c7d5fbb4fd1456c482bf0a981e8c4f2d101b043b89a7f391" exitCode=0 Nov 03 22:34:31.289256 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:31.289086 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" event={"ID":"aa304409-0e6c-4543-a6ae-3db66214e80d","Type":"ContainerDied","Data":"78d4402e5d9a4616c7d5fbb4fd1456c482bf0a981e8c4f2d101b043b89a7f391"} Nov 03 22:34:31.289573 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:31.289549 5196 scope.go:122] "RemoveContainer" containerID="78d4402e5d9a4616c7d5fbb4fd1456c482bf0a981e8c4f2d101b043b89a7f391" Nov 03 22:34:31.290932 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:31.290909 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" event={"ID":"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb","Type":"ContainerStarted","Data":"981bd9f52a26c18d870bac2da7b4736f76abc9a7b166395d84275f978d8812d5"} Nov 03 22:34:31.291028 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:31.290959 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" event={"ID":"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb","Type":"ContainerStarted","Data":"e5db1f0dab1ace1d151471a628a5e0bc9f381a831a5a20473119adf7e5d143b2"} Nov 03 22:34:31.291072 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:31.291022 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:34:31.291120 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:31.291107 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:34:31.291176 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:31.291159 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:34:31.534833 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:31.534699 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" podStartSLOduration=1.5346790110000001 podStartE2EDuration="1.534679011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:34:31.533706049 +0000 UTC m=+700.843265344" watchObservedRunningTime="2026-11-03 22:34:31.534679011 +0000 UTC m=+700.844238305" Nov 03 22:34:31.691341 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:31.691308 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:31.759883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:31.754972 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" Nov 03 22:34:32.019009 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:32.018978 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 22:34:32.352108 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:32.352066 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" event={"ID":"aa304409-0e6c-4543-a6ae-3db66214e80d","Type":"ContainerStarted","Data":"fe7fc0df29dd3cc632af54d2c3ad99ea040dedd418576baaa425cf3ebf832a65"} Nov 03 22:34:32.754964 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:32.754873 5196 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 03 22:34:32.754964 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:32.754934 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" podUID="5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 03 22:34:33.260695 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:33.260655 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 22:34:33.261844 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:33.261823 5196 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 03 22:34:33.261950 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:33.261888 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" podUID="aa304409-0e6c-4543-a6ae-3db66214e80d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 03 22:34:33.753424 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:33.753395 5196 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 03 22:34:33.753680 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:33.753653 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" podUID="5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 03 22:34:34.260222 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:34.260179 5196 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 03 22:34:34.260614 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:34.260246 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" podUID="aa304409-0e6c-4543-a6ae-3db66214e80d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 03 22:34:34.753367 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:34.753333 5196 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 03 22:34:34.753547 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:34.753385 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" podUID="5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 03 22:34:35.260839 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:35.260485 5196 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 03 22:34:35.260839 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:35.260538 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" podUID="aa304409-0e6c-4543-a6ae-3db66214e80d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 03 22:34:35.286996 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:35.286970 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65" Nov 03 22:34:35.753901 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:35.753829 5196 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 03 22:34:35.754147 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:35.754122 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" podUID="5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 03 22:34:36.260654 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:36.260615 5196 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 03 22:34:36.260833 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:36.260685 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" podUID="aa304409-0e6c-4543-a6ae-3db66214e80d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 03 22:34:36.753365 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:36.753335 5196 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 03 22:34:36.753844 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:36.753818 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" podUID="5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 03 22:34:36.933881 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:36.932971 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42"] Nov 03 22:34:36.943649 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:36.943623 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42" Nov 03 22:34:36.987968 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:36.987929 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42"] Nov 03 22:34:37.008940 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:37.005129 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-cbd26c-serving-cert\"" Nov 03 22:34:37.015740 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:37.015384 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a86e97c-52cb-4ec2-9ff2-250940649d94-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42\" (UID: \"3a86e97c-52cb-4ec2-9ff2-250940649d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42" Nov 03 22:34:37.015740 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:37.015474 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a86e97c-52cb-4ec2-9ff2-250940649d94-metrics-certs\") pod \"maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42\" (UID: \"3a86e97c-52cb-4ec2-9ff2-250940649d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42" Nov 03 22:34:37.015740 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:37.015504 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zlxh\" (UniqueName: \"kubernetes.io/projected/3a86e97c-52cb-4ec2-9ff2-250940649d94-kube-api-access-8zlxh\") pod \"maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42\" (UID: \"3a86e97c-52cb-4ec2-9ff2-250940649d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42" Nov 03 22:34:37.015740 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:37.015561 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a86e97c-52cb-4ec2-9ff2-250940649d94-maas-api-tls\") pod \"maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42\" (UID: \"3a86e97c-52cb-4ec2-9ff2-250940649d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42" Nov 03 22:34:37.140945 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:37.138720 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a86e97c-52cb-4ec2-9ff2-250940649d94-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42\" (UID: \"3a86e97c-52cb-4ec2-9ff2-250940649d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42" Nov 03 22:34:37.140945 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:37.139265 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:37.141891 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:37.137944 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a86e97c-52cb-4ec2-9ff2-250940649d94-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42\" (UID: \"3a86e97c-52cb-4ec2-9ff2-250940649d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42" Nov 03 22:34:37.141891 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:37.141401 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a86e97c-52cb-4ec2-9ff2-250940649d94-metrics-certs\") pod \"maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42\" (UID: \"3a86e97c-52cb-4ec2-9ff2-250940649d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42" Nov 03 22:34:37.141891 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:37.141434 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zlxh\" (UniqueName: \"kubernetes.io/projected/3a86e97c-52cb-4ec2-9ff2-250940649d94-kube-api-access-8zlxh\") pod \"maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42\" (UID: \"3a86e97c-52cb-4ec2-9ff2-250940649d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42" Nov 03 22:34:37.141891 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:37.141498 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a86e97c-52cb-4ec2-9ff2-250940649d94-maas-api-tls\") pod \"maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42\" (UID: \"3a86e97c-52cb-4ec2-9ff2-250940649d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42" Nov 03 22:34:37.141891 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:34:37.141742 5196 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-cbd26c-serving-cert: secret "maas-api-e2e-worker-w5-cbd26c-serving-cert" not found Nov 03 22:34:37.141891 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:34:37.141806 5196 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a86e97c-52cb-4ec2-9ff2-250940649d94-maas-api-tls podName:3a86e97c-52cb-4ec2-9ff2-250940649d94 nodeName:}" failed. No retries permitted until 2026-11-03 22:34:37.641785718 +0000 UTC m=+706.951345004 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3a86e97c-52cb-4ec2-9ff2-250940649d94-maas-api-tls") pod "maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42" (UID: "3a86e97c-52cb-4ec2-9ff2-250940649d94") : secret "maas-api-e2e-worker-w5-cbd26c-serving-cert" not found Nov 03 22:34:37.144246 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:37.144222 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a86e97c-52cb-4ec2-9ff2-250940649d94-metrics-certs\") pod \"maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42\" (UID: \"3a86e97c-52cb-4ec2-9ff2-250940649d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42" Nov 03 22:34:37.234884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:37.232811 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zlxh\" (UniqueName: \"kubernetes.io/projected/3a86e97c-52cb-4ec2-9ff2-250940649d94-kube-api-access-8zlxh\") pod \"maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42\" (UID: \"3a86e97c-52cb-4ec2-9ff2-250940649d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42" Nov 03 22:34:37.260914 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:37.260810 5196 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 03 22:34:37.260914 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:37.260889 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" podUID="aa304409-0e6c-4543-a6ae-3db66214e80d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 03 22:34:37.354649 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:37.354610 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:37.498060 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:37.498017 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:37.649140 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:37.649101 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a86e97c-52cb-4ec2-9ff2-250940649d94-maas-api-tls\") pod \"maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42\" (UID: \"3a86e97c-52cb-4ec2-9ff2-250940649d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42" Nov 03 22:34:37.651744 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:37.651719 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a86e97c-52cb-4ec2-9ff2-250940649d94-maas-api-tls\") pod \"maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42\" (UID: \"3a86e97c-52cb-4ec2-9ff2-250940649d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42" Nov 03 22:34:37.758501 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:37.758472 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" Nov 03 22:34:37.758906 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:37.758757 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" Nov 03 22:34:37.759565 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:37.759543 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" Nov 03 22:34:37.856242 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:37.856205 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42" Nov 03 22:34:38.231693 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:38.231666 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42"] Nov 03 22:34:38.233757 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:34:38.233731 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a86e97c_52cb_4ec2_9ff2_250940649d94.slice/crio-6d7d89843d296387f216a9f4b7c3c94273f0fd378060b70c4c497a44dfd44a51 WatchSource:0}: Error finding container 6d7d89843d296387f216a9f4b7c3c94273f0fd378060b70c4c497a44dfd44a51: Status 404 returned error can't find the container with id 6d7d89843d296387f216a9f4b7c3c94273f0fd378060b70c4c497a44dfd44a51 Nov 03 22:34:38.263821 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:38.263797 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 22:34:38.264126 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:38.264110 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 22:34:38.264836 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:38.264819 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 22:34:38.386655 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:38.386622 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42" event={"ID":"3a86e97c-52cb-4ec2-9ff2-250940649d94","Type":"ContainerStarted","Data":"6d7d89843d296387f216a9f4b7c3c94273f0fd378060b70c4c497a44dfd44a51"} Nov 03 22:34:39.247261 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:39.247224 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:39.393388 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:39.393356 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42" event={"ID":"3a86e97c-52cb-4ec2-9ff2-250940649d94","Type":"ContainerStarted","Data":"114e29f9265d8fef6116b09bb1ef50357e3a2f7e9953368cfef6e16318b46d34"} Nov 03 22:34:39.393628 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:39.393613 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42" Nov 03 22:34:39.513049 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:39.512998 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42" podStartSLOduration=2.710084082 podStartE2EDuration="3.512983974s" totalImagesPullingTime="802.899892ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:34:39.511159286 +0000 UTC m=+708.820718602" watchObservedRunningTime="2026-11-03 22:34:39.512983974 +0000 UTC m=+708.822543268" Nov 03 22:34:39.850734 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:39.850701 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:39.995782 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:39.995746 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:42.645915 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:42.645879 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:44.208888 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:44.208831 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:45.404715 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:45.404691 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42" Nov 03 22:34:47.698499 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:47.698466 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:49.867220 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:49.867188 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:50.460447 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:50.460414 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:50.605811 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:50.605780 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:50.749518 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:50.749437 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:52.941018 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:52.940989 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:55.115132 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:55.115103 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:55.696771 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:55.696739 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:55.845945 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:55.845914 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:34:55.918470 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:34:55.918331 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:34:58.213514 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:34:58.213478 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:00.116729 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:35:00.116647 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:00.347322 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:35:00.347290 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:03.251701 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:35:03.251671 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:03.396436 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:35:03.396404 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:03.850625 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:35:03.850590 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:03.998394 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:35:03.998361 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:04.181990 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:35:04.181911 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:05.126471 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:35:05.126441 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:07.221511 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:35:07.221484 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:08.005071 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:35:08.005040 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:09.864711 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:35:09.864683 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:10.491201 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:35:10.491169 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:10.665956 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:35:10.665902 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:13.213172 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:35:13.213137 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:16.214332 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:35:16.214297 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:16.897720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:35:16.897687 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:17.076143 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:35:17.076110 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:17.894704 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:35:17.894675 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:18.406002 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:35:18.405969 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:23.210737 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:35:23.210705 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:25.917612 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:35:25.917571 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:35:26.878868 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:35:26.878824 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:27.411470 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:35:27.411437 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:27.564618 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:35:27.564578 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:27.727579 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:35:27.727503 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:28.213148 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:35:28.213117 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:32.927817 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:35:32.927760 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:33.676516 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:35:33.676481 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:35.394610 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:35:35.394577 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:37.802826 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:35:37.802787 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:42.305398 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:35:42.305362 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:42.601929 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:35:42.601901 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:47.337883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:35:47.337111 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:52.098210 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:35:52.098169 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:53.318720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:35:53.318686 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:35:55.918411 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:35:55.918374 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:35:56.821592 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:35:56.821560 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:00.457064 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:00.457031 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:01.602112 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:01.602080 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:05.482984 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:05.482943 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:06.297528 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:06.297492 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:06.640537 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:06.640498 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6"] Nov 03 22:36:06.644238 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:06.644213 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" Nov 03 22:36:06.698248 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:06.698221 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-cbd26c\"/\"default-dockercfg-997r5\"" Nov 03 22:36:06.728288 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:06.728262 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:06.748776 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:06.748752 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-cbd26c\"/\"simulator-w5-cbd26c-kserve-self-signed-certs\"" Nov 03 22:36:06.749188 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:06.749171 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-cbd26c\"/\"openshift-service-ca.crt\"" Nov 03 22:36:06.751184 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:06.751159 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6"] Nov 03 22:36:06.751288 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:06.751238 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-cbd26c\"/\"kube-root-ca.crt\"" Nov 03 22:36:06.819201 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:06.819173 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82749c55-0fca-4a14-b6ba-7df122a6a407-dshm\") pod \"simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6\" (UID: \"82749c55-0fca-4a14-b6ba-7df122a6a407\") " pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" Nov 03 22:36:06.819407 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:06.819301 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82749c55-0fca-4a14-b6ba-7df122a6a407-model-cache\") pod \"simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6\" (UID: \"82749c55-0fca-4a14-b6ba-7df122a6a407\") " pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" Nov 03 22:36:06.819407 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:06.819348 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82749c55-0fca-4a14-b6ba-7df122a6a407-home\") pod \"simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6\" (UID: \"82749c55-0fca-4a14-b6ba-7df122a6a407\") " pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" Nov 03 22:36:06.819503 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:06.819437 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z5fq\" (UniqueName: \"kubernetes.io/projected/82749c55-0fca-4a14-b6ba-7df122a6a407-kube-api-access-5z5fq\") pod \"simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6\" (UID: \"82749c55-0fca-4a14-b6ba-7df122a6a407\") " pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" Nov 03 22:36:06.819503 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:06.819464 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82749c55-0fca-4a14-b6ba-7df122a6a407-tls-certs\") pod \"simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6\" (UID: \"82749c55-0fca-4a14-b6ba-7df122a6a407\") " pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" Nov 03 22:36:06.819582 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:06.819503 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82749c55-0fca-4a14-b6ba-7df122a6a407-tmp-dir\") pod \"simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6\" (UID: \"82749c55-0fca-4a14-b6ba-7df122a6a407\") " pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" Nov 03 22:36:06.920582 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:06.920494 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82749c55-0fca-4a14-b6ba-7df122a6a407-model-cache\") pod \"simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6\" (UID: \"82749c55-0fca-4a14-b6ba-7df122a6a407\") " pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" Nov 03 22:36:06.920582 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:06.920553 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82749c55-0fca-4a14-b6ba-7df122a6a407-home\") pod \"simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6\" (UID: \"82749c55-0fca-4a14-b6ba-7df122a6a407\") " pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" Nov 03 22:36:06.920819 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:06.920586 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z5fq\" (UniqueName: \"kubernetes.io/projected/82749c55-0fca-4a14-b6ba-7df122a6a407-kube-api-access-5z5fq\") pod \"simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6\" (UID: \"82749c55-0fca-4a14-b6ba-7df122a6a407\") " pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" Nov 03 22:36:06.920819 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:06.920616 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82749c55-0fca-4a14-b6ba-7df122a6a407-tls-certs\") pod \"simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6\" (UID: \"82749c55-0fca-4a14-b6ba-7df122a6a407\") " pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" Nov 03 22:36:06.920819 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:06.920667 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82749c55-0fca-4a14-b6ba-7df122a6a407-tmp-dir\") pod \"simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6\" (UID: \"82749c55-0fca-4a14-b6ba-7df122a6a407\") " pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" Nov 03 22:36:06.920819 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:06.920718 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82749c55-0fca-4a14-b6ba-7df122a6a407-dshm\") pod \"simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6\" (UID: \"82749c55-0fca-4a14-b6ba-7df122a6a407\") " pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" Nov 03 22:36:06.921069 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:06.921018 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82749c55-0fca-4a14-b6ba-7df122a6a407-model-cache\") pod \"simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6\" (UID: \"82749c55-0fca-4a14-b6ba-7df122a6a407\") " pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" Nov 03 22:36:06.921128 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:06.921060 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82749c55-0fca-4a14-b6ba-7df122a6a407-home\") pod \"simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6\" (UID: \"82749c55-0fca-4a14-b6ba-7df122a6a407\") " pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" Nov 03 22:36:06.921225 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:06.921196 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82749c55-0fca-4a14-b6ba-7df122a6a407-tmp-dir\") pod \"simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6\" (UID: \"82749c55-0fca-4a14-b6ba-7df122a6a407\") " pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" Nov 03 22:36:06.923330 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:06.923270 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82749c55-0fca-4a14-b6ba-7df122a6a407-dshm\") pod \"simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6\" (UID: \"82749c55-0fca-4a14-b6ba-7df122a6a407\") " pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" Nov 03 22:36:06.924131 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:06.924112 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82749c55-0fca-4a14-b6ba-7df122a6a407-tls-certs\") pod \"simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6\" (UID: \"82749c55-0fca-4a14-b6ba-7df122a6a407\") " pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" Nov 03 22:36:06.982956 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:06.982848 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z5fq\" (UniqueName: \"kubernetes.io/projected/82749c55-0fca-4a14-b6ba-7df122a6a407-kube-api-access-5z5fq\") pod \"simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6\" (UID: \"82749c55-0fca-4a14-b6ba-7df122a6a407\") " pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" Nov 03 22:36:07.255996 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:07.255923 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" Nov 03 22:36:07.629480 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:07.629455 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6"] Nov 03 22:36:07.631242 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:36:07.631215 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82749c55_0fca_4a14_b6ba_7df122a6a407.slice/crio-9e23e5d1ee8f209f4e07c3186fb2053e2e5296495a18e2980bc925885fe57783 WatchSource:0}: Error finding container 9e23e5d1ee8f209f4e07c3186fb2053e2e5296495a18e2980bc925885fe57783: Status 404 returned error can't find the container with id 9e23e5d1ee8f209f4e07c3186fb2053e2e5296495a18e2980bc925885fe57783 Nov 03 22:36:07.882308 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:07.882226 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" event={"ID":"82749c55-0fca-4a14-b6ba-7df122a6a407","Type":"ContainerStarted","Data":"9e23e5d1ee8f209f4e07c3186fb2053e2e5296495a18e2980bc925885fe57783"} Nov 03 22:36:09.894063 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:09.894028 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" event={"ID":"82749c55-0fca-4a14-b6ba-7df122a6a407","Type":"ContainerStarted","Data":"94d5702d4f2a5a0f8105bbcea39f754e07b84bbea99c7c24ad25c92f879b06d0"} Nov 03 22:36:10.012723 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:10.012676 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" podStartSLOduration=2.159803304 podStartE2EDuration="4.01266419s" totalImagesPullingTime="1.852860886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:36:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:36:10.012128844 +0000 UTC m=+799.321688138" watchObservedRunningTime="2026-11-03 22:36:10.01266419 +0000 UTC m=+799.322223489" Nov 03 22:36:10.256922 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:10.256814 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" Nov 03 22:36:11.021932 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:11.021899 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:15.815346 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:15.815317 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:17.256576 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:17.256540 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" Nov 03 22:36:17.269303 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:17.269275 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" Nov 03 22:36:17.946004 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:17.945977 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" Nov 03 22:36:20.609579 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:20.609538 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:23.346083 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:23.346047 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:25.434012 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:25.433976 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:25.917103 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:36:25.917067 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:36:30.139669 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:30.139572 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:34.901512 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:34.901478 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:39.612099 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:39.612062 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:40.197120 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:40.197094 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:44.411480 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:44.411443 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:45.744713 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:45.744679 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f"] Nov 03 22:36:45.748056 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:45.748035 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" Nov 03 22:36:45.801431 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:45.801408 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-7afb1c-openshift-default-dockercfg-nlmnh\"" Nov 03 22:36:45.854094 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:45.854048 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f"] Nov 03 22:36:45.907118 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:45.907079 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/038354d8-e0d0-4f26-8370-183f575f5710-istiod-ca-cert\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f\" (UID: \"038354d8-e0d0-4f26-8370-183f575f5710\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" Nov 03 22:36:45.907118 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:45.907131 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/038354d8-e0d0-4f26-8370-183f575f5710-istio-token\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f\" (UID: \"038354d8-e0d0-4f26-8370-183f575f5710\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" Nov 03 22:36:45.907378 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:45.907232 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djz54\" (UniqueName: \"kubernetes.io/projected/038354d8-e0d0-4f26-8370-183f575f5710-kube-api-access-djz54\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f\" (UID: \"038354d8-e0d0-4f26-8370-183f575f5710\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" Nov 03 22:36:45.907421 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:45.907393 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/038354d8-e0d0-4f26-8370-183f575f5710-istio-envoy\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f\" (UID: \"038354d8-e0d0-4f26-8370-183f575f5710\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" Nov 03 22:36:45.907459 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:45.907434 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/038354d8-e0d0-4f26-8370-183f575f5710-workload-certs\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f\" (UID: \"038354d8-e0d0-4f26-8370-183f575f5710\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" Nov 03 22:36:45.907498 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:45.907464 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/038354d8-e0d0-4f26-8370-183f575f5710-credential-socket\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f\" (UID: \"038354d8-e0d0-4f26-8370-183f575f5710\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" Nov 03 22:36:45.907540 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:45.907507 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/038354d8-e0d0-4f26-8370-183f575f5710-workload-socket\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f\" (UID: \"038354d8-e0d0-4f26-8370-183f575f5710\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" Nov 03 22:36:45.907623 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:45.907607 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/038354d8-e0d0-4f26-8370-183f575f5710-istio-data\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f\" (UID: \"038354d8-e0d0-4f26-8370-183f575f5710\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" Nov 03 22:36:45.907667 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:45.907645 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/038354d8-e0d0-4f26-8370-183f575f5710-istio-podinfo\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f\" (UID: \"038354d8-e0d0-4f26-8370-183f575f5710\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" Nov 03 22:36:46.009333 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:46.009235 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/038354d8-e0d0-4f26-8370-183f575f5710-istio-envoy\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f\" (UID: \"038354d8-e0d0-4f26-8370-183f575f5710\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" Nov 03 22:36:46.009333 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:46.009275 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/038354d8-e0d0-4f26-8370-183f575f5710-workload-certs\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f\" (UID: \"038354d8-e0d0-4f26-8370-183f575f5710\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" Nov 03 22:36:46.009333 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:46.009295 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/038354d8-e0d0-4f26-8370-183f575f5710-credential-socket\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f\" (UID: \"038354d8-e0d0-4f26-8370-183f575f5710\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" Nov 03 22:36:46.009333 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:46.009313 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/038354d8-e0d0-4f26-8370-183f575f5710-workload-socket\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f\" (UID: \"038354d8-e0d0-4f26-8370-183f575f5710\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" Nov 03 22:36:46.009675 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:46.009344 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/038354d8-e0d0-4f26-8370-183f575f5710-istio-data\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f\" (UID: \"038354d8-e0d0-4f26-8370-183f575f5710\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" Nov 03 22:36:46.009675 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:46.009366 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/038354d8-e0d0-4f26-8370-183f575f5710-istio-podinfo\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f\" (UID: \"038354d8-e0d0-4f26-8370-183f575f5710\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" Nov 03 22:36:46.009675 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:46.009435 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/038354d8-e0d0-4f26-8370-183f575f5710-istiod-ca-cert\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f\" (UID: \"038354d8-e0d0-4f26-8370-183f575f5710\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" Nov 03 22:36:46.009675 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:46.009478 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/038354d8-e0d0-4f26-8370-183f575f5710-istio-token\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f\" (UID: \"038354d8-e0d0-4f26-8370-183f575f5710\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" Nov 03 22:36:46.009675 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:46.009504 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djz54\" (UniqueName: \"kubernetes.io/projected/038354d8-e0d0-4f26-8370-183f575f5710-kube-api-access-djz54\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f\" (UID: \"038354d8-e0d0-4f26-8370-183f575f5710\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" Nov 03 22:36:46.009972 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:46.009786 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/038354d8-e0d0-4f26-8370-183f575f5710-istio-data\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f\" (UID: \"038354d8-e0d0-4f26-8370-183f575f5710\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" Nov 03 22:36:46.009972 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:46.009901 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/038354d8-e0d0-4f26-8370-183f575f5710-credential-socket\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f\" (UID: \"038354d8-e0d0-4f26-8370-183f575f5710\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" Nov 03 22:36:46.010385 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:46.010362 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/038354d8-e0d0-4f26-8370-183f575f5710-workload-certs\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f\" (UID: \"038354d8-e0d0-4f26-8370-183f575f5710\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" Nov 03 22:36:46.010519 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:46.010496 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/038354d8-e0d0-4f26-8370-183f575f5710-workload-socket\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f\" (UID: \"038354d8-e0d0-4f26-8370-183f575f5710\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" Nov 03 22:36:46.010835 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:46.010783 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/038354d8-e0d0-4f26-8370-183f575f5710-istiod-ca-cert\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f\" (UID: \"038354d8-e0d0-4f26-8370-183f575f5710\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" Nov 03 22:36:46.012005 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:46.011984 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/038354d8-e0d0-4f26-8370-183f575f5710-istio-envoy\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f\" (UID: \"038354d8-e0d0-4f26-8370-183f575f5710\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" Nov 03 22:36:46.012665 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:46.012640 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/038354d8-e0d0-4f26-8370-183f575f5710-istio-podinfo\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f\" (UID: \"038354d8-e0d0-4f26-8370-183f575f5710\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" Nov 03 22:36:46.071633 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:46.071596 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/038354d8-e0d0-4f26-8370-183f575f5710-istio-token\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f\" (UID: \"038354d8-e0d0-4f26-8370-183f575f5710\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" Nov 03 22:36:46.072044 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:46.072025 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djz54\" (UniqueName: \"kubernetes.io/projected/038354d8-e0d0-4f26-8370-183f575f5710-kube-api-access-djz54\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f\" (UID: \"038354d8-e0d0-4f26-8370-183f575f5710\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" Nov 03 22:36:46.361241 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:46.361208 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" Nov 03 22:36:46.827431 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:46.827403 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f"] Nov 03 22:36:46.828985 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:36:46.828959 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod038354d8_e0d0_4f26_8370_183f575f5710.slice/crio-3d1e3ead188c1d3a750ddc5e7065c9416c7d65115c412a65d85c0f8e0ca462a8 WatchSource:0}: Error finding container 3d1e3ead188c1d3a750ddc5e7065c9416c7d65115c412a65d85c0f8e0ca462a8: Status 404 returned error can't find the container with id 3d1e3ead188c1d3a750ddc5e7065c9416c7d65115c412a65d85c0f8e0ca462a8 Nov 03 22:36:46.831189 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:46.831157 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:36:46.831260 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:46.831224 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:36:46.831260 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:46.831256 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:36:47.095922 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:47.095849 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" event={"ID":"038354d8-e0d0-4f26-8370-183f575f5710","Type":"ContainerStarted","Data":"7718b72a0226f15298f2b384c46969941c3a29233f4f503e0b7d0f7a55e4c198"} Nov 03 22:36:47.096098 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:47.095941 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" event={"ID":"038354d8-e0d0-4f26-8370-183f575f5710","Type":"ContainerStarted","Data":"3d1e3ead188c1d3a750ddc5e7065c9416c7d65115c412a65d85c0f8e0ca462a8"} Nov 03 22:36:47.220018 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:47.219958 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" podStartSLOduration=2.219939895 podStartE2EDuration="2.219939895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:36:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:36:47.217441398 +0000 UTC m=+836.527000690" watchObservedRunningTime="2026-11-03 22:36:47.219939895 +0000 UTC m=+836.529499188" Nov 03 22:36:47.362185 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:47.362089 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" Nov 03 22:36:47.363640 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:47.363613 5196 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 22:36:47.363782 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:47.363662 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" podUID="038354d8-e0d0-4f26-8370-183f575f5710" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 22:36:48.362184 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:48.362152 5196 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 22:36:48.362698 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:48.362207 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" podUID="038354d8-e0d0-4f26-8370-183f575f5710" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 22:36:49.247070 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:49.247038 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:49.361687 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:49.361653 5196 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 22:36:49.361881 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:49.361707 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" podUID="038354d8-e0d0-4f26-8370-183f575f5710" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 22:36:50.362480 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:50.362443 5196 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 22:36:50.362930 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:50.362497 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" podUID="038354d8-e0d0-4f26-8370-183f575f5710" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 22:36:51.362399 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:51.362364 5196 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 22:36:51.362563 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:51.362416 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" podUID="038354d8-e0d0-4f26-8370-183f575f5710" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 22:36:52.362459 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:52.362427 5196 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 22:36:52.362629 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:52.362492 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" podUID="038354d8-e0d0-4f26-8370-183f575f5710" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 22:36:53.439275 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:53.439241 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" Nov 03 22:36:53.439957 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:53.439936 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" Nov 03 22:36:53.440780 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:53.440759 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" Nov 03 22:36:54.013653 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:54.013614 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:55.917644 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:36:55.917602 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:36:57.540043 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:57.540009 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc"] Nov 03 22:36:57.543813 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:57.543797 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" Nov 03 22:36:57.630152 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:57.630124 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34e9cf3a-3456-4835-8933-102ce735cccc-model-cache\") pod \"premium-w2-164c36-kserve-74c7c9fdcc-8hlrc\" (UID: \"34e9cf3a-3456-4835-8933-102ce735cccc\") " pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" Nov 03 22:36:57.630307 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:57.630187 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34e9cf3a-3456-4835-8933-102ce735cccc-tls-certs\") pod \"premium-w2-164c36-kserve-74c7c9fdcc-8hlrc\" (UID: \"34e9cf3a-3456-4835-8933-102ce735cccc\") " pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" Nov 03 22:36:57.630307 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:57.630270 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmwsq\" (UniqueName: \"kubernetes.io/projected/34e9cf3a-3456-4835-8933-102ce735cccc-kube-api-access-kmwsq\") pod \"premium-w2-164c36-kserve-74c7c9fdcc-8hlrc\" (UID: \"34e9cf3a-3456-4835-8933-102ce735cccc\") " pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" Nov 03 22:36:57.630385 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:57.630312 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34e9cf3a-3456-4835-8933-102ce735cccc-home\") pod \"premium-w2-164c36-kserve-74c7c9fdcc-8hlrc\" (UID: \"34e9cf3a-3456-4835-8933-102ce735cccc\") " pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" Nov 03 22:36:57.630419 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:57.630375 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34e9cf3a-3456-4835-8933-102ce735cccc-dshm\") pod \"premium-w2-164c36-kserve-74c7c9fdcc-8hlrc\" (UID: \"34e9cf3a-3456-4835-8933-102ce735cccc\") " pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" Nov 03 22:36:57.630452 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:57.630426 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34e9cf3a-3456-4835-8933-102ce735cccc-tmp-dir\") pod \"premium-w2-164c36-kserve-74c7c9fdcc-8hlrc\" (UID: \"34e9cf3a-3456-4835-8933-102ce735cccc\") " pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" Nov 03 22:36:57.635310 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:57.635266 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:57.647794 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:57.647769 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-164c36\"/\"default-dockercfg-rb9gv\"" Nov 03 22:36:57.647794 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:57.647774 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-164c36\"/\"kube-root-ca.crt\"" Nov 03 22:36:57.647985 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:57.647802 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-164c36\"/\"openshift-service-ca.crt\"" Nov 03 22:36:57.647985 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:57.647937 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-164c36\"/\"premium-w2-164c36-kserve-self-signed-certs\"" Nov 03 22:36:57.649842 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:57.649824 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc"] Nov 03 22:36:57.731499 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:57.731460 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34e9cf3a-3456-4835-8933-102ce735cccc-tls-certs\") pod \"premium-w2-164c36-kserve-74c7c9fdcc-8hlrc\" (UID: \"34e9cf3a-3456-4835-8933-102ce735cccc\") " pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" Nov 03 22:36:57.731687 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:57.731519 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmwsq\" (UniqueName: \"kubernetes.io/projected/34e9cf3a-3456-4835-8933-102ce735cccc-kube-api-access-kmwsq\") pod \"premium-w2-164c36-kserve-74c7c9fdcc-8hlrc\" (UID: \"34e9cf3a-3456-4835-8933-102ce735cccc\") " pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" Nov 03 22:36:57.731687 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:57.731555 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34e9cf3a-3456-4835-8933-102ce735cccc-home\") pod \"premium-w2-164c36-kserve-74c7c9fdcc-8hlrc\" (UID: \"34e9cf3a-3456-4835-8933-102ce735cccc\") " pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" Nov 03 22:36:57.731687 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:57.731587 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34e9cf3a-3456-4835-8933-102ce735cccc-dshm\") pod \"premium-w2-164c36-kserve-74c7c9fdcc-8hlrc\" (UID: \"34e9cf3a-3456-4835-8933-102ce735cccc\") " pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" Nov 03 22:36:57.731687 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:57.731621 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34e9cf3a-3456-4835-8933-102ce735cccc-tmp-dir\") pod \"premium-w2-164c36-kserve-74c7c9fdcc-8hlrc\" (UID: \"34e9cf3a-3456-4835-8933-102ce735cccc\") " pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" Nov 03 22:36:57.731928 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:57.731698 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34e9cf3a-3456-4835-8933-102ce735cccc-model-cache\") pod \"premium-w2-164c36-kserve-74c7c9fdcc-8hlrc\" (UID: \"34e9cf3a-3456-4835-8933-102ce735cccc\") " pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" Nov 03 22:36:57.732277 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:57.732253 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34e9cf3a-3456-4835-8933-102ce735cccc-tmp-dir\") pod \"premium-w2-164c36-kserve-74c7c9fdcc-8hlrc\" (UID: \"34e9cf3a-3456-4835-8933-102ce735cccc\") " pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" Nov 03 22:36:57.732704 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:57.732515 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34e9cf3a-3456-4835-8933-102ce735cccc-model-cache\") pod \"premium-w2-164c36-kserve-74c7c9fdcc-8hlrc\" (UID: \"34e9cf3a-3456-4835-8933-102ce735cccc\") " pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" Nov 03 22:36:57.732704 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:57.732534 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34e9cf3a-3456-4835-8933-102ce735cccc-home\") pod \"premium-w2-164c36-kserve-74c7c9fdcc-8hlrc\" (UID: \"34e9cf3a-3456-4835-8933-102ce735cccc\") " pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" Nov 03 22:36:57.734409 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:57.734388 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34e9cf3a-3456-4835-8933-102ce735cccc-dshm\") pod \"premium-w2-164c36-kserve-74c7c9fdcc-8hlrc\" (UID: \"34e9cf3a-3456-4835-8933-102ce735cccc\") " pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" Nov 03 22:36:57.734515 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:57.734501 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34e9cf3a-3456-4835-8933-102ce735cccc-tls-certs\") pod \"premium-w2-164c36-kserve-74c7c9fdcc-8hlrc\" (UID: \"34e9cf3a-3456-4835-8933-102ce735cccc\") " pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" Nov 03 22:36:57.793811 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:57.793730 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmwsq\" (UniqueName: \"kubernetes.io/projected/34e9cf3a-3456-4835-8933-102ce735cccc-kube-api-access-kmwsq\") pod \"premium-w2-164c36-kserve-74c7c9fdcc-8hlrc\" (UID: \"34e9cf3a-3456-4835-8933-102ce735cccc\") " pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" Nov 03 22:36:57.855691 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:57.855660 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" Nov 03 22:36:58.237286 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:58.237263 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc"] Nov 03 22:36:58.239504 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:36:58.239472 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34e9cf3a_3456_4835_8933_102ce735cccc.slice/crio-c8695b22ecdf3d1f97569320db04bbb22add9146bc97aa6f6000a4d2908e2b02 WatchSource:0}: Error finding container c8695b22ecdf3d1f97569320db04bbb22add9146bc97aa6f6000a4d2908e2b02: Status 404 returned error can't find the container with id c8695b22ecdf3d1f97569320db04bbb22add9146bc97aa6f6000a4d2908e2b02 Nov 03 22:36:59.178774 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:59.178739 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" event={"ID":"34e9cf3a-3456-4835-8933-102ce735cccc","Type":"ContainerStarted","Data":"e05cefab23c6656ebcf697f93755ee7a8e941637acf10f2594a65d1f99625170"} Nov 03 22:36:59.179285 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:59.178783 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" event={"ID":"34e9cf3a-3456-4835-8933-102ce735cccc","Type":"ContainerStarted","Data":"c8695b22ecdf3d1f97569320db04bbb22add9146bc97aa6f6000a4d2908e2b02"} Nov 03 22:36:59.297573 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:59.297522 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" podStartSLOduration=2.297508401 podStartE2EDuration="2.297508401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:36:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:36:59.296490128 +0000 UTC m=+848.606049432" watchObservedRunningTime="2026-11-03 22:36:59.297508401 +0000 UTC m=+848.607067694" Nov 03 22:36:59.547990 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:59.547910 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:36:59.856464 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:36:59.856432 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" Nov 03 22:37:00.807578 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:00.807537 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:03.704902 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:03.704872 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:05.828239 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:05.828209 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:07.856756 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:07.856722 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" Nov 03 22:37:07.869818 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:07.869793 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" Nov 03 22:37:08.239463 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:08.239385 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" Nov 03 22:37:08.410804 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:08.410758 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:13.207933 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:13.207890 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:15.690690 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:15.690655 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:18.045966 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:18.045937 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:22.808030 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:22.807991 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:25.918272 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:37:25.918232 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:37:27.622839 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:27.622801 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:29.019522 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:29.019484 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:30.803541 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:30.803507 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg"] Nov 03 22:37:30.807467 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:30.807445 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" Nov 03 22:37:30.861296 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:30.861265 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-b64e2e-openshift-default-dockercfg-g4cxs\"" Nov 03 22:37:30.888922 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:30.888895 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcsl7\" (UniqueName: \"kubernetes.io/projected/b8d72750-08ae-4995-ac23-6304ada9b53f-kube-api-access-tcsl7\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg\" (UID: \"b8d72750-08ae-4995-ac23-6304ada9b53f\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" Nov 03 22:37:30.889038 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:30.888945 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8d72750-08ae-4995-ac23-6304ada9b53f-istiod-ca-cert\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg\" (UID: \"b8d72750-08ae-4995-ac23-6304ada9b53f\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" Nov 03 22:37:30.889095 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:30.889081 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8d72750-08ae-4995-ac23-6304ada9b53f-istio-data\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg\" (UID: \"b8d72750-08ae-4995-ac23-6304ada9b53f\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" Nov 03 22:37:30.889132 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:30.889110 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d72750-08ae-4995-ac23-6304ada9b53f-credential-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg\" (UID: \"b8d72750-08ae-4995-ac23-6304ada9b53f\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" Nov 03 22:37:30.889132 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:30.889127 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d72750-08ae-4995-ac23-6304ada9b53f-workload-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg\" (UID: \"b8d72750-08ae-4995-ac23-6304ada9b53f\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" Nov 03 22:37:30.889225 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:30.889142 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8d72750-08ae-4995-ac23-6304ada9b53f-istio-envoy\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg\" (UID: \"b8d72750-08ae-4995-ac23-6304ada9b53f\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" Nov 03 22:37:30.889315 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:30.889290 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8d72750-08ae-4995-ac23-6304ada9b53f-workload-certs\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg\" (UID: \"b8d72750-08ae-4995-ac23-6304ada9b53f\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" Nov 03 22:37:30.889366 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:30.889341 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8d72750-08ae-4995-ac23-6304ada9b53f-istio-podinfo\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg\" (UID: \"b8d72750-08ae-4995-ac23-6304ada9b53f\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" Nov 03 22:37:30.889435 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:30.889422 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8d72750-08ae-4995-ac23-6304ada9b53f-istio-token\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg\" (UID: \"b8d72750-08ae-4995-ac23-6304ada9b53f\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" Nov 03 22:37:30.915121 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:30.915095 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg"] Nov 03 22:37:30.990234 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:30.990199 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8d72750-08ae-4995-ac23-6304ada9b53f-istio-token\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg\" (UID: \"b8d72750-08ae-4995-ac23-6304ada9b53f\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" Nov 03 22:37:30.990234 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:30.990240 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcsl7\" (UniqueName: \"kubernetes.io/projected/b8d72750-08ae-4995-ac23-6304ada9b53f-kube-api-access-tcsl7\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg\" (UID: \"b8d72750-08ae-4995-ac23-6304ada9b53f\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" Nov 03 22:37:30.990509 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:30.990382 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8d72750-08ae-4995-ac23-6304ada9b53f-istiod-ca-cert\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg\" (UID: \"b8d72750-08ae-4995-ac23-6304ada9b53f\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" Nov 03 22:37:30.990509 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:30.990479 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8d72750-08ae-4995-ac23-6304ada9b53f-istio-data\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg\" (UID: \"b8d72750-08ae-4995-ac23-6304ada9b53f\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" Nov 03 22:37:30.990622 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:30.990512 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d72750-08ae-4995-ac23-6304ada9b53f-credential-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg\" (UID: \"b8d72750-08ae-4995-ac23-6304ada9b53f\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" Nov 03 22:37:30.990622 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:30.990539 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d72750-08ae-4995-ac23-6304ada9b53f-workload-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg\" (UID: \"b8d72750-08ae-4995-ac23-6304ada9b53f\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" Nov 03 22:37:30.990622 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:30.990565 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8d72750-08ae-4995-ac23-6304ada9b53f-istio-envoy\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg\" (UID: \"b8d72750-08ae-4995-ac23-6304ada9b53f\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" Nov 03 22:37:30.991567 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:30.991247 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8d72750-08ae-4995-ac23-6304ada9b53f-workload-certs\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg\" (UID: \"b8d72750-08ae-4995-ac23-6304ada9b53f\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" Nov 03 22:37:30.991567 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:30.991343 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8d72750-08ae-4995-ac23-6304ada9b53f-istio-podinfo\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg\" (UID: \"b8d72750-08ae-4995-ac23-6304ada9b53f\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" Nov 03 22:37:30.991567 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:30.991362 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d72750-08ae-4995-ac23-6304ada9b53f-credential-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg\" (UID: \"b8d72750-08ae-4995-ac23-6304ada9b53f\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" Nov 03 22:37:30.991567 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:30.991369 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8d72750-08ae-4995-ac23-6304ada9b53f-istio-data\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg\" (UID: \"b8d72750-08ae-4995-ac23-6304ada9b53f\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" Nov 03 22:37:30.991567 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:30.991528 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8d72750-08ae-4995-ac23-6304ada9b53f-istiod-ca-cert\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg\" (UID: \"b8d72750-08ae-4995-ac23-6304ada9b53f\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" Nov 03 22:37:30.991964 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:30.991913 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d72750-08ae-4995-ac23-6304ada9b53f-workload-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg\" (UID: \"b8d72750-08ae-4995-ac23-6304ada9b53f\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" Nov 03 22:37:30.993885 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:30.993835 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8d72750-08ae-4995-ac23-6304ada9b53f-workload-certs\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg\" (UID: \"b8d72750-08ae-4995-ac23-6304ada9b53f\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" Nov 03 22:37:30.994014 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:30.993944 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8d72750-08ae-4995-ac23-6304ada9b53f-istio-envoy\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg\" (UID: \"b8d72750-08ae-4995-ac23-6304ada9b53f\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" Nov 03 22:37:30.994718 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:30.994695 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8d72750-08ae-4995-ac23-6304ada9b53f-istio-podinfo\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg\" (UID: \"b8d72750-08ae-4995-ac23-6304ada9b53f\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" Nov 03 22:37:31.055231 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:31.055158 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8d72750-08ae-4995-ac23-6304ada9b53f-istio-token\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg\" (UID: \"b8d72750-08ae-4995-ac23-6304ada9b53f\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" Nov 03 22:37:31.055231 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:31.055224 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcsl7\" (UniqueName: \"kubernetes.io/projected/b8d72750-08ae-4995-ac23-6304ada9b53f-kube-api-access-tcsl7\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg\" (UID: \"b8d72750-08ae-4995-ac23-6304ada9b53f\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" Nov 03 22:37:31.121760 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:31.121718 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" Nov 03 22:37:31.554010 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:31.553985 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg"] Nov 03 22:37:31.556410 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:37:31.556372 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8d72750_08ae_4995_ac23_6304ada9b53f.slice/crio-3ede1c71069cd497aef25919fe0d3b6baa2385f9fc0ef669448ccc9ea3ab27b2 WatchSource:0}: Error finding container 3ede1c71069cd497aef25919fe0d3b6baa2385f9fc0ef669448ccc9ea3ab27b2: Status 404 returned error can't find the container with id 3ede1c71069cd497aef25919fe0d3b6baa2385f9fc0ef669448ccc9ea3ab27b2 Nov 03 22:37:31.558754 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:31.558714 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:37:31.558840 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:31.558785 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:37:31.558902 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:31.558845 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:37:32.383532 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:32.383489 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" event={"ID":"b8d72750-08ae-4995-ac23-6304ada9b53f","Type":"ContainerStarted","Data":"0145bcb8d4e420d9fc45e443abe0fa5436144a62a0135c5dfcb7c81ef54ffd69"} Nov 03 22:37:32.383986 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:32.383561 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" event={"ID":"b8d72750-08ae-4995-ac23-6304ada9b53f","Type":"ContainerStarted","Data":"3ede1c71069cd497aef25919fe0d3b6baa2385f9fc0ef669448ccc9ea3ab27b2"} Nov 03 22:37:32.439238 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:32.439205 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:32.558518 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:32.558463 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" podStartSLOduration=2.5584496960000003 podStartE2EDuration="2.558449696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:32.556102402 +0000 UTC m=+881.865661713" watchObservedRunningTime="2026-11-03 22:37:32.558449696 +0000 UTC m=+881.868008988" Nov 03 22:37:33.122189 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:33.122154 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" Nov 03 22:37:33.123809 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:33.123779 5196 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 03 22:37:33.123982 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:33.123834 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" podUID="b8d72750-08ae-4995-ac23-6304ada9b53f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 03 22:37:34.122167 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:34.122125 5196 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 03 22:37:34.122556 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:34.122198 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" podUID="b8d72750-08ae-4995-ac23-6304ada9b53f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 03 22:37:35.122793 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:35.122757 5196 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 03 22:37:35.123278 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:35.122823 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" podUID="b8d72750-08ae-4995-ac23-6304ada9b53f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 03 22:37:36.123137 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:36.123101 5196 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 03 22:37:36.123545 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:36.123168 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" podUID="b8d72750-08ae-4995-ac23-6304ada9b53f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 03 22:37:37.122562 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:37.122528 5196 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 03 22:37:37.122813 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:37.122594 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" podUID="b8d72750-08ae-4995-ac23-6304ada9b53f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 03 22:37:37.345382 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:37.345348 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:38.232363 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:38.232330 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" Nov 03 22:37:38.232970 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:38.232913 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" Nov 03 22:37:38.233898 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:38.233880 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" Nov 03 22:37:42.142212 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:42.142173 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:44.972567 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:44.972533 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:47.023068 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:47.023033 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:51.738437 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:51.738401 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:37:55.920336 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:37:55.920294 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:37:56.544444 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:37:56.544396 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:01.214158 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:38:01.214065 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:01.406101 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:38:01.406058 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:06.138219 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:38:06.138170 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:06.285334 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:38:06.285304 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:10.918110 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:38:10.918072 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:15.739997 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:38:15.739957 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:20.395747 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:38:20.395714 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:20.612904 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:38:20.612874 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:23.696058 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:38:23.696007 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:23.840244 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:38:23.840216 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:25.404953 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:38:25.404847 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:25.918097 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:38:25.918053 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:38:30.114257 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:38:30.114218 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:34.911284 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:38:34.911251 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:39.836883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:38:39.836377 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:44.743467 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:38:44.743433 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:49.573750 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:38:49.573714 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:54.399137 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:38:54.399106 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:38:55.917641 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:38:55.917605 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:38:59.223516 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:38:59.223474 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:01.152679 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:01.152648 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442"] Nov 03 22:39:01.156584 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:01.156564 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" Nov 03 22:39:01.173500 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:01.173479 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-istiod-ca-cert\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442\" (UID: \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" Nov 03 22:39:01.173598 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:01.173521 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-istio-data\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442\" (UID: \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" Nov 03 22:39:01.173598 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:01.173538 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-credential-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442\" (UID: \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" Nov 03 22:39:01.173676 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:01.173642 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-istio-envoy\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442\" (UID: \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" Nov 03 22:39:01.173714 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:01.173685 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-istio-podinfo\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442\" (UID: \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" Nov 03 22:39:01.173747 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:01.173715 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-workload-certs\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442\" (UID: \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" Nov 03 22:39:01.173747 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:01.173739 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-workload-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442\" (UID: \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" Nov 03 22:39:01.173809 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:01.173786 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-istio-token\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442\" (UID: \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" Nov 03 22:39:01.173880 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:01.173843 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrb52\" (UniqueName: \"kubernetes.io/projected/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-kube-api-access-xrb52\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442\" (UID: \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" Nov 03 22:39:01.210009 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:01.209978 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-32449d-openshift-default-dockercfg-rc6q5\"" Nov 03 22:39:01.275110 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:01.275079 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-workload-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442\" (UID: \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" Nov 03 22:39:01.275257 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:01.275133 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-istio-token\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442\" (UID: \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" Nov 03 22:39:01.275257 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:01.275169 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrb52\" (UniqueName: \"kubernetes.io/projected/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-kube-api-access-xrb52\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442\" (UID: \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" Nov 03 22:39:01.275257 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:01.275236 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-istiod-ca-cert\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442\" (UID: \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" Nov 03 22:39:01.275427 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:01.275294 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-istio-data\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442\" (UID: \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" Nov 03 22:39:01.275427 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:01.275325 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-credential-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442\" (UID: \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" Nov 03 22:39:01.275427 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:01.275366 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-istio-envoy\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442\" (UID: \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" Nov 03 22:39:01.275427 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:01.275410 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-istio-podinfo\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442\" (UID: \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" Nov 03 22:39:01.275629 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:01.275441 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-workload-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442\" (UID: \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" Nov 03 22:39:01.275629 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:01.275469 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-workload-certs\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442\" (UID: \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" Nov 03 22:39:01.275733 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:01.275684 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-istio-data\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442\" (UID: \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" Nov 03 22:39:01.275788 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:01.275750 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-credential-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442\" (UID: \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" Nov 03 22:39:01.275930 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:01.275899 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-workload-certs\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442\" (UID: \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" Nov 03 22:39:01.276096 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:01.276074 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-istiod-ca-cert\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442\" (UID: \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" Nov 03 22:39:01.277644 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:01.277626 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-istio-envoy\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442\" (UID: \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" Nov 03 22:39:01.278018 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:01.277999 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-istio-podinfo\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442\" (UID: \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" Nov 03 22:39:01.289883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:01.289831 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442"] Nov 03 22:39:01.358771 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:01.358737 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrb52\" (UniqueName: \"kubernetes.io/projected/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-kube-api-access-xrb52\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442\" (UID: \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" Nov 03 22:39:01.370102 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:01.370070 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-istio-token\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442\" (UID: \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" Nov 03 22:39:01.470486 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:01.470398 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" Nov 03 22:39:01.519272 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:01.519241 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:01.906061 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:01.905891 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442"] Nov 03 22:39:01.908491 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:39:01.908444 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50bb7d6c_e09f_424f_8e80_8dbbbe4b7bf7.slice/crio-963b3633e70d0af4ce28479d18fcff917575c42c134c05c67ae42f52b5e6e85f WatchSource:0}: Error finding container 963b3633e70d0af4ce28479d18fcff917575c42c134c05c67ae42f52b5e6e85f: Status 404 returned error can't find the container with id 963b3633e70d0af4ce28479d18fcff917575c42c134c05c67ae42f52b5e6e85f Nov 03 22:39:01.910700 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:01.910668 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:39:01.910758 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:01.910735 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:39:01.910789 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:01.910764 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:39:02.015641 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:02.015372 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" event={"ID":"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7","Type":"ContainerStarted","Data":"893a9a0e8febbc315aec0d20d348e68cee3c49de2eb97cc2a88ea1237da6bf29"} Nov 03 22:39:02.015641 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:02.015442 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" event={"ID":"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7","Type":"ContainerStarted","Data":"963b3633e70d0af4ce28479d18fcff917575c42c134c05c67ae42f52b5e6e85f"} Nov 03 22:39:02.164709 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:02.164611 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" podStartSLOduration=1.164595373 podStartE2EDuration="1.164595373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:39:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:39:02.162178838 +0000 UTC m=+971.471738130" watchObservedRunningTime="2026-11-03 22:39:02.164595373 +0000 UTC m=+971.474154666" Nov 03 22:39:02.471222 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:02.471136 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" Nov 03 22:39:02.472635 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:02.472611 5196 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 03 22:39:02.472760 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:02.472668 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" podUID="50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 03 22:39:03.471572 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:03.471537 5196 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 03 22:39:03.472073 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:03.471602 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" podUID="50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 03 22:39:04.247118 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:04.247083 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:04.471018 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:04.470981 5196 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 03 22:39:04.471206 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:04.471039 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" podUID="50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 03 22:39:05.471644 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:05.471610 5196 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 03 22:39:05.472155 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:05.471671 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" podUID="50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 03 22:39:06.160062 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.160031 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt"] Nov 03 22:39:06.163650 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.163628 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" Nov 03 22:39:06.249445 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.249396 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knmnh\" (UniqueName: \"kubernetes.io/projected/6b095b36-a492-414f-92bd-f1e3ac60fe7c-kube-api-access-knmnh\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wtfpt\" (UID: \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" Nov 03 22:39:06.249445 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.249447 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b095b36-a492-414f-92bd-f1e3ac60fe7c-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wtfpt\" (UID: \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" Nov 03 22:39:06.249659 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.249567 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b095b36-a492-414f-92bd-f1e3ac60fe7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wtfpt\" (UID: \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" Nov 03 22:39:06.249659 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.249608 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b095b36-a492-414f-92bd-f1e3ac60fe7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wtfpt\" (UID: \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" Nov 03 22:39:06.249659 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.249635 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b095b36-a492-414f-92bd-f1e3ac60fe7c-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wtfpt\" (UID: \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" Nov 03 22:39:06.249816 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.249745 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b095b36-a492-414f-92bd-f1e3ac60fe7c-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wtfpt\" (UID: \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" Nov 03 22:39:06.249905 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.249868 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b095b36-a492-414f-92bd-f1e3ac60fe7c-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wtfpt\" (UID: \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" Nov 03 22:39:06.249964 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.249923 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b095b36-a492-414f-92bd-f1e3ac60fe7c-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wtfpt\" (UID: \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" Nov 03 22:39:06.250019 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.249966 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b095b36-a492-414f-92bd-f1e3ac60fe7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wtfpt\" (UID: \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" Nov 03 22:39:06.351084 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.351043 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b095b36-a492-414f-92bd-f1e3ac60fe7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wtfpt\" (UID: \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" Nov 03 22:39:06.351084 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.351089 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b095b36-a492-414f-92bd-f1e3ac60fe7c-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wtfpt\" (UID: \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" Nov 03 22:39:06.351352 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.351118 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b095b36-a492-414f-92bd-f1e3ac60fe7c-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wtfpt\" (UID: \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" Nov 03 22:39:06.351352 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.351162 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b095b36-a492-414f-92bd-f1e3ac60fe7c-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wtfpt\" (UID: \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" Nov 03 22:39:06.351352 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.351199 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b095b36-a492-414f-92bd-f1e3ac60fe7c-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wtfpt\" (UID: \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" Nov 03 22:39:06.351352 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.351247 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b095b36-a492-414f-92bd-f1e3ac60fe7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wtfpt\" (UID: \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" Nov 03 22:39:06.351352 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.351330 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knmnh\" (UniqueName: \"kubernetes.io/projected/6b095b36-a492-414f-92bd-f1e3ac60fe7c-kube-api-access-knmnh\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wtfpt\" (UID: \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" Nov 03 22:39:06.351616 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.351365 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b095b36-a492-414f-92bd-f1e3ac60fe7c-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wtfpt\" (UID: \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" Nov 03 22:39:06.351616 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.351443 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b095b36-a492-414f-92bd-f1e3ac60fe7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wtfpt\" (UID: \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" Nov 03 22:39:06.351733 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.351613 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b095b36-a492-414f-92bd-f1e3ac60fe7c-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wtfpt\" (UID: \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" Nov 03 22:39:06.351733 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.351684 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b095b36-a492-414f-92bd-f1e3ac60fe7c-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wtfpt\" (UID: \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" Nov 03 22:39:06.351809 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.351760 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b095b36-a492-414f-92bd-f1e3ac60fe7c-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wtfpt\" (UID: \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" Nov 03 22:39:06.351988 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.351964 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b095b36-a492-414f-92bd-f1e3ac60fe7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wtfpt\" (UID: \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" Nov 03 22:39:06.352128 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.352107 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b095b36-a492-414f-92bd-f1e3ac60fe7c-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wtfpt\" (UID: \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" Nov 03 22:39:06.353886 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.353838 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b095b36-a492-414f-92bd-f1e3ac60fe7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wtfpt\" (UID: \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" Nov 03 22:39:06.353981 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.353960 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b095b36-a492-414f-92bd-f1e3ac60fe7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wtfpt\" (UID: \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" Nov 03 22:39:06.368964 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.368941 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt"] Nov 03 22:39:06.422113 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.422029 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knmnh\" (UniqueName: \"kubernetes.io/projected/6b095b36-a492-414f-92bd-f1e3ac60fe7c-kube-api-access-knmnh\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wtfpt\" (UID: \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" Nov 03 22:39:06.423709 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.423651 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b095b36-a492-414f-92bd-f1e3ac60fe7c-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wtfpt\" (UID: \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" Nov 03 22:39:06.470959 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.470924 5196 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 03 22:39:06.471122 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.470984 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" podUID="50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 03 22:39:06.477137 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.477111 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" Nov 03 22:39:06.628523 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.628462 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:06.939343 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.939299 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt"] Nov 03 22:39:06.943947 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:39:06.943916 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b095b36_a492_414f_92bd_f1e3ac60fe7c.slice/crio-52b1f70b27d4981ad7e551752ef4b0cb6abc534848cfb2c6c46a281757340dbf WatchSource:0}: Error finding container 52b1f70b27d4981ad7e551752ef4b0cb6abc534848cfb2c6c46a281757340dbf: Status 404 returned error can't find the container with id 52b1f70b27d4981ad7e551752ef4b0cb6abc534848cfb2c6c46a281757340dbf Nov 03 22:39:06.944508 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.944484 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:06.946345 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.946318 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:39:06.946410 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.946377 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:39:06.946448 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:06.946412 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:39:07.049427 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:07.049398 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" event={"ID":"6b095b36-a492-414f-92bd-f1e3ac60fe7c","Type":"ContainerStarted","Data":"52b1f70b27d4981ad7e551752ef4b0cb6abc534848cfb2c6c46a281757340dbf"} Nov 03 22:39:07.471735 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:07.471703 5196 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 03 22:39:07.471940 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:07.471762 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" podUID="50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 03 22:39:08.056521 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:08.056487 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" event={"ID":"6b095b36-a492-414f-92bd-f1e3ac60fe7c","Type":"ContainerStarted","Data":"64d89eb2bd656eaaf7f579f29005ef02bbb4509284853a0a31e5c27695f36ae1"} Nov 03 22:39:08.287021 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:08.286971 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" podStartSLOduration=2.286956804 podStartE2EDuration="2.286956804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:39:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:39:08.235872101 +0000 UTC m=+977.545431389" watchObservedRunningTime="2026-11-03 22:39:08.286956804 +0000 UTC m=+977.596516096" Nov 03 22:39:08.477736 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:08.477694 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" Nov 03 22:39:08.479081 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:08.479051 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wtfpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 03 22:39:08.479231 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:08.479113 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" podUID="6b095b36-a492-414f-92bd-f1e3ac60fe7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 03 22:39:08.718817 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:08.717053 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" Nov 03 22:39:08.718817 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:08.718152 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" Nov 03 22:39:08.719365 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:08.719342 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" Nov 03 22:39:09.144379 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:09.144342 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:09.477926 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:09.477828 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wtfpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 03 22:39:09.478072 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:09.477927 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" podUID="6b095b36-a492-414f-92bd-f1e3ac60fe7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 03 22:39:10.477898 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:10.477840 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wtfpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 03 22:39:10.478328 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:10.477918 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" podUID="6b095b36-a492-414f-92bd-f1e3ac60fe7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 03 22:39:11.478454 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:11.478423 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wtfpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 03 22:39:11.478845 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:11.478480 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" podUID="6b095b36-a492-414f-92bd-f1e3ac60fe7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 03 22:39:12.478303 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:12.478267 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wtfpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 03 22:39:12.478472 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:12.478318 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" podUID="6b095b36-a492-414f-92bd-f1e3ac60fe7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 03 22:39:13.478246 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:13.478213 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wtfpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 03 22:39:13.478425 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:13.478265 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" podUID="6b095b36-a492-414f-92bd-f1e3ac60fe7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 03 22:39:14.010967 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:14.010937 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:14.478073 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:14.478037 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wtfpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 03 22:39:14.478255 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:14.478105 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" podUID="6b095b36-a492-414f-92bd-f1e3ac60fe7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 03 22:39:15.481534 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:15.481506 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" Nov 03 22:39:15.482053 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:15.481947 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" Nov 03 22:39:15.482767 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:15.482746 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" Nov 03 22:39:17.321422 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.321385 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj"] Nov 03 22:39:17.325559 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.325539 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" Nov 03 22:39:17.379118 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.379093 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-7540da-openshift-default-dockercfg-862xt\"" Nov 03 22:39:17.397752 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.397720 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0aafab2-abe8-4dd8-81ea-71278482df2b-workload-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj\" (UID: \"b0aafab2-abe8-4dd8-81ea-71278482df2b\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" Nov 03 22:39:17.397908 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.397765 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0aafab2-abe8-4dd8-81ea-71278482df2b-credential-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj\" (UID: \"b0aafab2-abe8-4dd8-81ea-71278482df2b\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" Nov 03 22:39:17.397980 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.397959 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0aafab2-abe8-4dd8-81ea-71278482df2b-workload-certs\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj\" (UID: \"b0aafab2-abe8-4dd8-81ea-71278482df2b\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" Nov 03 22:39:17.398021 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.397994 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0aafab2-abe8-4dd8-81ea-71278482df2b-istiod-ca-cert\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj\" (UID: \"b0aafab2-abe8-4dd8-81ea-71278482df2b\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" Nov 03 22:39:17.398065 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.398053 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0aafab2-abe8-4dd8-81ea-71278482df2b-istio-podinfo\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj\" (UID: \"b0aafab2-abe8-4dd8-81ea-71278482df2b\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" Nov 03 22:39:17.398247 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.398226 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0aafab2-abe8-4dd8-81ea-71278482df2b-istio-token\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj\" (UID: \"b0aafab2-abe8-4dd8-81ea-71278482df2b\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" Nov 03 22:39:17.398312 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.398282 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0aafab2-abe8-4dd8-81ea-71278482df2b-istio-data\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj\" (UID: \"b0aafab2-abe8-4dd8-81ea-71278482df2b\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" Nov 03 22:39:17.398421 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.398401 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpx22\" (UniqueName: \"kubernetes.io/projected/b0aafab2-abe8-4dd8-81ea-71278482df2b-kube-api-access-rpx22\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj\" (UID: \"b0aafab2-abe8-4dd8-81ea-71278482df2b\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" Nov 03 22:39:17.398466 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.398443 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0aafab2-abe8-4dd8-81ea-71278482df2b-istio-envoy\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj\" (UID: \"b0aafab2-abe8-4dd8-81ea-71278482df2b\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" Nov 03 22:39:17.432143 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.432112 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj"] Nov 03 22:39:17.499177 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.499139 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0aafab2-abe8-4dd8-81ea-71278482df2b-workload-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj\" (UID: \"b0aafab2-abe8-4dd8-81ea-71278482df2b\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" Nov 03 22:39:17.499413 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.499182 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0aafab2-abe8-4dd8-81ea-71278482df2b-credential-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj\" (UID: \"b0aafab2-abe8-4dd8-81ea-71278482df2b\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" Nov 03 22:39:17.499413 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.499258 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0aafab2-abe8-4dd8-81ea-71278482df2b-workload-certs\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj\" (UID: \"b0aafab2-abe8-4dd8-81ea-71278482df2b\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" Nov 03 22:39:17.499413 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.499282 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0aafab2-abe8-4dd8-81ea-71278482df2b-istiod-ca-cert\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj\" (UID: \"b0aafab2-abe8-4dd8-81ea-71278482df2b\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" Nov 03 22:39:17.499413 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.499329 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0aafab2-abe8-4dd8-81ea-71278482df2b-istio-podinfo\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj\" (UID: \"b0aafab2-abe8-4dd8-81ea-71278482df2b\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" Nov 03 22:39:17.499413 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.499408 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0aafab2-abe8-4dd8-81ea-71278482df2b-istio-token\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj\" (UID: \"b0aafab2-abe8-4dd8-81ea-71278482df2b\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" Nov 03 22:39:17.499693 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.499452 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0aafab2-abe8-4dd8-81ea-71278482df2b-istio-data\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj\" (UID: \"b0aafab2-abe8-4dd8-81ea-71278482df2b\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" Nov 03 22:39:17.499693 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.499491 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpx22\" (UniqueName: \"kubernetes.io/projected/b0aafab2-abe8-4dd8-81ea-71278482df2b-kube-api-access-rpx22\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj\" (UID: \"b0aafab2-abe8-4dd8-81ea-71278482df2b\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" Nov 03 22:39:17.499693 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.499529 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0aafab2-abe8-4dd8-81ea-71278482df2b-istio-envoy\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj\" (UID: \"b0aafab2-abe8-4dd8-81ea-71278482df2b\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" Nov 03 22:39:17.499693 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.499590 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0aafab2-abe8-4dd8-81ea-71278482df2b-workload-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj\" (UID: \"b0aafab2-abe8-4dd8-81ea-71278482df2b\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" Nov 03 22:39:17.499905 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.499691 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0aafab2-abe8-4dd8-81ea-71278482df2b-credential-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj\" (UID: \"b0aafab2-abe8-4dd8-81ea-71278482df2b\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" Nov 03 22:39:17.499905 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.499783 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0aafab2-abe8-4dd8-81ea-71278482df2b-workload-certs\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj\" (UID: \"b0aafab2-abe8-4dd8-81ea-71278482df2b\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" Nov 03 22:39:17.500118 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.500089 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0aafab2-abe8-4dd8-81ea-71278482df2b-istio-data\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj\" (UID: \"b0aafab2-abe8-4dd8-81ea-71278482df2b\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" Nov 03 22:39:17.500281 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.500260 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0aafab2-abe8-4dd8-81ea-71278482df2b-istiod-ca-cert\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj\" (UID: \"b0aafab2-abe8-4dd8-81ea-71278482df2b\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" Nov 03 22:39:17.502008 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.501982 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0aafab2-abe8-4dd8-81ea-71278482df2b-istio-envoy\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj\" (UID: \"b0aafab2-abe8-4dd8-81ea-71278482df2b\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" Nov 03 22:39:17.502207 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.502188 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0aafab2-abe8-4dd8-81ea-71278482df2b-istio-podinfo\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj\" (UID: \"b0aafab2-abe8-4dd8-81ea-71278482df2b\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" Nov 03 22:39:17.572993 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.572888 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0aafab2-abe8-4dd8-81ea-71278482df2b-istio-token\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj\" (UID: \"b0aafab2-abe8-4dd8-81ea-71278482df2b\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" Nov 03 22:39:17.574160 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.574137 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpx22\" (UniqueName: \"kubernetes.io/projected/b0aafab2-abe8-4dd8-81ea-71278482df2b-kube-api-access-rpx22\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj\" (UID: \"b0aafab2-abe8-4dd8-81ea-71278482df2b\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" Nov 03 22:39:17.640162 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.640131 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" Nov 03 22:39:17.669732 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.669704 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v"] Nov 03 22:39:17.674736 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.674715 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" Nov 03 22:39:17.739333 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.736000 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v"] Nov 03 22:39:17.803064 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.803028 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ca4784-6fe2-4001-ac15-25675fc41a02-istio-podinfo\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v\" (UID: \"f0ca4784-6fe2-4001-ac15-25675fc41a02\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" Nov 03 22:39:17.803064 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.803068 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ca4784-6fe2-4001-ac15-25675fc41a02-credential-socket\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v\" (UID: \"f0ca4784-6fe2-4001-ac15-25675fc41a02\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" Nov 03 22:39:17.803263 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.803087 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ca4784-6fe2-4001-ac15-25675fc41a02-workload-certs\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v\" (UID: \"f0ca4784-6fe2-4001-ac15-25675fc41a02\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" Nov 03 22:39:17.803263 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.803177 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0ca4784-6fe2-4001-ac15-25675fc41a02-istio-token\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v\" (UID: \"f0ca4784-6fe2-4001-ac15-25675fc41a02\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" Nov 03 22:39:17.803354 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.803340 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpmrb\" (UniqueName: \"kubernetes.io/projected/f0ca4784-6fe2-4001-ac15-25675fc41a02-kube-api-access-hpmrb\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v\" (UID: \"f0ca4784-6fe2-4001-ac15-25675fc41a02\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" Nov 03 22:39:17.803403 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.803390 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ca4784-6fe2-4001-ac15-25675fc41a02-istio-data\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v\" (UID: \"f0ca4784-6fe2-4001-ac15-25675fc41a02\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" Nov 03 22:39:17.803440 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.803412 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ca4784-6fe2-4001-ac15-25675fc41a02-istiod-ca-cert\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v\" (UID: \"f0ca4784-6fe2-4001-ac15-25675fc41a02\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" Nov 03 22:39:17.803508 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.803497 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ca4784-6fe2-4001-ac15-25675fc41a02-istio-envoy\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v\" (UID: \"f0ca4784-6fe2-4001-ac15-25675fc41a02\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" Nov 03 22:39:17.803542 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.803523 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ca4784-6fe2-4001-ac15-25675fc41a02-workload-socket\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v\" (UID: \"f0ca4784-6fe2-4001-ac15-25675fc41a02\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" Nov 03 22:39:17.904381 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.904350 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ca4784-6fe2-4001-ac15-25675fc41a02-istio-podinfo\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v\" (UID: \"f0ca4784-6fe2-4001-ac15-25675fc41a02\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" Nov 03 22:39:17.904583 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.904435 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ca4784-6fe2-4001-ac15-25675fc41a02-credential-socket\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v\" (UID: \"f0ca4784-6fe2-4001-ac15-25675fc41a02\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" Nov 03 22:39:17.904583 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.904461 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ca4784-6fe2-4001-ac15-25675fc41a02-workload-certs\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v\" (UID: \"f0ca4784-6fe2-4001-ac15-25675fc41a02\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" Nov 03 22:39:17.904583 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.904508 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0ca4784-6fe2-4001-ac15-25675fc41a02-istio-token\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v\" (UID: \"f0ca4784-6fe2-4001-ac15-25675fc41a02\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" Nov 03 22:39:17.904769 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.904620 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpmrb\" (UniqueName: \"kubernetes.io/projected/f0ca4784-6fe2-4001-ac15-25675fc41a02-kube-api-access-hpmrb\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v\" (UID: \"f0ca4784-6fe2-4001-ac15-25675fc41a02\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" Nov 03 22:39:17.904769 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.904686 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ca4784-6fe2-4001-ac15-25675fc41a02-istio-data\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v\" (UID: \"f0ca4784-6fe2-4001-ac15-25675fc41a02\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" Nov 03 22:39:17.904769 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.904712 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ca4784-6fe2-4001-ac15-25675fc41a02-istiod-ca-cert\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v\" (UID: \"f0ca4784-6fe2-4001-ac15-25675fc41a02\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" Nov 03 22:39:17.904998 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.904769 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ca4784-6fe2-4001-ac15-25675fc41a02-istio-envoy\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v\" (UID: \"f0ca4784-6fe2-4001-ac15-25675fc41a02\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" Nov 03 22:39:17.904998 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.904805 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ca4784-6fe2-4001-ac15-25675fc41a02-workload-socket\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v\" (UID: \"f0ca4784-6fe2-4001-ac15-25675fc41a02\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" Nov 03 22:39:17.904998 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.904908 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ca4784-6fe2-4001-ac15-25675fc41a02-credential-socket\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v\" (UID: \"f0ca4784-6fe2-4001-ac15-25675fc41a02\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" Nov 03 22:39:17.905603 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.905574 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ca4784-6fe2-4001-ac15-25675fc41a02-workload-certs\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v\" (UID: \"f0ca4784-6fe2-4001-ac15-25675fc41a02\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" Nov 03 22:39:17.905715 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.905623 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ca4784-6fe2-4001-ac15-25675fc41a02-istio-data\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v\" (UID: \"f0ca4784-6fe2-4001-ac15-25675fc41a02\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" Nov 03 22:39:17.905774 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.905724 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ca4784-6fe2-4001-ac15-25675fc41a02-workload-socket\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v\" (UID: \"f0ca4784-6fe2-4001-ac15-25675fc41a02\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" Nov 03 22:39:17.905934 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.905914 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ca4784-6fe2-4001-ac15-25675fc41a02-istiod-ca-cert\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v\" (UID: \"f0ca4784-6fe2-4001-ac15-25675fc41a02\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" Nov 03 22:39:17.907313 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.907284 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ca4784-6fe2-4001-ac15-25675fc41a02-istio-podinfo\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v\" (UID: \"f0ca4784-6fe2-4001-ac15-25675fc41a02\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" Nov 03 22:39:17.907406 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.907364 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ca4784-6fe2-4001-ac15-25675fc41a02-istio-envoy\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v\" (UID: \"f0ca4784-6fe2-4001-ac15-25675fc41a02\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" Nov 03 22:39:17.967575 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.967548 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0ca4784-6fe2-4001-ac15-25675fc41a02-istio-token\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v\" (UID: \"f0ca4784-6fe2-4001-ac15-25675fc41a02\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" Nov 03 22:39:17.967699 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.967681 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpmrb\" (UniqueName: \"kubernetes.io/projected/f0ca4784-6fe2-4001-ac15-25675fc41a02-kube-api-access-hpmrb\") pod \"e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v\" (UID: \"f0ca4784-6fe2-4001-ac15-25675fc41a02\") " pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" Nov 03 22:39:17.991322 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:17.991295 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" Nov 03 22:39:18.085964 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:18.084018 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj"] Nov 03 22:39:18.093956 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:18.093717 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:39:18.093956 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:18.093822 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:39:18.096263 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:18.094291 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:39:18.136348 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:18.136071 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" event={"ID":"b0aafab2-abe8-4dd8-81ea-71278482df2b","Type":"ContainerStarted","Data":"101c2806e8da948f7e83a9e4eca04fe0b31f172ffb739db820d609ee0b3f96bb"} Nov 03 22:39:18.400130 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:18.398969 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v"] Nov 03 22:39:18.403622 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:39:18.403587 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0ca4784_6fe2_4001_ac15_25675fc41a02.slice/crio-3df1b738af557f12b6097649bc0fe68c76ffe4f2a9b1721440d1811d5e853244 WatchSource:0}: Error finding container 3df1b738af557f12b6097649bc0fe68c76ffe4f2a9b1721440d1811d5e853244: Status 404 returned error can't find the container with id 3df1b738af557f12b6097649bc0fe68c76ffe4f2a9b1721440d1811d5e853244 Nov 03 22:39:18.406111 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:18.406072 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:39:18.406207 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:18.406178 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:39:18.406248 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:18.406226 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:39:18.981879 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:18.978106 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:19.175883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:19.169720 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" event={"ID":"f0ca4784-6fe2-4001-ac15-25675fc41a02","Type":"ContainerStarted","Data":"4b8f9e503d3700c74c892a7657e38946c9882fbfe1d07d5054b5c29be7bc4a55"} Nov 03 22:39:19.175883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:19.169902 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" event={"ID":"f0ca4784-6fe2-4001-ac15-25675fc41a02","Type":"ContainerStarted","Data":"3df1b738af557f12b6097649bc0fe68c76ffe4f2a9b1721440d1811d5e853244"} Nov 03 22:39:19.198881 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:19.188372 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" event={"ID":"b0aafab2-abe8-4dd8-81ea-71278482df2b","Type":"ContainerStarted","Data":"2103d3f8cb26b75106b0ad40d3b841a5c8c60ba6adf5bd50917c1b9551ed594b"} Nov 03 22:39:19.442637 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:19.442572 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" podStartSLOduration=2.442550766 podStartE2EDuration="2.442550766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:39:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:39:19.319201716 +0000 UTC m=+988.628761026" watchObservedRunningTime="2026-11-03 22:39:19.442550766 +0000 UTC m=+988.752110061" Nov 03 22:39:19.450881 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:19.445117 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" podStartSLOduration=2.445100701 podStartE2EDuration="2.445100701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:39:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:39:19.441526214 +0000 UTC m=+988.751085509" watchObservedRunningTime="2026-11-03 22:39:19.445100701 +0000 UTC m=+988.754659995" Nov 03 22:39:19.643248 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:19.643216 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" Nov 03 22:39:19.656375 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:19.656335 5196 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 03 22:39:19.656642 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:19.656615 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" podUID="b0aafab2-abe8-4dd8-81ea-71278482df2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 03 22:39:19.997158 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:19.997102 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" Nov 03 22:39:19.997370 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:19.997240 5196 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 03 22:39:19.997370 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:19.997301 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" podUID="f0ca4784-6fe2-4001-ac15-25675fc41a02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 03 22:39:20.245912 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:20.239969 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" podUID="34e9cf3a-3456-4835-8933-102ce735cccc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:39:20.651883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:20.650984 5196 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 03 22:39:20.651883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:20.651053 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" podUID="b0aafab2-abe8-4dd8-81ea-71278482df2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 03 22:39:20.997961 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:20.994219 5196 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 03 22:39:20.997961 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:20.994279 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" podUID="f0ca4784-6fe2-4001-ac15-25675fc41a02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 03 22:39:21.640683 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:21.640631 5196 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 03 22:39:21.640927 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:21.640697 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" podUID="b0aafab2-abe8-4dd8-81ea-71278482df2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 03 22:39:22.003321 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:22.003232 5196 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 03 22:39:22.003905 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:22.003849 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" podUID="f0ca4784-6fe2-4001-ac15-25675fc41a02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 03 22:39:22.640983 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:22.640939 5196 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 03 22:39:22.641200 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:22.641035 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" podUID="b0aafab2-abe8-4dd8-81ea-71278482df2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 03 22:39:22.994941 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:22.994196 5196 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 03 22:39:22.994941 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:22.994257 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" podUID="f0ca4784-6fe2-4001-ac15-25675fc41a02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 03 22:39:23.641903 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:23.641393 5196 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 03 22:39:23.641903 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:23.641454 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" podUID="b0aafab2-abe8-4dd8-81ea-71278482df2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 03 22:39:24.001971 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:23.994224 5196 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 03 22:39:24.001971 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:23.994299 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" podUID="f0ca4784-6fe2-4001-ac15-25675fc41a02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 03 22:39:24.302015 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:24.301902 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:24.642910 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:24.642221 5196 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 03 22:39:24.642910 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:24.642278 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" podUID="b0aafab2-abe8-4dd8-81ea-71278482df2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 03 22:39:24.996940 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:24.994220 5196 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 03 22:39:24.996940 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:24.994278 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" podUID="f0ca4784-6fe2-4001-ac15-25675fc41a02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 03 22:39:25.641386 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:25.641343 5196 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 03 22:39:25.641570 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:25.641419 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" podUID="b0aafab2-abe8-4dd8-81ea-71278482df2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 03 22:39:25.928090 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:39:25.927993 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:39:25.992226 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:25.992187 5196 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 03 22:39:25.992396 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:25.992254 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" podUID="f0ca4784-6fe2-4001-ac15-25675fc41a02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 03 22:39:26.971483 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:26.971449 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" Nov 03 22:39:26.975903 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:26.972604 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" Nov 03 22:39:26.976393 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:26.976367 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" Nov 03 22:39:26.995992 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:26.995968 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" Nov 03 22:39:26.996265 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:26.996240 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" Nov 03 22:39:26.997035 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:26.997021 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" Nov 03 22:39:29.088837 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:29.088806 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:34.003144 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:34.003113 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:38.953054 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:38.953020 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:43.664099 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:43.664036 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:48.398488 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:48.398454 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:53.268214 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:53.268180 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:55.917559 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:39:55.917523 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:39:57.780425 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:57.780394 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:39:58.005572 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:39:58.005537 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:01.869447 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:01.869409 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:02.801049 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:02.801015 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:06.947275 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:06.947244 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:07.520822 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:07.520794 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:12.407187 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:12.407157 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:16.309053 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:16.309016 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:17.201723 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:17.201693 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:20.361991 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:20.361939 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:21.997237 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:21.997195 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:25.921698 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:40:25.921660 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:40:26.725962 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:26.725930 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:27.335677 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:27.335635 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn"] Nov 03 22:40:27.339845 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:27.339828 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" Nov 03 22:40:27.414742 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:27.414708 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f2e3f52-0664-434f-919b-32736cbf0602-home\") pod \"distinct-w5-cbd26c-kserve-559694f4c6-tdkzn\" (UID: \"9f2e3f52-0664-434f-919b-32736cbf0602\") " pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" Nov 03 22:40:27.414742 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:27.414743 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f2e3f52-0664-434f-919b-32736cbf0602-model-cache\") pod \"distinct-w5-cbd26c-kserve-559694f4c6-tdkzn\" (UID: \"9f2e3f52-0664-434f-919b-32736cbf0602\") " pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" Nov 03 22:40:27.414980 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:27.414947 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f2e3f52-0664-434f-919b-32736cbf0602-tmp-dir\") pod \"distinct-w5-cbd26c-kserve-559694f4c6-tdkzn\" (UID: \"9f2e3f52-0664-434f-919b-32736cbf0602\") " pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" Nov 03 22:40:27.415098 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:27.415078 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f2e3f52-0664-434f-919b-32736cbf0602-tls-certs\") pod \"distinct-w5-cbd26c-kserve-559694f4c6-tdkzn\" (UID: \"9f2e3f52-0664-434f-919b-32736cbf0602\") " pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" Nov 03 22:40:27.415158 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:27.415143 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcpwb\" (UniqueName: \"kubernetes.io/projected/9f2e3f52-0664-434f-919b-32736cbf0602-kube-api-access-xcpwb\") pod \"distinct-w5-cbd26c-kserve-559694f4c6-tdkzn\" (UID: \"9f2e3f52-0664-434f-919b-32736cbf0602\") " pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" Nov 03 22:40:27.415243 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:27.415227 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f2e3f52-0664-434f-919b-32736cbf0602-dshm\") pod \"distinct-w5-cbd26c-kserve-559694f4c6-tdkzn\" (UID: \"9f2e3f52-0664-434f-919b-32736cbf0602\") " pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" Nov 03 22:40:27.422833 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:27.422797 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:27.444114 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:27.444091 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-cbd26c\"/\"distinct-w5-cbd26c-kserve-self-signed-certs\"" Nov 03 22:40:27.446416 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:27.446398 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn"] Nov 03 22:40:27.516669 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:27.516641 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f2e3f52-0664-434f-919b-32736cbf0602-dshm\") pod \"distinct-w5-cbd26c-kserve-559694f4c6-tdkzn\" (UID: \"9f2e3f52-0664-434f-919b-32736cbf0602\") " pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" Nov 03 22:40:27.516846 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:27.516690 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f2e3f52-0664-434f-919b-32736cbf0602-home\") pod \"distinct-w5-cbd26c-kserve-559694f4c6-tdkzn\" (UID: \"9f2e3f52-0664-434f-919b-32736cbf0602\") " pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" Nov 03 22:40:27.516846 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:27.516719 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f2e3f52-0664-434f-919b-32736cbf0602-model-cache\") pod \"distinct-w5-cbd26c-kserve-559694f4c6-tdkzn\" (UID: \"9f2e3f52-0664-434f-919b-32736cbf0602\") " pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" Nov 03 22:40:27.516846 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:27.516757 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f2e3f52-0664-434f-919b-32736cbf0602-tmp-dir\") pod \"distinct-w5-cbd26c-kserve-559694f4c6-tdkzn\" (UID: \"9f2e3f52-0664-434f-919b-32736cbf0602\") " pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" Nov 03 22:40:27.516846 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:27.516807 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f2e3f52-0664-434f-919b-32736cbf0602-tls-certs\") pod \"distinct-w5-cbd26c-kserve-559694f4c6-tdkzn\" (UID: \"9f2e3f52-0664-434f-919b-32736cbf0602\") " pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" Nov 03 22:40:27.516846 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:27.516833 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcpwb\" (UniqueName: \"kubernetes.io/projected/9f2e3f52-0664-434f-919b-32736cbf0602-kube-api-access-xcpwb\") pod \"distinct-w5-cbd26c-kserve-559694f4c6-tdkzn\" (UID: \"9f2e3f52-0664-434f-919b-32736cbf0602\") " pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" Nov 03 22:40:27.517487 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:27.517163 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f2e3f52-0664-434f-919b-32736cbf0602-model-cache\") pod \"distinct-w5-cbd26c-kserve-559694f4c6-tdkzn\" (UID: \"9f2e3f52-0664-434f-919b-32736cbf0602\") " pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" Nov 03 22:40:27.517487 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:27.517425 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f2e3f52-0664-434f-919b-32736cbf0602-tmp-dir\") pod \"distinct-w5-cbd26c-kserve-559694f4c6-tdkzn\" (UID: \"9f2e3f52-0664-434f-919b-32736cbf0602\") " pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" Nov 03 22:40:27.517641 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:27.517622 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f2e3f52-0664-434f-919b-32736cbf0602-home\") pod \"distinct-w5-cbd26c-kserve-559694f4c6-tdkzn\" (UID: \"9f2e3f52-0664-434f-919b-32736cbf0602\") " pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" Nov 03 22:40:27.519209 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:27.519184 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f2e3f52-0664-434f-919b-32736cbf0602-dshm\") pod \"distinct-w5-cbd26c-kserve-559694f4c6-tdkzn\" (UID: \"9f2e3f52-0664-434f-919b-32736cbf0602\") " pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" Nov 03 22:40:27.519537 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:27.519517 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f2e3f52-0664-434f-919b-32736cbf0602-tls-certs\") pod \"distinct-w5-cbd26c-kserve-559694f4c6-tdkzn\" (UID: \"9f2e3f52-0664-434f-919b-32736cbf0602\") " pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" Nov 03 22:40:27.579346 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:27.579310 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcpwb\" (UniqueName: \"kubernetes.io/projected/9f2e3f52-0664-434f-919b-32736cbf0602-kube-api-access-xcpwb\") pod \"distinct-w5-cbd26c-kserve-559694f4c6-tdkzn\" (UID: \"9f2e3f52-0664-434f-919b-32736cbf0602\") " pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" Nov 03 22:40:27.652489 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:27.652456 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" Nov 03 22:40:28.068397 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:28.068369 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn"] Nov 03 22:40:28.069720 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:40:28.069695 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f2e3f52_0664_434f_919b_32736cbf0602.slice/crio-0fc50d0fedce4b471a31a37d16b46d4c015f22b4b128c840a94359f5c8a59a1a WatchSource:0}: Error finding container 0fc50d0fedce4b471a31a37d16b46d4c015f22b4b128c840a94359f5c8a59a1a: Status 404 returned error can't find the container with id 0fc50d0fedce4b471a31a37d16b46d4c015f22b4b128c840a94359f5c8a59a1a Nov 03 22:40:28.696433 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:28.696394 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" event={"ID":"9f2e3f52-0664-434f-919b-32736cbf0602","Type":"ContainerStarted","Data":"14daded1822daa9e551f636f3236518944f047f327eca57cc49833e29b38b5e0"} Nov 03 22:40:28.696901 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:28.696441 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" event={"ID":"9f2e3f52-0664-434f-919b-32736cbf0602","Type":"ContainerStarted","Data":"0fc50d0fedce4b471a31a37d16b46d4c015f22b4b128c840a94359f5c8a59a1a"} Nov 03 22:40:28.818321 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:28.818268 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" podStartSLOduration=1.818256414 podStartE2EDuration="1.818256414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:40:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:40:28.818137797 +0000 UTC m=+1058.127697091" watchObservedRunningTime="2026-11-03 22:40:28.818256414 +0000 UTC m=+1058.127815706" Nov 03 22:40:29.653613 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:29.653578 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" Nov 03 22:40:31.502275 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:31.502212 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:31.531716 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:31.531688 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p"] Nov 03 22:40:31.535720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:31.535699 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" Nov 03 22:40:31.637061 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:31.637033 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-1d31f8\"/\"kube-root-ca.crt\"" Nov 03 22:40:31.642485 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:31.642460 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-1d31f8\"/\"openshift-service-ca.crt\"" Nov 03 22:40:31.642662 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:31.642607 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-1d31f8\"/\"distinct-w3-1d31f8-kserve-self-signed-certs\"" Nov 03 22:40:31.642747 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:31.642675 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-1d31f8\"/\"default-dockercfg-l54sw\"" Nov 03 22:40:31.645026 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:31.645000 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p"] Nov 03 22:40:31.645852 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:31.645829 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:31.664904 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:31.664881 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-tmp-dir\") pod \"distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p\" (UID: \"26fa51c2-0f2e-482f-9c67-0ea7924d7b3c\") " pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" Nov 03 22:40:31.665030 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:31.664916 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-dshm\") pod \"distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p\" (UID: \"26fa51c2-0f2e-482f-9c67-0ea7924d7b3c\") " pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" Nov 03 22:40:31.665030 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:31.664948 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-home\") pod \"distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p\" (UID: \"26fa51c2-0f2e-482f-9c67-0ea7924d7b3c\") " pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" Nov 03 22:40:31.665136 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:31.665111 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-tls-certs\") pod \"distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p\" (UID: \"26fa51c2-0f2e-482f-9c67-0ea7924d7b3c\") " pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" Nov 03 22:40:31.665189 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:31.665166 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-model-cache\") pod \"distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p\" (UID: \"26fa51c2-0f2e-482f-9c67-0ea7924d7b3c\") " pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" Nov 03 22:40:31.665316 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:31.665290 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbfnl\" (UniqueName: \"kubernetes.io/projected/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-kube-api-access-fbfnl\") pod \"distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p\" (UID: \"26fa51c2-0f2e-482f-9c67-0ea7924d7b3c\") " pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" Nov 03 22:40:31.766807 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:31.766702 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-home\") pod \"distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p\" (UID: \"26fa51c2-0f2e-482f-9c67-0ea7924d7b3c\") " pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" Nov 03 22:40:31.766807 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:31.766759 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-tls-certs\") pod \"distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p\" (UID: \"26fa51c2-0f2e-482f-9c67-0ea7924d7b3c\") " pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" Nov 03 22:40:31.766807 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:31.766787 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-model-cache\") pod \"distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p\" (UID: \"26fa51c2-0f2e-482f-9c67-0ea7924d7b3c\") " pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" Nov 03 22:40:31.767145 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:31.766829 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbfnl\" (UniqueName: \"kubernetes.io/projected/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-kube-api-access-fbfnl\") pod \"distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p\" (UID: \"26fa51c2-0f2e-482f-9c67-0ea7924d7b3c\") " pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" Nov 03 22:40:31.767145 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:31.766922 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-tmp-dir\") pod \"distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p\" (UID: \"26fa51c2-0f2e-482f-9c67-0ea7924d7b3c\") " pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" Nov 03 22:40:31.767145 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:31.766943 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-dshm\") pod \"distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p\" (UID: \"26fa51c2-0f2e-482f-9c67-0ea7924d7b3c\") " pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" Nov 03 22:40:31.767320 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:31.767190 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-home\") pod \"distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p\" (UID: \"26fa51c2-0f2e-482f-9c67-0ea7924d7b3c\") " pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" Nov 03 22:40:31.767320 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:31.767208 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-model-cache\") pod \"distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p\" (UID: \"26fa51c2-0f2e-482f-9c67-0ea7924d7b3c\") " pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" Nov 03 22:40:31.767320 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:31.767269 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-tmp-dir\") pod \"distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p\" (UID: \"26fa51c2-0f2e-482f-9c67-0ea7924d7b3c\") " pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" Nov 03 22:40:31.769318 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:31.769293 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-dshm\") pod \"distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p\" (UID: \"26fa51c2-0f2e-482f-9c67-0ea7924d7b3c\") " pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" Nov 03 22:40:31.769482 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:31.769465 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-tls-certs\") pod \"distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p\" (UID: \"26fa51c2-0f2e-482f-9c67-0ea7924d7b3c\") " pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" Nov 03 22:40:31.828940 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:31.828912 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbfnl\" (UniqueName: \"kubernetes.io/projected/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-kube-api-access-fbfnl\") pod \"distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p\" (UID: \"26fa51c2-0f2e-482f-9c67-0ea7924d7b3c\") " pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" Nov 03 22:40:31.855306 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:31.855277 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" Nov 03 22:40:32.266829 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:32.266792 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p"] Nov 03 22:40:32.268290 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:40:32.268263 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26fa51c2_0f2e_482f_9c67_0ea7924d7b3c.slice/crio-6bf83f66d5585d5ec2677d8911e1b1f08b4a5fe6fdc57ffa0d11661431e5f371 WatchSource:0}: Error finding container 6bf83f66d5585d5ec2677d8911e1b1f08b4a5fe6fdc57ffa0d11661431e5f371: Status 404 returned error can't find the container with id 6bf83f66d5585d5ec2677d8911e1b1f08b4a5fe6fdc57ffa0d11661431e5f371 Nov 03 22:40:32.729573 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:32.729529 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" event={"ID":"26fa51c2-0f2e-482f-9c67-0ea7924d7b3c","Type":"ContainerStarted","Data":"04a6c30c79f2ec1112aa05316ce756ef91d51fa48ad93415c09b03f8c1f0778a"} Nov 03 22:40:32.729573 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:32.729576 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" event={"ID":"26fa51c2-0f2e-482f-9c67-0ea7924d7b3c","Type":"ContainerStarted","Data":"6bf83f66d5585d5ec2677d8911e1b1f08b4a5fe6fdc57ffa0d11661431e5f371"} Nov 03 22:40:32.849566 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:32.849508 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" podStartSLOduration=1.849494568 podStartE2EDuration="1.849494568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:40:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:40:32.847367436 +0000 UTC m=+1062.156926728" watchObservedRunningTime="2026-11-03 22:40:32.849494568 +0000 UTC m=+1062.159053896" Nov 03 22:40:32.855782 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:32.855758 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" Nov 03 22:40:36.218639 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:36.218598 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:37.653328 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:37.653280 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" Nov 03 22:40:37.673707 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:37.673678 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" Nov 03 22:40:37.783540 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:37.783501 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" Nov 03 22:40:41.012152 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:41.012122 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:41.855917 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:41.855888 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" Nov 03 22:40:41.868185 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:41.868164 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" Nov 03 22:40:42.535007 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:42.534967 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:42.860132 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:42.860099 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" Nov 03 22:40:45.807975 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:45.807945 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:48.558430 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:48.558399 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk"] Nov 03 22:40:48.562173 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:48.562158 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" Nov 03 22:40:48.595644 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:48.595617 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03244edf-87f5-4237-be87-8a523c4d96b3-istio-data\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk\" (UID: \"03244edf-87f5-4237-be87-8a523c4d96b3\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" Nov 03 22:40:48.595774 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:48.595691 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03244edf-87f5-4237-be87-8a523c4d96b3-istio-podinfo\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk\" (UID: \"03244edf-87f5-4237-be87-8a523c4d96b3\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" Nov 03 22:40:48.595774 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:48.595717 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03244edf-87f5-4237-be87-8a523c4d96b3-istiod-ca-cert\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk\" (UID: \"03244edf-87f5-4237-be87-8a523c4d96b3\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" Nov 03 22:40:48.595897 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:48.595820 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03244edf-87f5-4237-be87-8a523c4d96b3-credential-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk\" (UID: \"03244edf-87f5-4237-be87-8a523c4d96b3\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" Nov 03 22:40:48.595944 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:48.595924 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03244edf-87f5-4237-be87-8a523c4d96b3-workload-certs\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk\" (UID: \"03244edf-87f5-4237-be87-8a523c4d96b3\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" Nov 03 22:40:48.595986 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:48.595946 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03244edf-87f5-4237-be87-8a523c4d96b3-istio-envoy\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk\" (UID: \"03244edf-87f5-4237-be87-8a523c4d96b3\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" Nov 03 22:40:48.595986 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:48.595971 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03244edf-87f5-4237-be87-8a523c4d96b3-istio-token\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk\" (UID: \"03244edf-87f5-4237-be87-8a523c4d96b3\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" Nov 03 22:40:48.596090 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:48.596027 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03244edf-87f5-4237-be87-8a523c4d96b3-workload-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk\" (UID: \"03244edf-87f5-4237-be87-8a523c4d96b3\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" Nov 03 22:40:48.596143 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:48.596100 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd76q\" (UniqueName: \"kubernetes.io/projected/03244edf-87f5-4237-be87-8a523c4d96b3-kube-api-access-fd76q\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk\" (UID: \"03244edf-87f5-4237-be87-8a523c4d96b3\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" Nov 03 22:40:48.615550 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:48.615528 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-cbd26c-openshift-default-dockercfg-5rskr\"" Nov 03 22:40:48.669902 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:48.669850 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk"] Nov 03 22:40:48.697221 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:48.697191 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03244edf-87f5-4237-be87-8a523c4d96b3-istio-podinfo\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk\" (UID: \"03244edf-87f5-4237-be87-8a523c4d96b3\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" Nov 03 22:40:48.697221 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:48.697222 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03244edf-87f5-4237-be87-8a523c4d96b3-istiod-ca-cert\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk\" (UID: \"03244edf-87f5-4237-be87-8a523c4d96b3\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" Nov 03 22:40:48.697513 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:48.697273 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03244edf-87f5-4237-be87-8a523c4d96b3-credential-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk\" (UID: \"03244edf-87f5-4237-be87-8a523c4d96b3\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" Nov 03 22:40:48.697513 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:48.697295 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03244edf-87f5-4237-be87-8a523c4d96b3-workload-certs\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk\" (UID: \"03244edf-87f5-4237-be87-8a523c4d96b3\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" Nov 03 22:40:48.697513 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:48.697312 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03244edf-87f5-4237-be87-8a523c4d96b3-istio-envoy\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk\" (UID: \"03244edf-87f5-4237-be87-8a523c4d96b3\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" Nov 03 22:40:48.697513 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:48.697336 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03244edf-87f5-4237-be87-8a523c4d96b3-istio-token\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk\" (UID: \"03244edf-87f5-4237-be87-8a523c4d96b3\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" Nov 03 22:40:48.697513 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:48.697360 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03244edf-87f5-4237-be87-8a523c4d96b3-workload-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk\" (UID: \"03244edf-87f5-4237-be87-8a523c4d96b3\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" Nov 03 22:40:48.697513 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:48.697403 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd76q\" (UniqueName: \"kubernetes.io/projected/03244edf-87f5-4237-be87-8a523c4d96b3-kube-api-access-fd76q\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk\" (UID: \"03244edf-87f5-4237-be87-8a523c4d96b3\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" Nov 03 22:40:48.697513 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:48.697453 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03244edf-87f5-4237-be87-8a523c4d96b3-istio-data\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk\" (UID: \"03244edf-87f5-4237-be87-8a523c4d96b3\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" Nov 03 22:40:48.697925 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:48.697757 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03244edf-87f5-4237-be87-8a523c4d96b3-credential-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk\" (UID: \"03244edf-87f5-4237-be87-8a523c4d96b3\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" Nov 03 22:40:48.697925 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:48.697834 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03244edf-87f5-4237-be87-8a523c4d96b3-istio-data\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk\" (UID: \"03244edf-87f5-4237-be87-8a523c4d96b3\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" Nov 03 22:40:48.698130 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:48.698054 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03244edf-87f5-4237-be87-8a523c4d96b3-workload-certs\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk\" (UID: \"03244edf-87f5-4237-be87-8a523c4d96b3\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" Nov 03 22:40:48.698130 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:48.698073 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03244edf-87f5-4237-be87-8a523c4d96b3-workload-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk\" (UID: \"03244edf-87f5-4237-be87-8a523c4d96b3\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" Nov 03 22:40:48.698469 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:48.698428 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03244edf-87f5-4237-be87-8a523c4d96b3-istiod-ca-cert\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk\" (UID: \"03244edf-87f5-4237-be87-8a523c4d96b3\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" Nov 03 22:40:48.700004 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:48.699984 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03244edf-87f5-4237-be87-8a523c4d96b3-istio-podinfo\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk\" (UID: \"03244edf-87f5-4237-be87-8a523c4d96b3\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" Nov 03 22:40:48.700187 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:48.700169 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03244edf-87f5-4237-be87-8a523c4d96b3-istio-envoy\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk\" (UID: \"03244edf-87f5-4237-be87-8a523c4d96b3\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" Nov 03 22:40:48.759695 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:48.759654 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03244edf-87f5-4237-be87-8a523c4d96b3-istio-token\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk\" (UID: \"03244edf-87f5-4237-be87-8a523c4d96b3\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" Nov 03 22:40:48.759830 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:48.759712 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd76q\" (UniqueName: \"kubernetes.io/projected/03244edf-87f5-4237-be87-8a523c4d96b3-kube-api-access-fd76q\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk\" (UID: \"03244edf-87f5-4237-be87-8a523c4d96b3\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" Nov 03 22:40:48.875832 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:48.875805 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" Nov 03 22:40:49.308542 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:49.308518 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk"] Nov 03 22:40:49.309921 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:40:49.309894 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03244edf_87f5_4237_be87_8a523c4d96b3.slice/crio-ed9329b89804d5084dfb863d140e2495666945bcf11b1a04ee4c1879440e4820 WatchSource:0}: Error finding container ed9329b89804d5084dfb863d140e2495666945bcf11b1a04ee4c1879440e4820: Status 404 returned error can't find the container with id ed9329b89804d5084dfb863d140e2495666945bcf11b1a04ee4c1879440e4820 Nov 03 22:40:49.312332 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:49.312297 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:40:49.312410 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:49.312367 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:40:49.312410 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:49.312398 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:40:49.847034 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:49.846998 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" event={"ID":"03244edf-87f5-4237-be87-8a523c4d96b3","Type":"ContainerStarted","Data":"48aff0ee2038482766ac246ed8191f082ac6300f2908702843d62a0ed2e9274d"} Nov 03 22:40:49.847403 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:49.847063 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" event={"ID":"03244edf-87f5-4237-be87-8a523c4d96b3","Type":"ContainerStarted","Data":"ed9329b89804d5084dfb863d140e2495666945bcf11b1a04ee4c1879440e4820"} Nov 03 22:40:49.876376 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:49.876351 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" Nov 03 22:40:49.970462 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:49.970409 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" podStartSLOduration=1.970394196 podStartE2EDuration="1.970394196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:40:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:40:49.969381228 +0000 UTC m=+1079.278940524" watchObservedRunningTime="2026-11-03 22:40:49.970394196 +0000 UTC m=+1079.279953488" Nov 03 22:40:50.752131 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:50.752088 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:50.877566 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:50.877531 5196 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 03 22:40:50.878076 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:50.877591 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" podUID="03244edf-87f5-4237-be87-8a523c4d96b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 03 22:40:51.876172 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:51.876142 5196 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 03 22:40:51.876422 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:51.876193 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" podUID="03244edf-87f5-4237-be87-8a523c4d96b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 03 22:40:52.876095 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:52.876060 5196 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 03 22:40:52.876473 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:52.876119 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" podUID="03244edf-87f5-4237-be87-8a523c4d96b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 03 22:40:53.876416 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:53.876378 5196 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 03 22:40:53.876805 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:53.876428 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" podUID="03244edf-87f5-4237-be87-8a523c4d96b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 03 22:40:54.877084 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:54.877047 5196 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 03 22:40:54.877469 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:54.877118 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" podUID="03244edf-87f5-4237-be87-8a523c4d96b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 03 22:40:55.620065 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:55.620034 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:55.861617 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:55.861589 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:40:55.876462 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:55.876397 5196 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 03 22:40:55.876588 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:55.876462 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" podUID="03244edf-87f5-4237-be87-8a523c4d96b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 03 22:40:55.954809 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:40:55.954757 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:40:56.880733 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:56.880702 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" Nov 03 22:40:56.881141 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:56.881115 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" Nov 03 22:40:56.882088 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:40:56.882069 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" Nov 03 22:41:00.504017 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:00.503881 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:02.093167 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:02.093137 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5"] Nov 03 22:41:02.103351 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:02.103309 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" Nov 03 22:41:02.181351 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:02.181317 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d2e868-b385-44e6-8a56-e933b33f5b7a-credential-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5\" (UID: \"12d2e868-b385-44e6-8a56-e933b33f5b7a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" Nov 03 22:41:02.181518 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:02.181403 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d2e868-b385-44e6-8a56-e933b33f5b7a-istio-podinfo\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5\" (UID: \"12d2e868-b385-44e6-8a56-e933b33f5b7a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" Nov 03 22:41:02.181518 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:02.181479 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d2e868-b385-44e6-8a56-e933b33f5b7a-istio-token\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5\" (UID: \"12d2e868-b385-44e6-8a56-e933b33f5b7a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" Nov 03 22:41:02.181605 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:02.181585 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5dw6\" (UniqueName: \"kubernetes.io/projected/12d2e868-b385-44e6-8a56-e933b33f5b7a-kube-api-access-z5dw6\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5\" (UID: \"12d2e868-b385-44e6-8a56-e933b33f5b7a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" Nov 03 22:41:02.181645 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:02.181622 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d2e868-b385-44e6-8a56-e933b33f5b7a-istio-envoy\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5\" (UID: \"12d2e868-b385-44e6-8a56-e933b33f5b7a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" Nov 03 22:41:02.181706 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:02.181690 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d2e868-b385-44e6-8a56-e933b33f5b7a-workload-certs\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5\" (UID: \"12d2e868-b385-44e6-8a56-e933b33f5b7a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" Nov 03 22:41:02.181759 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:02.181744 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d2e868-b385-44e6-8a56-e933b33f5b7a-istio-data\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5\" (UID: \"12d2e868-b385-44e6-8a56-e933b33f5b7a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" Nov 03 22:41:02.181804 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:02.181789 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d2e868-b385-44e6-8a56-e933b33f5b7a-workload-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5\" (UID: \"12d2e868-b385-44e6-8a56-e933b33f5b7a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" Nov 03 22:41:02.181900 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:02.181884 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d2e868-b385-44e6-8a56-e933b33f5b7a-istiod-ca-cert\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5\" (UID: \"12d2e868-b385-44e6-8a56-e933b33f5b7a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" Nov 03 22:41:02.203000 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:02.202962 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:02.210599 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:02.210573 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5"] Nov 03 22:41:02.283483 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:02.283439 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d2e868-b385-44e6-8a56-e933b33f5b7a-istio-envoy\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5\" (UID: \"12d2e868-b385-44e6-8a56-e933b33f5b7a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" Nov 03 22:41:02.283678 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:02.283513 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d2e868-b385-44e6-8a56-e933b33f5b7a-workload-certs\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5\" (UID: \"12d2e868-b385-44e6-8a56-e933b33f5b7a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" Nov 03 22:41:02.283678 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:02.283554 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d2e868-b385-44e6-8a56-e933b33f5b7a-istio-data\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5\" (UID: \"12d2e868-b385-44e6-8a56-e933b33f5b7a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" Nov 03 22:41:02.283678 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:02.283595 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d2e868-b385-44e6-8a56-e933b33f5b7a-workload-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5\" (UID: \"12d2e868-b385-44e6-8a56-e933b33f5b7a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" Nov 03 22:41:02.283678 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:02.283651 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d2e868-b385-44e6-8a56-e933b33f5b7a-istiod-ca-cert\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5\" (UID: \"12d2e868-b385-44e6-8a56-e933b33f5b7a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" Nov 03 22:41:02.283952 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:02.283746 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d2e868-b385-44e6-8a56-e933b33f5b7a-credential-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5\" (UID: \"12d2e868-b385-44e6-8a56-e933b33f5b7a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" Nov 03 22:41:02.283952 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:02.283810 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d2e868-b385-44e6-8a56-e933b33f5b7a-istio-podinfo\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5\" (UID: \"12d2e868-b385-44e6-8a56-e933b33f5b7a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" Nov 03 22:41:02.283952 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:02.283848 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d2e868-b385-44e6-8a56-e933b33f5b7a-istio-token\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5\" (UID: \"12d2e868-b385-44e6-8a56-e933b33f5b7a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" Nov 03 22:41:02.284112 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:02.283957 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5dw6\" (UniqueName: \"kubernetes.io/projected/12d2e868-b385-44e6-8a56-e933b33f5b7a-kube-api-access-z5dw6\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5\" (UID: \"12d2e868-b385-44e6-8a56-e933b33f5b7a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" Nov 03 22:41:02.284171 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:02.284145 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d2e868-b385-44e6-8a56-e933b33f5b7a-istio-data\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5\" (UID: \"12d2e868-b385-44e6-8a56-e933b33f5b7a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" Nov 03 22:41:02.284413 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:02.284384 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d2e868-b385-44e6-8a56-e933b33f5b7a-workload-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5\" (UID: \"12d2e868-b385-44e6-8a56-e933b33f5b7a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" Nov 03 22:41:02.284750 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:02.284722 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d2e868-b385-44e6-8a56-e933b33f5b7a-credential-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5\" (UID: \"12d2e868-b385-44e6-8a56-e933b33f5b7a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" Nov 03 22:41:02.284936 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:02.284913 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d2e868-b385-44e6-8a56-e933b33f5b7a-istiod-ca-cert\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5\" (UID: \"12d2e868-b385-44e6-8a56-e933b33f5b7a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" Nov 03 22:41:02.285053 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:02.285025 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d2e868-b385-44e6-8a56-e933b33f5b7a-workload-certs\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5\" (UID: \"12d2e868-b385-44e6-8a56-e933b33f5b7a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" Nov 03 22:41:02.286518 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:02.286484 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d2e868-b385-44e6-8a56-e933b33f5b7a-istio-envoy\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5\" (UID: \"12d2e868-b385-44e6-8a56-e933b33f5b7a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" Nov 03 22:41:02.287869 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:02.287824 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d2e868-b385-44e6-8a56-e933b33f5b7a-istio-podinfo\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5\" (UID: \"12d2e868-b385-44e6-8a56-e933b33f5b7a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" Nov 03 22:41:02.346692 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:02.346600 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d2e868-b385-44e6-8a56-e933b33f5b7a-istio-token\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5\" (UID: \"12d2e868-b385-44e6-8a56-e933b33f5b7a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" Nov 03 22:41:02.346898 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:02.346734 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5dw6\" (UniqueName: \"kubernetes.io/projected/12d2e868-b385-44e6-8a56-e933b33f5b7a-kube-api-access-z5dw6\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5\" (UID: \"12d2e868-b385-44e6-8a56-e933b33f5b7a\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" Nov 03 22:41:02.398722 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:02.398683 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:02.417587 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:02.417564 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" Nov 03 22:41:02.855012 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:02.854985 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5"] Nov 03 22:41:02.856355 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:41:02.856327 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12d2e868_b385_44e6_8a56_e933b33f5b7a.slice/crio-22d1d0266a630dcd1afc0b635d22ddc4367868e64fc33bcba766ee3656556d3a WatchSource:0}: Error finding container 22d1d0266a630dcd1afc0b635d22ddc4367868e64fc33bcba766ee3656556d3a: Status 404 returned error can't find the container with id 22d1d0266a630dcd1afc0b635d22ddc4367868e64fc33bcba766ee3656556d3a Nov 03 22:41:02.859389 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:02.859358 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:41:02.859471 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:02.859425 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:41:02.859471 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:02.859454 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:41:02.950491 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:02.950455 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" event={"ID":"12d2e868-b385-44e6-8a56-e933b33f5b7a","Type":"ContainerStarted","Data":"22d1d0266a630dcd1afc0b635d22ddc4367868e64fc33bcba766ee3656556d3a"} Nov 03 22:41:03.960217 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:03.960178 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" event={"ID":"12d2e868-b385-44e6-8a56-e933b33f5b7a","Type":"ContainerStarted","Data":"33112d1a0e294842f71f78bb7cb01b1cb5422e598e6b6ce48458ef7e0bff1b5f"} Nov 03 22:41:04.195591 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:04.195516 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" podStartSLOduration=2.195498478 podStartE2EDuration="2.195498478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:41:04.141479531 +0000 UTC m=+1093.451038846" watchObservedRunningTime="2026-11-03 22:41:04.195498478 +0000 UTC m=+1093.505057773" Nov 03 22:41:04.420004 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:04.419958 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" Nov 03 22:41:04.420264 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:04.420239 5196 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 03 22:41:04.420348 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:04.420293 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" podUID="12d2e868-b385-44e6-8a56-e933b33f5b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 03 22:41:05.365805 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:05.365770 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:05.424924 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:05.420303 5196 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 03 22:41:05.424924 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:05.420366 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" podUID="12d2e868-b385-44e6-8a56-e933b33f5b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 03 22:41:06.418683 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:06.418639 5196 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 03 22:41:06.419227 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:06.418712 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" podUID="12d2e868-b385-44e6-8a56-e933b33f5b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 03 22:41:07.342415 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:07.342376 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:07.418221 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:07.418187 5196 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 03 22:41:07.418385 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:07.418242 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" podUID="12d2e868-b385-44e6-8a56-e933b33f5b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 03 22:41:08.418437 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:08.418384 5196 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 03 22:41:08.419106 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:08.418441 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" podUID="12d2e868-b385-44e6-8a56-e933b33f5b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 03 22:41:09.418358 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:09.418323 5196 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 03 22:41:09.418542 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:09.418378 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" podUID="12d2e868-b385-44e6-8a56-e933b33f5b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 03 22:41:10.210576 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:10.210540 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:10.422791 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:10.422764 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" Nov 03 22:41:10.423271 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:10.423157 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" Nov 03 22:41:10.423908 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:10.423887 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" Nov 03 22:41:12.368043 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:12.368008 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:15.009543 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:15.009512 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:18.701592 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:18.701539 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:19.802156 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:19.802120 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:24.701769 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:24.701738 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:25.918074 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:41:25.918037 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:41:28.996643 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:28.996613 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:29.503700 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:29.503666 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:34.205776 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:34.205744 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:34.372511 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:34.372479 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n"] Nov 03 22:41:34.376494 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:34.376477 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" Nov 03 22:41:34.430066 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:34.430008 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-1d31f8-openshift-default-dockercfg-wj4vg\"" Nov 03 22:41:34.469847 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:34.469780 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-istiod-ca-cert\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n\" (UID: \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" Nov 03 22:41:34.469847 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:34.469817 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-workload-socket\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n\" (UID: \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" Nov 03 22:41:34.470036 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:34.469941 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgwjh\" (UniqueName: \"kubernetes.io/projected/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-kube-api-access-vgwjh\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n\" (UID: \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" Nov 03 22:41:34.470036 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:34.469975 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-istio-envoy\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n\" (UID: \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" Nov 03 22:41:34.470036 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:34.470023 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-credential-socket\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n\" (UID: \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" Nov 03 22:41:34.470133 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:34.470081 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-istio-data\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n\" (UID: \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" Nov 03 22:41:34.470133 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:34.470104 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-istio-podinfo\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n\" (UID: \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" Nov 03 22:41:34.470215 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:34.470202 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-workload-certs\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n\" (UID: \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" Nov 03 22:41:34.470290 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:34.470274 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-istio-token\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n\" (UID: \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" Nov 03 22:41:34.484131 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:34.484106 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n"] Nov 03 22:41:34.571044 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:34.571018 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-istio-token\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n\" (UID: \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" Nov 03 22:41:34.571210 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:34.571095 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-istiod-ca-cert\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n\" (UID: \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" Nov 03 22:41:34.571210 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:34.571119 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-workload-socket\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n\" (UID: \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" Nov 03 22:41:34.571210 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:34.571146 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgwjh\" (UniqueName: \"kubernetes.io/projected/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-kube-api-access-vgwjh\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n\" (UID: \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" Nov 03 22:41:34.571374 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:34.571280 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-istio-envoy\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n\" (UID: \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" Nov 03 22:41:34.571374 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:34.571333 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-credential-socket\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n\" (UID: \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" Nov 03 22:41:34.571491 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:34.571376 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-istio-data\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n\" (UID: \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" Nov 03 22:41:34.571491 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:34.571403 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-istio-podinfo\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n\" (UID: \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" Nov 03 22:41:34.571593 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:34.571502 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-workload-certs\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n\" (UID: \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" Nov 03 22:41:34.571653 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:34.571622 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-workload-socket\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n\" (UID: \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" Nov 03 22:41:34.571828 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:34.571808 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-credential-socket\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n\" (UID: \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" Nov 03 22:41:34.571927 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:34.571832 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-istio-data\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n\" (UID: \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" Nov 03 22:41:34.571927 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:34.571900 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-workload-certs\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n\" (UID: \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" Nov 03 22:41:34.572200 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:34.571973 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-istiod-ca-cert\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n\" (UID: \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" Nov 03 22:41:34.573687 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:34.573660 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-istio-envoy\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n\" (UID: \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" Nov 03 22:41:34.574131 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:34.574110 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-istio-podinfo\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n\" (UID: \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" Nov 03 22:41:34.633956 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:34.632815 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-istio-token\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n\" (UID: \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" Nov 03 22:41:34.633956 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:34.633755 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgwjh\" (UniqueName: \"kubernetes.io/projected/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-kube-api-access-vgwjh\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n\" (UID: \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" Nov 03 22:41:34.689876 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:34.689836 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" Nov 03 22:41:35.120406 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:35.120378 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n"] Nov 03 22:41:35.121987 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:41:35.121962 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05bd48b9_ff9d_4511_9c48_9878b3ae40b3.slice/crio-f3441ae4fcdbc6e1a95aef74dfde2f640034db57b2d7659a35e88085f12c3959 WatchSource:0}: Error finding container f3441ae4fcdbc6e1a95aef74dfde2f640034db57b2d7659a35e88085f12c3959: Status 404 returned error can't find the container with id f3441ae4fcdbc6e1a95aef74dfde2f640034db57b2d7659a35e88085f12c3959 Nov 03 22:41:35.124428 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:35.124393 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:41:35.124515 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:35.124461 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:41:35.124515 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:35.124489 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:41:35.181929 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:35.181893 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" event={"ID":"05bd48b9-ff9d-4511-9c48-9878b3ae40b3","Type":"ContainerStarted","Data":"f3441ae4fcdbc6e1a95aef74dfde2f640034db57b2d7659a35e88085f12c3959"} Nov 03 22:41:36.189736 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:36.189701 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" event={"ID":"05bd48b9-ff9d-4511-9c48-9878b3ae40b3","Type":"ContainerStarted","Data":"e17b0e300b97020c9d7cc98e22a5f4587e929ee39d67dbc057cf160aabe36a17"} Nov 03 22:41:36.369164 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:36.369103 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" podStartSLOduration=2.369085397 podStartE2EDuration="2.369085397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:41:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:41:36.366500301 +0000 UTC m=+1125.676059614" watchObservedRunningTime="2026-11-03 22:41:36.369085397 +0000 UTC m=+1125.678644760" Nov 03 22:41:36.705005 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:36.704968 5196 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 03 22:41:36.705345 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:36.705319 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" podUID="05bd48b9-ff9d-4511-9c48-9878b3ae40b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 03 22:41:36.705718 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:36.705258 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" Nov 03 22:41:37.697885 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:37.695249 5196 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 03 22:41:37.697885 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:37.695308 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" podUID="05bd48b9-ff9d-4511-9c48-9878b3ae40b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 03 22:41:38.694883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:38.694219 5196 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 03 22:41:38.694883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:38.694281 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" podUID="05bd48b9-ff9d-4511-9c48-9878b3ae40b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 03 22:41:39.046242 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:39.046152 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:39.692241 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:39.691215 5196 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 03 22:41:39.692241 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:39.691285 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" podUID="05bd48b9-ff9d-4511-9c48-9878b3ae40b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 03 22:41:40.692702 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:40.692218 5196 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 03 22:41:40.692702 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:40.692293 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" podUID="05bd48b9-ff9d-4511-9c48-9878b3ae40b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 03 22:41:41.690994 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:41.690954 5196 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 03 22:41:41.691204 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:41.691029 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" podUID="05bd48b9-ff9d-4511-9c48-9878b3ae40b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 03 22:41:42.695035 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:42.695004 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" Nov 03 22:41:42.695427 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:42.695393 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" Nov 03 22:41:42.696142 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:42.696124 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" Nov 03 22:41:43.861240 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:43.861206 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:48.598081 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:48.598052 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:49.665637 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:49.665590 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:53.308406 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:53.308367 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:41:55.919459 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:41:55.919423 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:41:58.045629 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:41:58.045594 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:02.567834 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:02.567794 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:04.085423 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:04.085387 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:07.054361 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:07.054333 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:07.201827 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:07.201781 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:07.692584 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:07.692552 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:07.835105 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:07.835075 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:12.498089 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:12.498056 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:17.311266 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:17.311230 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:22.013498 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:22.013464 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:25.917872 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:42:25.917823 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:42:26.819141 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:26.819113 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:30.270613 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:30.270537 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:31.593624 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:31.593594 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:36.358107 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:36.358055 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:41.134381 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:41.134345 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:41.773439 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:41.773396 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:41.933893 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:41.933299 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:43.302725 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:43.302687 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:46.008454 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:46.008422 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:49.304995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:49.304952 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:50.822055 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:50.822023 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:54.609351 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:54.609318 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd"] Nov 03 22:42:54.613064 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:54.613046 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" Nov 03 22:42:54.685623 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:54.685587 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rl4d\" (UniqueName: \"kubernetes.io/projected/9d8b4fed-b4b0-49ce-9238-e23175e90b70-kube-api-access-9rl4d\") pod \"distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd\" (UID: \"9d8b4fed-b4b0-49ce-9238-e23175e90b70\") " pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" Nov 03 22:42:54.685815 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:54.685761 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d8b4fed-b4b0-49ce-9238-e23175e90b70-tmp-dir\") pod \"distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd\" (UID: \"9d8b4fed-b4b0-49ce-9238-e23175e90b70\") " pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" Nov 03 22:42:54.685815 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:54.685803 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d8b4fed-b4b0-49ce-9238-e23175e90b70-dshm\") pod \"distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd\" (UID: \"9d8b4fed-b4b0-49ce-9238-e23175e90b70\") " pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" Nov 03 22:42:54.685971 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:54.685829 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d8b4fed-b4b0-49ce-9238-e23175e90b70-model-cache\") pod \"distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd\" (UID: \"9d8b4fed-b4b0-49ce-9238-e23175e90b70\") " pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" Nov 03 22:42:54.686025 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:54.685985 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d8b4fed-b4b0-49ce-9238-e23175e90b70-home\") pod \"distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd\" (UID: \"9d8b4fed-b4b0-49ce-9238-e23175e90b70\") " pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" Nov 03 22:42:54.686076 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:54.686057 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d8b4fed-b4b0-49ce-9238-e23175e90b70-tls-certs\") pod \"distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd\" (UID: \"9d8b4fed-b4b0-49ce-9238-e23175e90b70\") " pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" Nov 03 22:42:54.702391 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:54.702364 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:54.717142 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:54.717113 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-1d31f8\"/\"distinct-2-w3-1d31f8-kserve-self-signed-certs\"" Nov 03 22:42:54.720222 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:54.720194 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd"] Nov 03 22:42:54.786952 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:54.786916 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d8b4fed-b4b0-49ce-9238-e23175e90b70-home\") pod \"distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd\" (UID: \"9d8b4fed-b4b0-49ce-9238-e23175e90b70\") " pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" Nov 03 22:42:54.787132 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:54.787007 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d8b4fed-b4b0-49ce-9238-e23175e90b70-tls-certs\") pod \"distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd\" (UID: \"9d8b4fed-b4b0-49ce-9238-e23175e90b70\") " pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" Nov 03 22:42:54.787132 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:54.787064 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rl4d\" (UniqueName: \"kubernetes.io/projected/9d8b4fed-b4b0-49ce-9238-e23175e90b70-kube-api-access-9rl4d\") pod \"distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd\" (UID: \"9d8b4fed-b4b0-49ce-9238-e23175e90b70\") " pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" Nov 03 22:42:54.787256 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:54.787153 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d8b4fed-b4b0-49ce-9238-e23175e90b70-tmp-dir\") pod \"distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd\" (UID: \"9d8b4fed-b4b0-49ce-9238-e23175e90b70\") " pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" Nov 03 22:42:54.787256 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:54.787192 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d8b4fed-b4b0-49ce-9238-e23175e90b70-dshm\") pod \"distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd\" (UID: \"9d8b4fed-b4b0-49ce-9238-e23175e90b70\") " pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" Nov 03 22:42:54.787256 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:54.787219 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d8b4fed-b4b0-49ce-9238-e23175e90b70-model-cache\") pod \"distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd\" (UID: \"9d8b4fed-b4b0-49ce-9238-e23175e90b70\") " pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" Nov 03 22:42:54.787388 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:54.787291 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d8b4fed-b4b0-49ce-9238-e23175e90b70-home\") pod \"distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd\" (UID: \"9d8b4fed-b4b0-49ce-9238-e23175e90b70\") " pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" Nov 03 22:42:54.787502 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:54.787484 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d8b4fed-b4b0-49ce-9238-e23175e90b70-tmp-dir\") pod \"distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd\" (UID: \"9d8b4fed-b4b0-49ce-9238-e23175e90b70\") " pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" Nov 03 22:42:54.787606 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:54.787588 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d8b4fed-b4b0-49ce-9238-e23175e90b70-model-cache\") pod \"distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd\" (UID: \"9d8b4fed-b4b0-49ce-9238-e23175e90b70\") " pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" Nov 03 22:42:54.789461 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:54.789437 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d8b4fed-b4b0-49ce-9238-e23175e90b70-dshm\") pod \"distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd\" (UID: \"9d8b4fed-b4b0-49ce-9238-e23175e90b70\") " pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" Nov 03 22:42:54.789810 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:54.789791 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d8b4fed-b4b0-49ce-9238-e23175e90b70-tls-certs\") pod \"distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd\" (UID: \"9d8b4fed-b4b0-49ce-9238-e23175e90b70\") " pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" Nov 03 22:42:54.852074 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:54.852034 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rl4d\" (UniqueName: \"kubernetes.io/projected/9d8b4fed-b4b0-49ce-9238-e23175e90b70-kube-api-access-9rl4d\") pod \"distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd\" (UID: \"9d8b4fed-b4b0-49ce-9238-e23175e90b70\") " pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" Nov 03 22:42:54.925273 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:54.925189 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" Nov 03 22:42:55.322438 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:55.322412 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd"] Nov 03 22:42:55.323904 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:42:55.323875 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d8b4fed_b4b0_49ce_9238_e23175e90b70.slice/crio-c3fef13c29f5fbf3a1cd3448be2d976639383e809144bed468749e30b97b05a7 WatchSource:0}: Error finding container c3fef13c29f5fbf3a1cd3448be2d976639383e809144bed468749e30b97b05a7: Status 404 returned error can't find the container with id c3fef13c29f5fbf3a1cd3448be2d976639383e809144bed468749e30b97b05a7 Nov 03 22:42:55.675475 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:55.675437 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:55.740924 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:55.740848 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" event={"ID":"9d8b4fed-b4b0-49ce-9238-e23175e90b70","Type":"ContainerStarted","Data":"a0a53dce8b1236693f2cd3a6ca67797ad4fa6a5b7b6d183257902d4ae4935cd5"} Nov 03 22:42:55.741131 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:55.740938 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" event={"ID":"9d8b4fed-b4b0-49ce-9238-e23175e90b70","Type":"ContainerStarted","Data":"c3fef13c29f5fbf3a1cd3448be2d976639383e809144bed468749e30b97b05a7"} Nov 03 22:42:55.864704 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:55.864650 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" podStartSLOduration=1.864635098 podStartE2EDuration="1.864635098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:55.86244559 +0000 UTC m=+1205.172004882" watchObservedRunningTime="2026-11-03 22:42:55.864635098 +0000 UTC m=+1205.174194464" Nov 03 22:42:55.919890 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:42:55.919825 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:42:55.926144 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:55.926096 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" Nov 03 22:42:59.864211 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:59.864181 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp"] Nov 03 22:42:59.868002 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:59.867981 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" Nov 03 22:42:59.927455 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:59.927424 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp"] Nov 03 22:42:59.952034 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:59.952009 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-7afb1c\"/\"openshift-service-ca.crt\"" Nov 03 22:42:59.971422 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:59.971397 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:42:59.974902 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:59.974882 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-7afb1c\"/\"kube-root-ca.crt\"" Nov 03 22:42:59.974985 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:59.974934 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-7afb1c\"/\"distinct-2-w1-7afb1c-kserve-self-signed-certs\"" Nov 03 22:42:59.974985 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:59.974948 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-7afb1c\"/\"default-dockercfg-6f5cb\"" Nov 03 22:42:59.981268 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:59.981247 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abb461c4-9707-4b5c-aba0-793f7bb993e5-tls-certs\") pod \"distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp\" (UID: \"abb461c4-9707-4b5c-aba0-793f7bb993e5\") " pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" Nov 03 22:42:59.981349 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:59.981276 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbrg4\" (UniqueName: \"kubernetes.io/projected/abb461c4-9707-4b5c-aba0-793f7bb993e5-kube-api-access-jbrg4\") pod \"distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp\" (UID: \"abb461c4-9707-4b5c-aba0-793f7bb993e5\") " pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" Nov 03 22:42:59.981349 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:59.981330 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abb461c4-9707-4b5c-aba0-793f7bb993e5-model-cache\") pod \"distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp\" (UID: \"abb461c4-9707-4b5c-aba0-793f7bb993e5\") " pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" Nov 03 22:42:59.981436 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:59.981408 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abb461c4-9707-4b5c-aba0-793f7bb993e5-dshm\") pod \"distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp\" (UID: \"abb461c4-9707-4b5c-aba0-793f7bb993e5\") " pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" Nov 03 22:42:59.981436 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:59.981432 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abb461c4-9707-4b5c-aba0-793f7bb993e5-home\") pod \"distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp\" (UID: \"abb461c4-9707-4b5c-aba0-793f7bb993e5\") " pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" Nov 03 22:42:59.981510 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:42:59.981477 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abb461c4-9707-4b5c-aba0-793f7bb993e5-tmp-dir\") pod \"distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp\" (UID: \"abb461c4-9707-4b5c-aba0-793f7bb993e5\") " pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" Nov 03 22:43:00.083197 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:00.083160 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abb461c4-9707-4b5c-aba0-793f7bb993e5-tls-certs\") pod \"distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp\" (UID: \"abb461c4-9707-4b5c-aba0-793f7bb993e5\") " pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" Nov 03 22:43:00.083492 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:00.083203 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbrg4\" (UniqueName: \"kubernetes.io/projected/abb461c4-9707-4b5c-aba0-793f7bb993e5-kube-api-access-jbrg4\") pod \"distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp\" (UID: \"abb461c4-9707-4b5c-aba0-793f7bb993e5\") " pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" Nov 03 22:43:00.083492 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:00.083265 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abb461c4-9707-4b5c-aba0-793f7bb993e5-model-cache\") pod \"distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp\" (UID: \"abb461c4-9707-4b5c-aba0-793f7bb993e5\") " pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" Nov 03 22:43:00.083492 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:00.083321 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abb461c4-9707-4b5c-aba0-793f7bb993e5-dshm\") pod \"distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp\" (UID: \"abb461c4-9707-4b5c-aba0-793f7bb993e5\") " pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" Nov 03 22:43:00.083492 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:00.083355 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abb461c4-9707-4b5c-aba0-793f7bb993e5-home\") pod \"distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp\" (UID: \"abb461c4-9707-4b5c-aba0-793f7bb993e5\") " pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" Nov 03 22:43:00.083492 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:00.083424 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abb461c4-9707-4b5c-aba0-793f7bb993e5-tmp-dir\") pod \"distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp\" (UID: \"abb461c4-9707-4b5c-aba0-793f7bb993e5\") " pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" Nov 03 22:43:00.083781 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:00.083757 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abb461c4-9707-4b5c-aba0-793f7bb993e5-model-cache\") pod \"distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp\" (UID: \"abb461c4-9707-4b5c-aba0-793f7bb993e5\") " pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" Nov 03 22:43:00.083920 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:00.083902 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abb461c4-9707-4b5c-aba0-793f7bb993e5-tmp-dir\") pod \"distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp\" (UID: \"abb461c4-9707-4b5c-aba0-793f7bb993e5\") " pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" Nov 03 22:43:00.084046 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:00.084022 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abb461c4-9707-4b5c-aba0-793f7bb993e5-home\") pod \"distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp\" (UID: \"abb461c4-9707-4b5c-aba0-793f7bb993e5\") " pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" Nov 03 22:43:00.086108 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:00.086085 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abb461c4-9707-4b5c-aba0-793f7bb993e5-dshm\") pod \"distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp\" (UID: \"abb461c4-9707-4b5c-aba0-793f7bb993e5\") " pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" Nov 03 22:43:00.086669 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:00.086649 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abb461c4-9707-4b5c-aba0-793f7bb993e5-tls-certs\") pod \"distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp\" (UID: \"abb461c4-9707-4b5c-aba0-793f7bb993e5\") " pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" Nov 03 22:43:00.146134 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:00.146060 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbrg4\" (UniqueName: \"kubernetes.io/projected/abb461c4-9707-4b5c-aba0-793f7bb993e5-kube-api-access-jbrg4\") pod \"distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp\" (UID: \"abb461c4-9707-4b5c-aba0-793f7bb993e5\") " pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" Nov 03 22:43:00.180371 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:00.180334 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" Nov 03 22:43:00.440365 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:00.440281 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:00.607001 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:00.606972 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp"] Nov 03 22:43:00.608582 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:43:00.608554 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabb461c4_9707_4b5c_aba0_793f7bb993e5.slice/crio-d9592524f951d87d9e1e180c5e36d306e54c89fe1204f7385f44e8a6398c1e28 WatchSource:0}: Error finding container d9592524f951d87d9e1e180c5e36d306e54c89fe1204f7385f44e8a6398c1e28: Status 404 returned error can't find the container with id d9592524f951d87d9e1e180c5e36d306e54c89fe1204f7385f44e8a6398c1e28 Nov 03 22:43:00.771063 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:00.770969 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" event={"ID":"abb461c4-9707-4b5c-aba0-793f7bb993e5","Type":"ContainerStarted","Data":"beee901e27bf51231fbdbc562b681a34e9337823bd6b83d32898ad74583915ce"} Nov 03 22:43:00.771063 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:00.771025 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" event={"ID":"abb461c4-9707-4b5c-aba0-793f7bb993e5","Type":"ContainerStarted","Data":"d9592524f951d87d9e1e180c5e36d306e54c89fe1204f7385f44e8a6398c1e28"} Nov 03 22:43:00.890432 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:00.890379 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" podStartSLOduration=1.890367122 podStartE2EDuration="1.890367122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:00.889545371 +0000 UTC m=+1210.199104684" watchObservedRunningTime="2026-11-03 22:43:00.890367122 +0000 UTC m=+1210.199926435" Nov 03 22:43:01.180728 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:01.180696 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" Nov 03 22:43:02.967991 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:02.967953 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:03.038381 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.038350 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk"] Nov 03 22:43:03.042399 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.042384 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" Nov 03 22:43:03.124649 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.124616 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcfxl\" (UniqueName: \"kubernetes.io/projected/25d699f6-aaae-480e-9efb-45afb8297cce-kube-api-access-bcfxl\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk\" (UID: \"25d699f6-aaae-480e-9efb-45afb8297cce\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" Nov 03 22:43:03.124782 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.124669 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d699f6-aaae-480e-9efb-45afb8297cce-workload-certs\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk\" (UID: \"25d699f6-aaae-480e-9efb-45afb8297cce\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" Nov 03 22:43:03.124782 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.124760 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d699f6-aaae-480e-9efb-45afb8297cce-workload-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk\" (UID: \"25d699f6-aaae-480e-9efb-45afb8297cce\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" Nov 03 22:43:03.124872 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.124792 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d699f6-aaae-480e-9efb-45afb8297cce-istio-envoy\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk\" (UID: \"25d699f6-aaae-480e-9efb-45afb8297cce\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" Nov 03 22:43:03.124872 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.124813 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d699f6-aaae-480e-9efb-45afb8297cce-istio-token\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk\" (UID: \"25d699f6-aaae-480e-9efb-45afb8297cce\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" Nov 03 22:43:03.124872 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.124832 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d699f6-aaae-480e-9efb-45afb8297cce-istiod-ca-cert\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk\" (UID: \"25d699f6-aaae-480e-9efb-45afb8297cce\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" Nov 03 22:43:03.124978 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.124964 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d699f6-aaae-480e-9efb-45afb8297cce-credential-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk\" (UID: \"25d699f6-aaae-480e-9efb-45afb8297cce\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" Nov 03 22:43:03.125037 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.125023 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d699f6-aaae-480e-9efb-45afb8297cce-istio-data\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk\" (UID: \"25d699f6-aaae-480e-9efb-45afb8297cce\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" Nov 03 22:43:03.125118 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.125105 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d699f6-aaae-480e-9efb-45afb8297cce-istio-podinfo\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk\" (UID: \"25d699f6-aaae-480e-9efb-45afb8297cce\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" Nov 03 22:43:03.226070 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.225988 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d699f6-aaae-480e-9efb-45afb8297cce-istio-token\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk\" (UID: \"25d699f6-aaae-480e-9efb-45afb8297cce\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" Nov 03 22:43:03.226070 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.226023 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d699f6-aaae-480e-9efb-45afb8297cce-istiod-ca-cert\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk\" (UID: \"25d699f6-aaae-480e-9efb-45afb8297cce\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" Nov 03 22:43:03.226070 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.226054 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d699f6-aaae-480e-9efb-45afb8297cce-credential-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk\" (UID: \"25d699f6-aaae-480e-9efb-45afb8297cce\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" Nov 03 22:43:03.226301 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.226126 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d699f6-aaae-480e-9efb-45afb8297cce-istio-data\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk\" (UID: \"25d699f6-aaae-480e-9efb-45afb8297cce\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" Nov 03 22:43:03.226301 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.226197 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d699f6-aaae-480e-9efb-45afb8297cce-istio-podinfo\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk\" (UID: \"25d699f6-aaae-480e-9efb-45afb8297cce\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" Nov 03 22:43:03.226301 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.226271 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcfxl\" (UniqueName: \"kubernetes.io/projected/25d699f6-aaae-480e-9efb-45afb8297cce-kube-api-access-bcfxl\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk\" (UID: \"25d699f6-aaae-480e-9efb-45afb8297cce\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" Nov 03 22:43:03.226452 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.226319 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d699f6-aaae-480e-9efb-45afb8297cce-workload-certs\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk\" (UID: \"25d699f6-aaae-480e-9efb-45afb8297cce\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" Nov 03 22:43:03.226452 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.226377 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d699f6-aaae-480e-9efb-45afb8297cce-workload-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk\" (UID: \"25d699f6-aaae-480e-9efb-45afb8297cce\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" Nov 03 22:43:03.226452 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.226426 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d699f6-aaae-480e-9efb-45afb8297cce-istio-envoy\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk\" (UID: \"25d699f6-aaae-480e-9efb-45afb8297cce\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" Nov 03 22:43:03.226607 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.226552 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d699f6-aaae-480e-9efb-45afb8297cce-istio-data\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk\" (UID: \"25d699f6-aaae-480e-9efb-45afb8297cce\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" Nov 03 22:43:03.226664 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.226603 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d699f6-aaae-480e-9efb-45afb8297cce-credential-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk\" (UID: \"25d699f6-aaae-480e-9efb-45afb8297cce\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" Nov 03 22:43:03.226880 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.226834 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d699f6-aaae-480e-9efb-45afb8297cce-workload-certs\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk\" (UID: \"25d699f6-aaae-480e-9efb-45afb8297cce\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" Nov 03 22:43:03.226997 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.226887 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d699f6-aaae-480e-9efb-45afb8297cce-workload-socket\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk\" (UID: \"25d699f6-aaae-480e-9efb-45afb8297cce\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" Nov 03 22:43:03.226997 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.226894 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d699f6-aaae-480e-9efb-45afb8297cce-istiod-ca-cert\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk\" (UID: \"25d699f6-aaae-480e-9efb-45afb8297cce\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" Nov 03 22:43:03.228912 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.228883 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d699f6-aaae-480e-9efb-45afb8297cce-istio-envoy\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk\" (UID: \"25d699f6-aaae-480e-9efb-45afb8297cce\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" Nov 03 22:43:03.229238 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.229220 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d699f6-aaae-480e-9efb-45afb8297cce-istio-podinfo\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk\" (UID: \"25d699f6-aaae-480e-9efb-45afb8297cce\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" Nov 03 22:43:03.250354 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.250327 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk"] Nov 03 22:43:03.288547 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.288510 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d699f6-aaae-480e-9efb-45afb8297cce-istio-token\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk\" (UID: \"25d699f6-aaae-480e-9efb-45afb8297cce\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" Nov 03 22:43:03.288547 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.288526 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcfxl\" (UniqueName: \"kubernetes.io/projected/25d699f6-aaae-480e-9efb-45afb8297cce-kube-api-access-bcfxl\") pod \"e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk\" (UID: \"25d699f6-aaae-480e-9efb-45afb8297cce\") " pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" Nov 03 22:43:03.358069 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.358039 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" Nov 03 22:43:03.439473 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.439442 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk"] Nov 03 22:43:03.445679 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.445655 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" Nov 03 22:43:03.506835 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.506737 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk"] Nov 03 22:43:03.531347 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.531308 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/552d9e04-ea9c-4e58-b861-dd19b8def5ec-credential-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk\" (UID: \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" Nov 03 22:43:03.531513 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.531366 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/552d9e04-ea9c-4e58-b861-dd19b8def5ec-istiod-ca-cert\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk\" (UID: \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" Nov 03 22:43:03.531513 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.531479 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/552d9e04-ea9c-4e58-b861-dd19b8def5ec-istio-podinfo\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk\" (UID: \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" Nov 03 22:43:03.531513 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.531508 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/552d9e04-ea9c-4e58-b861-dd19b8def5ec-istio-token\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk\" (UID: \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" Nov 03 22:43:03.531626 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.531578 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/552d9e04-ea9c-4e58-b861-dd19b8def5ec-workload-certs\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk\" (UID: \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" Nov 03 22:43:03.531626 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.531611 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-579zq\" (UniqueName: \"kubernetes.io/projected/552d9e04-ea9c-4e58-b861-dd19b8def5ec-kube-api-access-579zq\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk\" (UID: \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" Nov 03 22:43:03.531687 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.531656 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/552d9e04-ea9c-4e58-b861-dd19b8def5ec-istio-envoy\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk\" (UID: \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" Nov 03 22:43:03.531760 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.531719 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/552d9e04-ea9c-4e58-b861-dd19b8def5ec-istio-data\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk\" (UID: \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" Nov 03 22:43:03.531907 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.531885 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/552d9e04-ea9c-4e58-b861-dd19b8def5ec-workload-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk\" (UID: \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" Nov 03 22:43:03.633407 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.633373 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/552d9e04-ea9c-4e58-b861-dd19b8def5ec-istio-podinfo\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk\" (UID: \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" Nov 03 22:43:03.633407 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.633410 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/552d9e04-ea9c-4e58-b861-dd19b8def5ec-istio-token\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk\" (UID: \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" Nov 03 22:43:03.633640 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.633451 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/552d9e04-ea9c-4e58-b861-dd19b8def5ec-workload-certs\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk\" (UID: \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" Nov 03 22:43:03.633640 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.633483 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-579zq\" (UniqueName: \"kubernetes.io/projected/552d9e04-ea9c-4e58-b861-dd19b8def5ec-kube-api-access-579zq\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk\" (UID: \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" Nov 03 22:43:03.633640 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.633513 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/552d9e04-ea9c-4e58-b861-dd19b8def5ec-istio-envoy\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk\" (UID: \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" Nov 03 22:43:03.633640 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.633570 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/552d9e04-ea9c-4e58-b861-dd19b8def5ec-istio-data\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk\" (UID: \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" Nov 03 22:43:03.633876 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.633712 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/552d9e04-ea9c-4e58-b861-dd19b8def5ec-workload-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk\" (UID: \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" Nov 03 22:43:03.633876 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.633827 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/552d9e04-ea9c-4e58-b861-dd19b8def5ec-credential-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk\" (UID: \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" Nov 03 22:43:03.634002 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.633880 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/552d9e04-ea9c-4e58-b861-dd19b8def5ec-istiod-ca-cert\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk\" (UID: \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" Nov 03 22:43:03.634002 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.633932 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/552d9e04-ea9c-4e58-b861-dd19b8def5ec-workload-certs\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk\" (UID: \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" Nov 03 22:43:03.634109 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.634020 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/552d9e04-ea9c-4e58-b861-dd19b8def5ec-istio-data\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk\" (UID: \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" Nov 03 22:43:03.634173 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.634158 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/552d9e04-ea9c-4e58-b861-dd19b8def5ec-credential-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk\" (UID: \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" Nov 03 22:43:03.634396 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.634219 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/552d9e04-ea9c-4e58-b861-dd19b8def5ec-workload-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk\" (UID: \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" Nov 03 22:43:03.634806 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.634771 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/552d9e04-ea9c-4e58-b861-dd19b8def5ec-istiod-ca-cert\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk\" (UID: \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" Nov 03 22:43:03.636061 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.636040 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/552d9e04-ea9c-4e58-b861-dd19b8def5ec-istio-envoy\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk\" (UID: \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" Nov 03 22:43:03.636175 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.636161 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/552d9e04-ea9c-4e58-b861-dd19b8def5ec-istio-podinfo\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk\" (UID: \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" Nov 03 22:43:03.695341 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.695314 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/552d9e04-ea9c-4e58-b861-dd19b8def5ec-istio-token\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk\" (UID: \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" Nov 03 22:43:03.695533 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.695516 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-579zq\" (UniqueName: \"kubernetes.io/projected/552d9e04-ea9c-4e58-b861-dd19b8def5ec-kube-api-access-579zq\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk\" (UID: \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" Nov 03 22:43:03.766594 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.766541 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" Nov 03 22:43:03.817100 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.816755 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk"] Nov 03 22:43:03.817100 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:43:03.816757 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25d699f6_aaae_480e_9efb_45afb8297cce.slice/crio-92ebaeba25a33c19b5ec8f5495e5e58889f8d57f4789925e98cd06d49f23043a WatchSource:0}: Error finding container 92ebaeba25a33c19b5ec8f5495e5e58889f8d57f4789925e98cd06d49f23043a: Status 404 returned error can't find the container with id 92ebaeba25a33c19b5ec8f5495e5e58889f8d57f4789925e98cd06d49f23043a Nov 03 22:43:03.819649 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.819615 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:43:03.819728 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.819689 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:43:03.819764 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:03.819732 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:43:04.156045 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:04.155988 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk"] Nov 03 22:43:04.159043 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:43:04.159010 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod552d9e04_ea9c_4e58_b861_dd19b8def5ec.slice/crio-e7ed88a204d9bb0a52833aa863d8c584310bed67e4807b55c9c3ff72ad54d73a WatchSource:0}: Error finding container e7ed88a204d9bb0a52833aa863d8c584310bed67e4807b55c9c3ff72ad54d73a: Status 404 returned error can't find the container with id e7ed88a204d9bb0a52833aa863d8c584310bed67e4807b55c9c3ff72ad54d73a Nov 03 22:43:04.162342 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:04.162303 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:43:04.162437 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:04.162387 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:43:04.162437 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:04.162426 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:43:04.841501 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:04.841467 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" event={"ID":"25d699f6-aaae-480e-9efb-45afb8297cce","Type":"ContainerStarted","Data":"b6ebc6d92b6c75854e3b164675a8823dfd49bc87618514b10a26d81a95e7a675"} Nov 03 22:43:04.841501 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:04.841516 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" event={"ID":"25d699f6-aaae-480e-9efb-45afb8297cce","Type":"ContainerStarted","Data":"92ebaeba25a33c19b5ec8f5495e5e58889f8d57f4789925e98cd06d49f23043a"} Nov 03 22:43:04.843106 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:04.843082 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" event={"ID":"552d9e04-ea9c-4e58-b861-dd19b8def5ec","Type":"ContainerStarted","Data":"751410b70db17196ae0eaf2ae0e80045a36bfee238b7c6a80af9b021dacd7e4e"} Nov 03 22:43:04.843248 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:04.843120 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" event={"ID":"552d9e04-ea9c-4e58-b861-dd19b8def5ec","Type":"ContainerStarted","Data":"e7ed88a204d9bb0a52833aa863d8c584310bed67e4807b55c9c3ff72ad54d73a"} Nov 03 22:43:04.926016 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:04.925976 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" Nov 03 22:43:04.941479 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:04.941448 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" Nov 03 22:43:05.077840 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:05.077786 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" podStartSLOduration=3.077768153 podStartE2EDuration="3.077768153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:05.028581865 +0000 UTC m=+1214.338141191" watchObservedRunningTime="2026-11-03 22:43:05.077768153 +0000 UTC m=+1214.387327449" Nov 03 22:43:05.257361 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:05.257282 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:05.286871 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:05.286816 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" podStartSLOduration=2.286801607 podStartE2EDuration="2.286801607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:05.283761971 +0000 UTC m=+1214.593321287" watchObservedRunningTime="2026-11-03 22:43:05.286801607 +0000 UTC m=+1214.596360900" Nov 03 22:43:05.365381 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:05.365028 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" Nov 03 22:43:05.365381 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:05.365213 5196 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 22:43:05.365381 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:05.365285 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" podUID="25d699f6-aaae-480e-9efb-45afb8297cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 22:43:05.766762 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:05.766726 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" Nov 03 22:43:05.768072 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:05.768048 5196 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 22:43:05.768186 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:05.768099 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" podUID="552d9e04-ea9c-4e58-b861-dd19b8def5ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 22:43:05.863069 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:05.863044 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" Nov 03 22:43:06.359338 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:06.359292 5196 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 22:43:06.359924 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:06.359368 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" podUID="25d699f6-aaae-480e-9efb-45afb8297cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 22:43:06.767919 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:06.767806 5196 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 22:43:06.768094 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:06.767984 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" podUID="552d9e04-ea9c-4e58-b861-dd19b8def5ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 22:43:07.358612 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:07.358576 5196 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 22:43:07.358815 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:07.358633 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" podUID="25d699f6-aaae-480e-9efb-45afb8297cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 22:43:07.767376 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:07.767299 5196 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 22:43:07.767766 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:07.767377 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" podUID="552d9e04-ea9c-4e58-b861-dd19b8def5ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 22:43:08.189129 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:08.189095 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:08.358993 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:08.358962 5196 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 22:43:08.359173 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:08.359012 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" podUID="25d699f6-aaae-480e-9efb-45afb8297cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 22:43:08.767656 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:08.767621 5196 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 22:43:08.768181 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:08.767841 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" podUID="552d9e04-ea9c-4e58-b861-dd19b8def5ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 22:43:09.381312 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:09.381276 5196 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 22:43:09.381615 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:09.381584 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" podUID="25d699f6-aaae-480e-9efb-45afb8297cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 22:43:09.772566 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:09.772480 5196 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 22:43:09.773173 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:09.773140 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" podUID="552d9e04-ea9c-4e58-b861-dd19b8def5ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 22:43:10.187883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:10.187126 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" Nov 03 22:43:10.363551 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:10.363516 5196 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 22:43:10.363818 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:10.363794 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" podUID="25d699f6-aaae-480e-9efb-45afb8297cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 22:43:10.786984 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:10.786941 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" podUID="9f2e3f52-0664-434f-919b-32736cbf0602" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 22:43:10.788054 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:10.787384 5196 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 22:43:10.788054 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:10.787809 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" podUID="552d9e04-ea9c-4e58-b861-dd19b8def5ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 22:43:10.853974 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:10.853934 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" podUID="9d8b4fed-b4b0-49ce-9238-e23175e90b70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:43:11.114882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:11.114323 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" Nov 03 22:43:11.405471 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:11.388365 5196 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 22:43:11.405471 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:11.388436 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" podUID="25d699f6-aaae-480e-9efb-45afb8297cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 22:43:11.418578 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:11.418544 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:11.794259 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:11.794164 5196 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 22:43:11.794259 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:11.794226 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" podUID="552d9e04-ea9c-4e58-b861-dd19b8def5ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 22:43:11.818904 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:11.817600 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" Nov 03 22:43:12.359076 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:12.359040 5196 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 22:43:12.359261 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:12.359117 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" podUID="25d699f6-aaae-480e-9efb-45afb8297cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 22:43:13.419036 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:13.418990 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" Nov 03 22:43:13.419777 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:13.419753 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" Nov 03 22:43:13.421064 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:13.421037 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" Nov 03 22:43:13.478100 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:13.478068 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" Nov 03 22:43:13.478526 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:13.478502 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" Nov 03 22:43:13.479637 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:13.479616 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" Nov 03 22:43:16.302443 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:16.302394 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:19.214519 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:19.214489 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8"] Nov 03 22:43:19.218597 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:19.218579 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" Nov 03 22:43:19.329625 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:19.329590 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8"] Nov 03 22:43:19.354647 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:19.354608 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-istio-token\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8\" (UID: \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" Nov 03 22:43:19.354827 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:19.354732 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-credential-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8\" (UID: \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" Nov 03 22:43:19.354827 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:19.354764 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-workload-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8\" (UID: \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" Nov 03 22:43:19.354827 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:19.354796 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-workload-certs\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8\" (UID: \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" Nov 03 22:43:19.355033 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:19.354849 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-istio-data\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8\" (UID: \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" Nov 03 22:43:19.355033 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:19.354995 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-istio-envoy\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8\" (UID: \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" Nov 03 22:43:19.355144 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:19.355101 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2cdm\" (UniqueName: \"kubernetes.io/projected/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-kube-api-access-s2cdm\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8\" (UID: \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" Nov 03 22:43:19.355200 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:19.355160 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-istiod-ca-cert\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8\" (UID: \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" Nov 03 22:43:19.355256 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:19.355199 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-istio-podinfo\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8\" (UID: \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" Nov 03 22:43:19.456700 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:19.456659 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-credential-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8\" (UID: \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" Nov 03 22:43:19.456700 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:19.456700 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-workload-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8\" (UID: \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" Nov 03 22:43:19.456964 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:19.456715 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-workload-certs\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8\" (UID: \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" Nov 03 22:43:19.456964 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:19.456731 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-istio-data\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8\" (UID: \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" Nov 03 22:43:19.456964 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:19.456767 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-istio-envoy\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8\" (UID: \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" Nov 03 22:43:19.456964 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:19.456843 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2cdm\" (UniqueName: \"kubernetes.io/projected/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-kube-api-access-s2cdm\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8\" (UID: \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" Nov 03 22:43:19.456964 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:19.456926 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-istiod-ca-cert\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8\" (UID: \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" Nov 03 22:43:19.456964 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:19.456966 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-istio-podinfo\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8\" (UID: \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" Nov 03 22:43:19.457269 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:19.457072 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-istio-token\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8\" (UID: \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" Nov 03 22:43:19.457269 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:19.457111 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-credential-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8\" (UID: \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" Nov 03 22:43:19.457269 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:19.457126 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-workload-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8\" (UID: \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" Nov 03 22:43:19.457269 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:19.457195 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-workload-certs\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8\" (UID: \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" Nov 03 22:43:19.457269 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:19.457235 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-istio-data\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8\" (UID: \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" Nov 03 22:43:19.457800 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:19.457778 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-istiod-ca-cert\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8\" (UID: \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" Nov 03 22:43:19.459487 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:19.459463 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-istio-envoy\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8\" (UID: \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" Nov 03 22:43:19.459718 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:19.459697 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-istio-podinfo\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8\" (UID: \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" Nov 03 22:43:19.519176 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:19.519120 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-istio-token\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8\" (UID: \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" Nov 03 22:43:19.519176 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:19.519164 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2cdm\" (UniqueName: \"kubernetes.io/projected/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-kube-api-access-s2cdm\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8\" (UID: \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" Nov 03 22:43:19.532601 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:19.532581 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" Nov 03 22:43:19.968248 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:19.968224 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8"] Nov 03 22:43:19.969914 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:43:19.969880 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9bf76ea_2e4e_46ee_9c23_3f084e9f8475.slice/crio-4d1ee345ed8de9c7c2418920c289078f4c491a2118eaa04473b1c1012232ca33 WatchSource:0}: Error finding container 4d1ee345ed8de9c7c2418920c289078f4c491a2118eaa04473b1c1012232ca33: Status 404 returned error can't find the container with id 4d1ee345ed8de9c7c2418920c289078f4c491a2118eaa04473b1c1012232ca33 Nov 03 22:43:19.972439 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:19.972409 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:43:19.972495 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:19.972478 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:43:19.972534 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:19.972506 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:43:20.038944 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:20.038912 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" event={"ID":"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475","Type":"ContainerStarted","Data":"4d1ee345ed8de9c7c2418920c289078f4c491a2118eaa04473b1c1012232ca33"} Nov 03 22:43:21.015928 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:21.015875 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:21.054377 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:21.054329 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" event={"ID":"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475","Type":"ContainerStarted","Data":"9a957a86b24bfae22736f54372a25239a3d03fcb45f7ce9cf94a7eced1aeb886"} Nov 03 22:43:21.532721 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:21.532692 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" Nov 03 22:43:21.534822 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:21.534783 5196 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 22:43:21.534990 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:21.534843 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" podUID="c9bf76ea-2e4e-46ee-9c23-3f084e9f8475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 22:43:22.533202 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:22.533168 5196 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 22:43:22.533615 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:22.533218 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" podUID="c9bf76ea-2e4e-46ee-9c23-3f084e9f8475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 22:43:23.533305 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:23.533274 5196 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 22:43:23.533713 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:23.533336 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" podUID="c9bf76ea-2e4e-46ee-9c23-3f084e9f8475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 22:43:24.533847 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:24.533801 5196 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 22:43:24.534252 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:24.533900 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" podUID="c9bf76ea-2e4e-46ee-9c23-3f084e9f8475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 22:43:25.533988 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:25.533957 5196 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 22:43:25.534388 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:25.534018 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" podUID="c9bf76ea-2e4e-46ee-9c23-3f084e9f8475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 22:43:25.846429 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:25.846398 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:25.979080 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:43:25.979046 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:43:26.533814 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:26.533777 5196 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 03 22:43:26.534041 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:26.533830 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" podUID="c9bf76ea-2e4e-46ee-9c23-3f084e9f8475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 03 22:43:27.537414 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:27.537385 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" Nov 03 22:43:27.537795 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:27.537665 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" Nov 03 22:43:27.538456 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:27.538437 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" Nov 03 22:43:27.711198 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:27.711142 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" podStartSLOduration=8.711128562 podStartE2EDuration="8.711128562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:21.236783676 +0000 UTC m=+1230.546342970" watchObservedRunningTime="2026-11-03 22:43:27.711128562 +0000 UTC m=+1237.020687936" Nov 03 22:43:30.622771 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:30.622738 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:34.867101 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:34.867061 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h"] Nov 03 22:43:34.872302 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:34.872284 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" Nov 03 22:43:34.965099 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:34.965070 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h"] Nov 03 22:43:34.977568 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:34.977543 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-istio-data\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h\" (UID: \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" Nov 03 22:43:34.977717 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:34.977667 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-credential-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h\" (UID: \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" Nov 03 22:43:34.977717 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:34.977695 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-istio-podinfo\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h\" (UID: \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" Nov 03 22:43:34.977717 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:34.977715 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-istio-envoy\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h\" (UID: \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" Nov 03 22:43:34.977840 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:34.977803 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-istiod-ca-cert\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h\" (UID: \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" Nov 03 22:43:34.977896 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:34.977840 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-istio-token\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h\" (UID: \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" Nov 03 22:43:34.977932 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:34.977914 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-workload-certs\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h\" (UID: \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" Nov 03 22:43:34.978008 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:34.977993 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sprd\" (UniqueName: \"kubernetes.io/projected/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-kube-api-access-5sprd\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h\" (UID: \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" Nov 03 22:43:34.978050 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:34.978018 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-workload-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h\" (UID: \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" Nov 03 22:43:35.078944 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.078915 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-istio-data\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h\" (UID: \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" Nov 03 22:43:35.079105 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.078996 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-credential-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h\" (UID: \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" Nov 03 22:43:35.079105 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.079015 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-istio-podinfo\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h\" (UID: \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" Nov 03 22:43:35.079105 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.079034 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-istio-envoy\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h\" (UID: \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" Nov 03 22:43:35.079105 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.079089 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-istiod-ca-cert\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h\" (UID: \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" Nov 03 22:43:35.079335 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.079131 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-istio-token\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h\" (UID: \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" Nov 03 22:43:35.079335 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.079175 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-workload-certs\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h\" (UID: \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" Nov 03 22:43:35.079335 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.079248 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sprd\" (UniqueName: \"kubernetes.io/projected/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-kube-api-access-5sprd\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h\" (UID: \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" Nov 03 22:43:35.079335 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.079274 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-workload-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h\" (UID: \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" Nov 03 22:43:35.079556 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.079363 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-istio-data\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h\" (UID: \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" Nov 03 22:43:35.079611 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.079551 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-workload-certs\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h\" (UID: \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" Nov 03 22:43:35.079681 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.079658 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-credential-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h\" (UID: \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" Nov 03 22:43:35.079744 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.079660 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-workload-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h\" (UID: \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" Nov 03 22:43:35.079993 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.079900 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-istiod-ca-cert\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h\" (UID: \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" Nov 03 22:43:35.081535 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.081508 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-istio-envoy\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h\" (UID: \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" Nov 03 22:43:35.081633 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.081596 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-istio-podinfo\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h\" (UID: \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" Nov 03 22:43:35.141358 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.141270 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-istio-token\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h\" (UID: \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" Nov 03 22:43:35.141358 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.141341 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sprd\" (UniqueName: \"kubernetes.io/projected/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-kube-api-access-5sprd\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h\" (UID: \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" Nov 03 22:43:35.186166 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.186141 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" Nov 03 22:43:35.303291 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.303260 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c"] Nov 03 22:43:35.308274 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.308254 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" Nov 03 22:43:35.337798 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.337759 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:35.379412 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.379373 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c"] Nov 03 22:43:35.414614 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.414518 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-workload-certs\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c\" (UID: \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" Nov 03 22:43:35.414794 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.414611 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-istiod-ca-cert\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c\" (UID: \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" Nov 03 22:43:35.415686 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.415659 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-istio-envoy\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c\" (UID: \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" Nov 03 22:43:35.416935 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.416909 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-istio-data\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c\" (UID: \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" Nov 03 22:43:35.417073 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.417000 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-istio-token\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c\" (UID: \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" Nov 03 22:43:35.417178 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.417139 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-istio-podinfo\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c\" (UID: \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" Nov 03 22:43:35.417335 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.417307 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-credential-socket\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c\" (UID: \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" Nov 03 22:43:35.417426 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.417389 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shjhk\" (UniqueName: \"kubernetes.io/projected/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-kube-api-access-shjhk\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c\" (UID: \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" Nov 03 22:43:35.417498 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.417474 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-workload-socket\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c\" (UID: \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" Nov 03 22:43:35.518223 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.518187 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-istio-token\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c\" (UID: \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" Nov 03 22:43:35.518418 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.518244 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-istio-podinfo\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c\" (UID: \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" Nov 03 22:43:35.518418 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.518284 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-credential-socket\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c\" (UID: \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" Nov 03 22:43:35.518418 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.518301 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shjhk\" (UniqueName: \"kubernetes.io/projected/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-kube-api-access-shjhk\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c\" (UID: \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" Nov 03 22:43:35.518418 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.518344 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-workload-socket\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c\" (UID: \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" Nov 03 22:43:35.518418 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.518414 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-workload-certs\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c\" (UID: \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" Nov 03 22:43:35.518677 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.518498 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-istiod-ca-cert\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c\" (UID: \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" Nov 03 22:43:35.518677 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.518582 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-istio-envoy\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c\" (UID: \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" Nov 03 22:43:35.518677 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.518629 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-istio-data\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c\" (UID: \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" Nov 03 22:43:35.518846 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.518820 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-credential-socket\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c\" (UID: \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" Nov 03 22:43:35.518949 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.518887 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-workload-socket\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c\" (UID: \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" Nov 03 22:43:35.518949 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.518890 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-workload-certs\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c\" (UID: \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" Nov 03 22:43:35.519219 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.519196 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-istiod-ca-cert\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c\" (UID: \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" Nov 03 22:43:35.519309 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.519289 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-istio-data\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c\" (UID: \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" Nov 03 22:43:35.520877 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.520839 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-istio-envoy\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c\" (UID: \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" Nov 03 22:43:35.521001 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.520984 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-istio-podinfo\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c\" (UID: \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" Nov 03 22:43:35.580508 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.580460 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-istio-token\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c\" (UID: \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" Nov 03 22:43:35.580693 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.580666 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shjhk\" (UniqueName: \"kubernetes.io/projected/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-kube-api-access-shjhk\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c\" (UID: \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" Nov 03 22:43:35.624490 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.624467 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" Nov 03 22:43:35.626064 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.626043 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h"] Nov 03 22:43:35.627034 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:43:35.627013 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcea0ff59_b9f4_49e6_995e_0c568ca8dd9c.slice/crio-e5e1b4aa7a939f0a1ac0312f1c548fb58459ea3ea6c8a96862ca39b000436269 WatchSource:0}: Error finding container e5e1b4aa7a939f0a1ac0312f1c548fb58459ea3ea6c8a96862ca39b000436269: Status 404 returned error can't find the container with id e5e1b4aa7a939f0a1ac0312f1c548fb58459ea3ea6c8a96862ca39b000436269 Nov 03 22:43:35.629645 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.629613 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:43:35.629700 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.629680 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:43:35.629741 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:35.629712 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:43:36.065906 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:36.065877 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c"] Nov 03 22:43:36.067756 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:43:36.067734 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod734b0dd8_e9c1_4e23_b58d_a5dbf1ee5db0.slice/crio-bea4d0e91dc3afd112d225c41b4a3e472b4f8fdf0b54b5b1246c4db0a6a7124e WatchSource:0}: Error finding container bea4d0e91dc3afd112d225c41b4a3e472b4f8fdf0b54b5b1246c4db0a6a7124e: Status 404 returned error can't find the container with id bea4d0e91dc3afd112d225c41b4a3e472b4f8fdf0b54b5b1246c4db0a6a7124e Nov 03 22:43:36.070891 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:36.070830 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:43:36.070961 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:36.070917 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:43:36.070961 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:36.070948 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:43:36.152916 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:36.152841 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" event={"ID":"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0","Type":"ContainerStarted","Data":"bea4d0e91dc3afd112d225c41b4a3e472b4f8fdf0b54b5b1246c4db0a6a7124e"} Nov 03 22:43:36.154812 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:36.154783 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" event={"ID":"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c","Type":"ContainerStarted","Data":"e3afd01a447b93d65eacbf6d57ae5355d38e6eab04b6cedc9423f275527b2770"} Nov 03 22:43:36.154936 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:36.154834 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" event={"ID":"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c","Type":"ContainerStarted","Data":"e5e1b4aa7a939f0a1ac0312f1c548fb58459ea3ea6c8a96862ca39b000436269"} Nov 03 22:43:36.187695 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:36.186739 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" Nov 03 22:43:36.188399 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:36.188363 5196 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 22:43:36.188548 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:36.188415 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" podUID="cea0ff59-b9f4-49e6-995e-0c568ca8dd9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 22:43:36.280210 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:36.280163 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" podStartSLOduration=2.280149881 podStartE2EDuration="2.280149881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:36.277829609 +0000 UTC m=+1245.587388902" watchObservedRunningTime="2026-11-03 22:43:36.280149881 +0000 UTC m=+1245.589709173" Nov 03 22:43:37.164674 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:37.164635 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" event={"ID":"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0","Type":"ContainerStarted","Data":"c4da88202d281925af5ce892dda476ca0edd932079194e3ce9a72b2b5aa40adf"} Nov 03 22:43:37.187294 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:37.187265 5196 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 22:43:37.187438 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:37.187309 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" podUID="cea0ff59-b9f4-49e6-995e-0c568ca8dd9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 22:43:37.385730 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:37.385680 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" podStartSLOduration=2.385666441 podStartE2EDuration="2.385666441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:37.343196008 +0000 UTC m=+1246.652755323" watchObservedRunningTime="2026-11-03 22:43:37.385666441 +0000 UTC m=+1246.695225804" Nov 03 22:43:37.626884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:37.625922 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" Nov 03 22:43:37.630880 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:37.627108 5196 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 22:43:37.630880 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:37.627171 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" podUID="734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 22:43:38.187264 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:38.187237 5196 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 22:43:38.187720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:38.187308 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" podUID="cea0ff59-b9f4-49e6-995e-0c568ca8dd9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 22:43:38.627883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:38.627239 5196 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 22:43:38.627883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:38.627301 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" podUID="734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 22:43:39.186828 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:39.186799 5196 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 22:43:39.187024 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:39.186846 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" podUID="cea0ff59-b9f4-49e6-995e-0c568ca8dd9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 22:43:39.631998 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:39.631958 5196 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 22:43:39.634321 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:39.634289 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" podUID="734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 22:43:40.186495 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:40.186459 5196 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 22:43:40.186707 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:40.186509 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" podUID="cea0ff59-b9f4-49e6-995e-0c568ca8dd9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 22:43:40.310882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:40.310702 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:40.625884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:40.625550 5196 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 22:43:40.625884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:40.625613 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" podUID="734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 22:43:41.187127 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:41.187097 5196 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 22:43:41.187607 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:41.187150 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" podUID="cea0ff59-b9f4-49e6-995e-0c568ca8dd9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 22:43:41.625327 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:41.625288 5196 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 22:43:41.625642 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:41.625612 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" podUID="734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 22:43:42.186746 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:42.186711 5196 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 22:43:42.186931 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:42.186795 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" podUID="cea0ff59-b9f4-49e6-995e-0c568ca8dd9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 22:43:42.625074 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:42.625038 5196 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 03 22:43:42.625469 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:42.625108 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" podUID="734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 03 22:43:43.212799 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:43.212771 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" Nov 03 22:43:43.213313 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:43.213289 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" Nov 03 22:43:43.214194 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:43.214171 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" Nov 03 22:43:43.665667 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:43.665603 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" Nov 03 22:43:43.667139 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:43.667113 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" Nov 03 22:43:43.667802 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:43.667781 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" Nov 03 22:43:45.315911 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:45.308768 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:50.225248 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:50.225211 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:54.998248 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:54.998211 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:43:55.923568 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:43:55.923529 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:43:59.815736 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:43:59.815660 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:03.269497 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:03.269463 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:03.910259 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:03.910224 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9"] Nov 03 22:44:03.915505 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:03.915489 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" Nov 03 22:44:03.963397 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:03.963348 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45021679-817b-417f-9f0c-dcbad61cc1d3-workload-certs\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9\" (UID: \"45021679-817b-417f-9f0c-dcbad61cc1d3\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" Nov 03 22:44:03.963594 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:03.963414 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45021679-817b-417f-9f0c-dcbad61cc1d3-workload-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9\" (UID: \"45021679-817b-417f-9f0c-dcbad61cc1d3\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" Nov 03 22:44:03.963594 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:03.963517 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45021679-817b-417f-9f0c-dcbad61cc1d3-istiod-ca-cert\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9\" (UID: \"45021679-817b-417f-9f0c-dcbad61cc1d3\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" Nov 03 22:44:03.963727 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:03.963620 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45021679-817b-417f-9f0c-dcbad61cc1d3-credential-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9\" (UID: \"45021679-817b-417f-9f0c-dcbad61cc1d3\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" Nov 03 22:44:03.963727 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:03.963696 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b6zt\" (UniqueName: \"kubernetes.io/projected/45021679-817b-417f-9f0c-dcbad61cc1d3-kube-api-access-8b6zt\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9\" (UID: \"45021679-817b-417f-9f0c-dcbad61cc1d3\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" Nov 03 22:44:03.963837 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:03.963809 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45021679-817b-417f-9f0c-dcbad61cc1d3-istio-token\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9\" (UID: \"45021679-817b-417f-9f0c-dcbad61cc1d3\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" Nov 03 22:44:03.963914 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:03.963836 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45021679-817b-417f-9f0c-dcbad61cc1d3-istio-podinfo\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9\" (UID: \"45021679-817b-417f-9f0c-dcbad61cc1d3\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" Nov 03 22:44:03.963979 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:03.963920 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45021679-817b-417f-9f0c-dcbad61cc1d3-istio-data\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9\" (UID: \"45021679-817b-417f-9f0c-dcbad61cc1d3\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" Nov 03 22:44:03.964114 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:03.964094 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45021679-817b-417f-9f0c-dcbad61cc1d3-istio-envoy\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9\" (UID: \"45021679-817b-417f-9f0c-dcbad61cc1d3\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" Nov 03 22:44:04.065280 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.065240 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45021679-817b-417f-9f0c-dcbad61cc1d3-credential-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9\" (UID: \"45021679-817b-417f-9f0c-dcbad61cc1d3\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" Nov 03 22:44:04.065502 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.065341 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b6zt\" (UniqueName: \"kubernetes.io/projected/45021679-817b-417f-9f0c-dcbad61cc1d3-kube-api-access-8b6zt\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9\" (UID: \"45021679-817b-417f-9f0c-dcbad61cc1d3\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" Nov 03 22:44:04.065502 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.065400 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45021679-817b-417f-9f0c-dcbad61cc1d3-istio-token\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9\" (UID: \"45021679-817b-417f-9f0c-dcbad61cc1d3\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" Nov 03 22:44:04.065502 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.065427 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45021679-817b-417f-9f0c-dcbad61cc1d3-istio-podinfo\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9\" (UID: \"45021679-817b-417f-9f0c-dcbad61cc1d3\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" Nov 03 22:44:04.065502 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.065449 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45021679-817b-417f-9f0c-dcbad61cc1d3-istio-data\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9\" (UID: \"45021679-817b-417f-9f0c-dcbad61cc1d3\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" Nov 03 22:44:04.065746 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.065540 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45021679-817b-417f-9f0c-dcbad61cc1d3-istio-envoy\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9\" (UID: \"45021679-817b-417f-9f0c-dcbad61cc1d3\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" Nov 03 22:44:04.065746 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.065574 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45021679-817b-417f-9f0c-dcbad61cc1d3-workload-certs\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9\" (UID: \"45021679-817b-417f-9f0c-dcbad61cc1d3\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" Nov 03 22:44:04.065746 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.065620 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45021679-817b-417f-9f0c-dcbad61cc1d3-workload-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9\" (UID: \"45021679-817b-417f-9f0c-dcbad61cc1d3\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" Nov 03 22:44:04.065746 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.065635 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45021679-817b-417f-9f0c-dcbad61cc1d3-istiod-ca-cert\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9\" (UID: \"45021679-817b-417f-9f0c-dcbad61cc1d3\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" Nov 03 22:44:04.066689 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.066387 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45021679-817b-417f-9f0c-dcbad61cc1d3-credential-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9\" (UID: \"45021679-817b-417f-9f0c-dcbad61cc1d3\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" Nov 03 22:44:04.066689 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.066448 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45021679-817b-417f-9f0c-dcbad61cc1d3-istiod-ca-cert\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9\" (UID: \"45021679-817b-417f-9f0c-dcbad61cc1d3\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" Nov 03 22:44:04.066689 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.066645 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45021679-817b-417f-9f0c-dcbad61cc1d3-workload-certs\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9\" (UID: \"45021679-817b-417f-9f0c-dcbad61cc1d3\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" Nov 03 22:44:04.066985 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.066725 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45021679-817b-417f-9f0c-dcbad61cc1d3-workload-socket\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9\" (UID: \"45021679-817b-417f-9f0c-dcbad61cc1d3\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" Nov 03 22:44:04.066985 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.066803 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45021679-817b-417f-9f0c-dcbad61cc1d3-istio-data\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9\" (UID: \"45021679-817b-417f-9f0c-dcbad61cc1d3\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" Nov 03 22:44:04.069509 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.069478 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45021679-817b-417f-9f0c-dcbad61cc1d3-istio-podinfo\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9\" (UID: \"45021679-817b-417f-9f0c-dcbad61cc1d3\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" Nov 03 22:44:04.070068 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.070044 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45021679-817b-417f-9f0c-dcbad61cc1d3-istio-envoy\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9\" (UID: \"45021679-817b-417f-9f0c-dcbad61cc1d3\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" Nov 03 22:44:04.125328 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.125296 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9"] Nov 03 22:44:04.146267 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.146232 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b6zt\" (UniqueName: \"kubernetes.io/projected/45021679-817b-417f-9f0c-dcbad61cc1d3-kube-api-access-8b6zt\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9\" (UID: \"45021679-817b-417f-9f0c-dcbad61cc1d3\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" Nov 03 22:44:04.146579 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.146557 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45021679-817b-417f-9f0c-dcbad61cc1d3-istio-token\") pod \"e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9\" (UID: \"45021679-817b-417f-9f0c-dcbad61cc1d3\") " pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" Nov 03 22:44:04.152719 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.152696 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46"] Nov 03 22:44:04.157752 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.157736 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" Nov 03 22:44:04.223357 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.223324 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46"] Nov 03 22:44:04.231206 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.231100 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" Nov 03 22:44:04.267932 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.267896 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-workload-certs\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46\" (UID: \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" Nov 03 22:44:04.268065 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.267952 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwlk2\" (UniqueName: \"kubernetes.io/projected/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-kube-api-access-zwlk2\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46\" (UID: \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" Nov 03 22:44:04.268254 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.268229 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-istiod-ca-cert\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46\" (UID: \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" Nov 03 22:44:04.268383 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.268362 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-workload-socket\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46\" (UID: \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" Nov 03 22:44:04.268451 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.268421 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-istio-podinfo\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46\" (UID: \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" Nov 03 22:44:04.268511 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.268460 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-credential-socket\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46\" (UID: \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" Nov 03 22:44:04.268511 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.268485 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-istio-token\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46\" (UID: \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" Nov 03 22:44:04.268625 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.268565 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-istio-envoy\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46\" (UID: \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" Nov 03 22:44:04.268625 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.268608 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-istio-data\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46\" (UID: \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" Nov 03 22:44:04.369780 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.369745 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-istiod-ca-cert\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46\" (UID: \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" Nov 03 22:44:04.370238 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.369800 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-workload-socket\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46\" (UID: \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" Nov 03 22:44:04.370238 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.369829 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-istio-podinfo\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46\" (UID: \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" Nov 03 22:44:04.370238 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.369874 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-credential-socket\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46\" (UID: \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" Nov 03 22:44:04.370238 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.369899 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-istio-token\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46\" (UID: \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" Nov 03 22:44:04.370238 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.369926 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-istio-envoy\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46\" (UID: \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" Nov 03 22:44:04.370238 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.369955 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-istio-data\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46\" (UID: \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" Nov 03 22:44:04.370238 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.370032 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-workload-certs\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46\" (UID: \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" Nov 03 22:44:04.370238 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.370058 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwlk2\" (UniqueName: \"kubernetes.io/projected/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-kube-api-access-zwlk2\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46\" (UID: \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" Nov 03 22:44:04.370679 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.370295 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-workload-socket\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46\" (UID: \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" Nov 03 22:44:04.370679 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.370591 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-credential-socket\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46\" (UID: \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" Nov 03 22:44:04.370679 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.370627 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-workload-certs\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46\" (UID: \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" Nov 03 22:44:04.370679 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.370673 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-istiod-ca-cert\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46\" (UID: \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" Nov 03 22:44:04.370885 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.370700 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-istio-data\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46\" (UID: \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" Nov 03 22:44:04.372945 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.372914 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-istio-envoy\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46\" (UID: \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" Nov 03 22:44:04.373102 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.373086 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-istio-podinfo\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46\" (UID: \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" Nov 03 22:44:04.432182 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.432155 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-istio-token\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46\" (UID: \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" Nov 03 22:44:04.432472 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.432453 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwlk2\" (UniqueName: \"kubernetes.io/projected/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-kube-api-access-zwlk2\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46\" (UID: \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" Nov 03 22:44:04.492245 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.492185 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" Nov 03 22:44:04.677401 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.677372 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9"] Nov 03 22:44:04.679005 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:44:04.678980 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45021679_817b_417f_9f0c_dcbad61cc1d3.slice/crio-8032eef40add27d2ac4c8123da9e819bdedb22956ef35bf2ef265ff9e11c4e79 WatchSource:0}: Error finding container 8032eef40add27d2ac4c8123da9e819bdedb22956ef35bf2ef265ff9e11c4e79: Status 404 returned error can't find the container with id 8032eef40add27d2ac4c8123da9e819bdedb22956ef35bf2ef265ff9e11c4e79 Nov 03 22:44:04.682212 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.682177 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:44:04.682300 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.682254 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:44:04.682300 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.682290 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:44:04.747821 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.747771 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:04.909202 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.909170 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46"] Nov 03 22:44:04.910470 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:44:04.910447 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc974ac3a_2c08_40df_a4a1_c62b3d3fa040.slice/crio-7ddc2bb8df66d4752d1072e32b6534eabbbceeb9b8893da6023435f58b63d9b5 WatchSource:0}: Error finding container 7ddc2bb8df66d4752d1072e32b6534eabbbceeb9b8893da6023435f58b63d9b5: Status 404 returned error can't find the container with id 7ddc2bb8df66d4752d1072e32b6534eabbbceeb9b8893da6023435f58b63d9b5 Nov 03 22:44:04.913034 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.913001 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:44:04.913139 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.913079 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:44:04.913139 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:04.913122 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:44:05.458428 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:05.458390 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" event={"ID":"45021679-817b-417f-9f0c-dcbad61cc1d3","Type":"ContainerStarted","Data":"0c3d9a242e94fe9503229343f2383db85ab47b8d460c16e08eb5830b71b80051"} Nov 03 22:44:05.458939 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:05.458450 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" event={"ID":"45021679-817b-417f-9f0c-dcbad61cc1d3","Type":"ContainerStarted","Data":"8032eef40add27d2ac4c8123da9e819bdedb22956ef35bf2ef265ff9e11c4e79"} Nov 03 22:44:05.460189 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:05.460157 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" event={"ID":"c974ac3a-2c08-40df-a4a1-c62b3d3fa040","Type":"ContainerStarted","Data":"f972cb5d9f3b6cd667637ec463a0527751eec2f01cc0de27c88d6a9cb6e97993"} Nov 03 22:44:05.460309 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:05.460218 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" event={"ID":"c974ac3a-2c08-40df-a4a1-c62b3d3fa040","Type":"ContainerStarted","Data":"7ddc2bb8df66d4752d1072e32b6534eabbbceeb9b8893da6023435f58b63d9b5"} Nov 03 22:44:05.501883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:05.496221 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" Nov 03 22:44:05.515951 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:05.515029 5196 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 22:44:05.515951 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:05.515091 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" podUID="c974ac3a-2c08-40df-a4a1-c62b3d3fa040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 22:44:05.585477 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:05.585423 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" podStartSLOduration=2.585408462 podStartE2EDuration="2.585408462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:05.583468526 +0000 UTC m=+1274.893027815" watchObservedRunningTime="2026-11-03 22:44:05.585408462 +0000 UTC m=+1274.894967756" Nov 03 22:44:06.252464 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:06.252430 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" Nov 03 22:44:06.253240 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:06.253193 5196 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 22:44:06.253463 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:06.253424 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" podUID="45021679-817b-417f-9f0c-dcbad61cc1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 22:44:06.498884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:06.498331 5196 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 22:44:06.498884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:06.498395 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" podUID="c974ac3a-2c08-40df-a4a1-c62b3d3fa040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 22:44:07.239895 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:07.236373 5196 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 22:44:07.239895 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:07.236435 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" podUID="45021679-817b-417f-9f0c-dcbad61cc1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 22:44:07.521961 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:07.514297 5196 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 22:44:07.521961 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:07.514361 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" podUID="c974ac3a-2c08-40df-a4a1-c62b3d3fa040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 22:44:08.242507 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:08.241047 5196 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 22:44:08.242507 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:08.241102 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" podUID="45021679-817b-417f-9f0c-dcbad61cc1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 22:44:08.418883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:08.418683 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" podStartSLOduration=4.418661171 podStartE2EDuration="4.418661171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:05.744310775 +0000 UTC m=+1275.053870067" watchObservedRunningTime="2026-11-03 22:44:08.418661171 +0000 UTC m=+1277.728220462" Nov 03 22:44:08.438925 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:08.438871 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn"] Nov 03 22:44:08.439164 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:08.439130 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" podUID="26ad9218-689e-4989-8fb5-4f311773e668" containerName="istio-proxy" containerID="cri-o://bac6b676c2ff28f05520593ed372758812778294eaacb203f01e7afd94597904" gracePeriod=30 Nov 03 22:44:08.472570 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:08.472536 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt"] Nov 03 22:44:08.473062 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:08.473029 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" podUID="6b095b36-a492-414f-92bd-f1e3ac60fe7c" containerName="istio-proxy" containerID="cri-o://64d89eb2bd656eaaf7f579f29005ef02bbb4509284853a0a31e5c27695f36ae1" gracePeriod=30 Nov 03 22:44:08.552135 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:08.552042 5196 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 22:44:08.552729 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:08.552693 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" podUID="c974ac3a-2c08-40df-a4a1-c62b3d3fa040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 22:44:08.738258 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:08.718751 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:09.238884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:09.238270 5196 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 22:44:09.238884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:09.238334 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" podUID="45021679-817b-417f-9f0c-dcbad61cc1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 22:44:09.510932 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:09.493275 5196 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 22:44:09.510932 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:09.493347 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" podUID="c974ac3a-2c08-40df-a4a1-c62b3d3fa040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 22:44:10.018883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:10.018413 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:10.243824 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:10.243788 5196 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 22:44:10.244605 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:10.244574 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" podUID="45021679-817b-417f-9f0c-dcbad61cc1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 22:44:10.516182 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:10.516146 5196 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 22:44:10.516561 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:10.516532 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" podUID="c974ac3a-2c08-40df-a4a1-c62b3d3fa040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 22:44:11.231885 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:11.231806 5196 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 22:44:11.232404 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:11.231885 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" podUID="45021679-817b-417f-9f0c-dcbad61cc1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 22:44:11.492969 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:11.492880 5196 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 22:44:11.492969 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:11.492936 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" podUID="c974ac3a-2c08-40df-a4a1-c62b3d3fa040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 22:44:11.730806 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:11.730764 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vs7zn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:44:11.731026 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:11.730825 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" podUID="26ad9218-689e-4989-8fb5-4f311773e668" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:44:12.236991 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:12.236960 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" Nov 03 22:44:12.237440 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:12.237418 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" Nov 03 22:44:12.238314 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:12.238291 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" Nov 03 22:44:12.555474 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:12.555394 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" Nov 03 22:44:12.556742 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:12.556529 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" Nov 03 22:44:12.557284 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:12.557266 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" Nov 03 22:44:13.734134 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.734070 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" Nov 03 22:44:13.780185 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.780163 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" Nov 03 22:44:13.796601 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.793739 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26ad9218-689e-4989-8fb5-4f311773e668-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ad9218-689e-4989-8fb5-4f311773e668-credential-socket\") pod \"26ad9218-689e-4989-8fb5-4f311773e668\" (UID: \"26ad9218-689e-4989-8fb5-4f311773e668\") " Nov 03 22:44:13.796601 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.793832 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26ad9218-689e-4989-8fb5-4f311773e668-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26ad9218-689e-4989-8fb5-4f311773e668-workload-certs\") pod \"26ad9218-689e-4989-8fb5-4f311773e668\" (UID: \"26ad9218-689e-4989-8fb5-4f311773e668\") " Nov 03 22:44:13.796601 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.793893 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26ad9218-689e-4989-8fb5-4f311773e668-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26ad9218-689e-4989-8fb5-4f311773e668-istio-podinfo\") pod \"26ad9218-689e-4989-8fb5-4f311773e668\" (UID: \"26ad9218-689e-4989-8fb5-4f311773e668\") " Nov 03 22:44:13.796601 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.793970 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26ad9218-689e-4989-8fb5-4f311773e668-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26ad9218-689e-4989-8fb5-4f311773e668-istio-data\") pod \"26ad9218-689e-4989-8fb5-4f311773e668\" (UID: \"26ad9218-689e-4989-8fb5-4f311773e668\") " Nov 03 22:44:13.796601 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.794009 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26ad9218-689e-4989-8fb5-4f311773e668-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ad9218-689e-4989-8fb5-4f311773e668-workload-socket\") pod \"26ad9218-689e-4989-8fb5-4f311773e668\" (UID: \"26ad9218-689e-4989-8fb5-4f311773e668\") " Nov 03 22:44:13.796601 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.794076 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26ad9218-689e-4989-8fb5-4f311773e668-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26ad9218-689e-4989-8fb5-4f311773e668-istiod-ca-cert\") pod \"26ad9218-689e-4989-8fb5-4f311773e668\" (UID: \"26ad9218-689e-4989-8fb5-4f311773e668\") " Nov 03 22:44:13.796601 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.794113 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26ad9218-689e-4989-8fb5-4f311773e668-kube-api-access-cv6xm\" (UniqueName: \"kubernetes.io/projected/26ad9218-689e-4989-8fb5-4f311773e668-kube-api-access-cv6xm\") pod \"26ad9218-689e-4989-8fb5-4f311773e668\" (UID: \"26ad9218-689e-4989-8fb5-4f311773e668\") " Nov 03 22:44:13.796601 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.794243 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26ad9218-689e-4989-8fb5-4f311773e668-istio-token\" (UniqueName: \"kubernetes.io/projected/26ad9218-689e-4989-8fb5-4f311773e668-istio-token\") pod \"26ad9218-689e-4989-8fb5-4f311773e668\" (UID: \"26ad9218-689e-4989-8fb5-4f311773e668\") " Nov 03 22:44:13.796601 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.794295 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26ad9218-689e-4989-8fb5-4f311773e668-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26ad9218-689e-4989-8fb5-4f311773e668-istio-envoy\") pod \"26ad9218-689e-4989-8fb5-4f311773e668\" (UID: \"26ad9218-689e-4989-8fb5-4f311773e668\") " Nov 03 22:44:13.797157 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.796708 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ad9218-689e-4989-8fb5-4f311773e668-istio-data" pod "26ad9218-689e-4989-8fb5-4f311773e668" (UID: "26ad9218-689e-4989-8fb5-4f311773e668"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:13.797157 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.796978 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ad9218-689e-4989-8fb5-4f311773e668-workload-socket" pod "26ad9218-689e-4989-8fb5-4f311773e668" (UID: "26ad9218-689e-4989-8fb5-4f311773e668"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:13.797157 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.796982 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26ad9218-689e-4989-8fb5-4f311773e668-istiod-ca-cert" pod "26ad9218-689e-4989-8fb5-4f311773e668" (UID: "26ad9218-689e-4989-8fb5-4f311773e668"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:13.797157 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.796954 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ad9218-689e-4989-8fb5-4f311773e668-credential-socket" pod "26ad9218-689e-4989-8fb5-4f311773e668" (UID: "26ad9218-689e-4989-8fb5-4f311773e668"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:13.797157 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.796998 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ad9218-689e-4989-8fb5-4f311773e668-workload-certs" pod "26ad9218-689e-4989-8fb5-4f311773e668" (UID: "26ad9218-689e-4989-8fb5-4f311773e668"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:13.800519 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.800498 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ad9218-689e-4989-8fb5-4f311773e668-istio-envoy" pod "26ad9218-689e-4989-8fb5-4f311773e668" (UID: "26ad9218-689e-4989-8fb5-4f311773e668"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:13.800644 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.800626 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ad9218-689e-4989-8fb5-4f311773e668-kube-api-access-cv6xm" pod "26ad9218-689e-4989-8fb5-4f311773e668" (UID: "26ad9218-689e-4989-8fb5-4f311773e668"). InnerVolumeSpecName "kube-api-access-cv6xm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:13.800806 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.800790 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ad9218-689e-4989-8fb5-4f311773e668-istio-token" pod "26ad9218-689e-4989-8fb5-4f311773e668" (UID: "26ad9218-689e-4989-8fb5-4f311773e668"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:13.800885 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.800797 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26ad9218-689e-4989-8fb5-4f311773e668-istio-podinfo" pod "26ad9218-689e-4989-8fb5-4f311773e668" (UID: "26ad9218-689e-4989-8fb5-4f311773e668"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:44:13.895757 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.895729 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b095b36-a492-414f-92bd-f1e3ac60fe7c-kube-api-access-knmnh\" (UniqueName: \"kubernetes.io/projected/6b095b36-a492-414f-92bd-f1e3ac60fe7c-kube-api-access-knmnh\") pod \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\" (UID: \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\") " Nov 03 22:44:13.895916 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.895768 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b095b36-a492-414f-92bd-f1e3ac60fe7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b095b36-a492-414f-92bd-f1e3ac60fe7c-istiod-ca-cert\") pod \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\" (UID: \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\") " Nov 03 22:44:13.895916 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.895891 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b095b36-a492-414f-92bd-f1e3ac60fe7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b095b36-a492-414f-92bd-f1e3ac60fe7c-workload-certs\") pod \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\" (UID: \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\") " Nov 03 22:44:13.896044 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.895919 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b095b36-a492-414f-92bd-f1e3ac60fe7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b095b36-a492-414f-92bd-f1e3ac60fe7c-credential-socket\") pod \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\" (UID: \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\") " Nov 03 22:44:13.896044 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.895974 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b095b36-a492-414f-92bd-f1e3ac60fe7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b095b36-a492-414f-92bd-f1e3ac60fe7c-istio-data\") pod \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\" (UID: \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\") " Nov 03 22:44:13.896044 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.896035 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b095b36-a492-414f-92bd-f1e3ac60fe7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b095b36-a492-414f-92bd-f1e3ac60fe7c-istio-envoy\") pod \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\" (UID: \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\") " Nov 03 22:44:13.896250 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.896085 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b095b36-a492-414f-92bd-f1e3ac60fe7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b095b36-a492-414f-92bd-f1e3ac60fe7c-istio-podinfo\") pod \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\" (UID: \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\") " Nov 03 22:44:13.896250 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.896130 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b095b36-a492-414f-92bd-f1e3ac60fe7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b095b36-a492-414f-92bd-f1e3ac60fe7c-workload-socket\") pod \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\" (UID: \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\") " Nov 03 22:44:13.896250 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.896168 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b095b36-a492-414f-92bd-f1e3ac60fe7c-workload-certs" pod "6b095b36-a492-414f-92bd-f1e3ac60fe7c" (UID: "6b095b36-a492-414f-92bd-f1e3ac60fe7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:13.896250 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.896174 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b095b36-a492-414f-92bd-f1e3ac60fe7c-istiod-ca-cert" pod "6b095b36-a492-414f-92bd-f1e3ac60fe7c" (UID: "6b095b36-a492-414f-92bd-f1e3ac60fe7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:13.896250 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.896228 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b095b36-a492-414f-92bd-f1e3ac60fe7c-istio-token\" (UniqueName: \"kubernetes.io/projected/6b095b36-a492-414f-92bd-f1e3ac60fe7c-istio-token\") pod \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\" (UID: \"6b095b36-a492-414f-92bd-f1e3ac60fe7c\") " Nov 03 22:44:13.896479 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.896245 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b095b36-a492-414f-92bd-f1e3ac60fe7c-credential-socket" pod "6b095b36-a492-414f-92bd-f1e3ac60fe7c" (UID: "6b095b36-a492-414f-92bd-f1e3ac60fe7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:13.896479 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.896388 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b095b36-a492-414f-92bd-f1e3ac60fe7c-workload-socket" pod "6b095b36-a492-414f-92bd-f1e3ac60fe7c" (UID: "6b095b36-a492-414f-92bd-f1e3ac60fe7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:13.896769 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.896740 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b095b36-a492-414f-92bd-f1e3ac60fe7c-istio-data" pod "6b095b36-a492-414f-92bd-f1e3ac60fe7c" (UID: "6b095b36-a492-414f-92bd-f1e3ac60fe7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:13.896884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.896838 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ad9218-689e-4989-8fb5-4f311773e668-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:13.896884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.896874 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b095b36-a492-414f-92bd-f1e3ac60fe7c-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:13.896884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.896888 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26ad9218-689e-4989-8fb5-4f311773e668-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:13.896884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.896897 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26ad9218-689e-4989-8fb5-4f311773e668-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:13.896884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.896906 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26ad9218-689e-4989-8fb5-4f311773e668-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:13.896884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.896914 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b095b36-a492-414f-92bd-f1e3ac60fe7c-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:13.896884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.896922 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b095b36-a492-414f-92bd-f1e3ac60fe7c-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:13.896884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.896931 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ad9218-689e-4989-8fb5-4f311773e668-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:13.896884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.896939 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26ad9218-689e-4989-8fb5-4f311773e668-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:13.896884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.896948 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv6xm\" (UniqueName: \"kubernetes.io/projected/26ad9218-689e-4989-8fb5-4f311773e668-kube-api-access-cv6xm\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:13.896884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.896957 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b095b36-a492-414f-92bd-f1e3ac60fe7c-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:13.896884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.896966 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26ad9218-689e-4989-8fb5-4f311773e668-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:13.896884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.896974 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26ad9218-689e-4989-8fb5-4f311773e668-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:13.898139 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.898117 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b095b36-a492-414f-92bd-f1e3ac60fe7c-kube-api-access-knmnh" pod "6b095b36-a492-414f-92bd-f1e3ac60fe7c" (UID: "6b095b36-a492-414f-92bd-f1e3ac60fe7c"). InnerVolumeSpecName "kube-api-access-knmnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:13.898249 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.898225 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b095b36-a492-414f-92bd-f1e3ac60fe7c-istio-podinfo" pod "6b095b36-a492-414f-92bd-f1e3ac60fe7c" (UID: "6b095b36-a492-414f-92bd-f1e3ac60fe7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:44:13.898560 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.898541 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b095b36-a492-414f-92bd-f1e3ac60fe7c-istio-token" pod "6b095b36-a492-414f-92bd-f1e3ac60fe7c" (UID: "6b095b36-a492-414f-92bd-f1e3ac60fe7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:13.899364 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.899344 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b095b36-a492-414f-92bd-f1e3ac60fe7c-istio-envoy" pod "6b095b36-a492-414f-92bd-f1e3ac60fe7c" (UID: "6b095b36-a492-414f-92bd-f1e3ac60fe7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:13.998647 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.998567 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b095b36-a492-414f-92bd-f1e3ac60fe7c-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:13.998647 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.998594 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b095b36-a492-414f-92bd-f1e3ac60fe7c-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:13.998647 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.998604 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knmnh\" (UniqueName: \"kubernetes.io/projected/6b095b36-a492-414f-92bd-f1e3ac60fe7c-kube-api-access-knmnh\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:13.998647 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.998615 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b095b36-a492-414f-92bd-f1e3ac60fe7c-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:13.998647 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:13.998624 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b095b36-a492-414f-92bd-f1e3ac60fe7c-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:14.553591 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:14.553555 5196 generic.go:356] "Generic (PLEG): container finished" podID="6b095b36-a492-414f-92bd-f1e3ac60fe7c" containerID="64d89eb2bd656eaaf7f579f29005ef02bbb4509284853a0a31e5c27695f36ae1" exitCode=0 Nov 03 22:44:14.553779 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:14.553620 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" Nov 03 22:44:14.553779 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:14.553641 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" event={"ID":"6b095b36-a492-414f-92bd-f1e3ac60fe7c","Type":"ContainerDied","Data":"64d89eb2bd656eaaf7f579f29005ef02bbb4509284853a0a31e5c27695f36ae1"} Nov 03 22:44:14.553779 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:14.553693 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt" event={"ID":"6b095b36-a492-414f-92bd-f1e3ac60fe7c","Type":"ContainerDied","Data":"52b1f70b27d4981ad7e551752ef4b0cb6abc534848cfb2c6c46a281757340dbf"} Nov 03 22:44:14.553779 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:14.553717 5196 scope.go:122] "RemoveContainer" containerID="64d89eb2bd656eaaf7f579f29005ef02bbb4509284853a0a31e5c27695f36ae1" Nov 03 22:44:14.555170 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:14.555143 5196 generic.go:356] "Generic (PLEG): container finished" podID="26ad9218-689e-4989-8fb5-4f311773e668" containerID="bac6b676c2ff28f05520593ed372758812778294eaacb203f01e7afd94597904" exitCode=0 Nov 03 22:44:14.555288 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:14.555202 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" Nov 03 22:44:14.555288 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:14.555229 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" event={"ID":"26ad9218-689e-4989-8fb5-4f311773e668","Type":"ContainerDied","Data":"bac6b676c2ff28f05520593ed372758812778294eaacb203f01e7afd94597904"} Nov 03 22:44:14.555288 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:14.555276 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn" event={"ID":"26ad9218-689e-4989-8fb5-4f311773e668","Type":"ContainerDied","Data":"e5b89ac48c34eb3ad468801275c3d93ff0a0da7cc022ebd59fa87b3ef5f458f3"} Nov 03 22:44:14.565427 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:14.565218 5196 scope.go:122] "RemoveContainer" containerID="64d89eb2bd656eaaf7f579f29005ef02bbb4509284853a0a31e5c27695f36ae1" Nov 03 22:44:14.565537 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:44:14.565515 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64d89eb2bd656eaaf7f579f29005ef02bbb4509284853a0a31e5c27695f36ae1\": container with ID starting with 64d89eb2bd656eaaf7f579f29005ef02bbb4509284853a0a31e5c27695f36ae1 not found: ID does not exist" containerID="64d89eb2bd656eaaf7f579f29005ef02bbb4509284853a0a31e5c27695f36ae1" Nov 03 22:44:14.565646 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:14.565542 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64d89eb2bd656eaaf7f579f29005ef02bbb4509284853a0a31e5c27695f36ae1"} err="failed to get container status \"64d89eb2bd656eaaf7f579f29005ef02bbb4509284853a0a31e5c27695f36ae1\": rpc error: code = NotFound desc = could not find container \"64d89eb2bd656eaaf7f579f29005ef02bbb4509284853a0a31e5c27695f36ae1\": container with ID starting with 64d89eb2bd656eaaf7f579f29005ef02bbb4509284853a0a31e5c27695f36ae1 not found: ID does not exist" Nov 03 22:44:14.565646 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:14.565559 5196 scope.go:122] "RemoveContainer" containerID="bac6b676c2ff28f05520593ed372758812778294eaacb203f01e7afd94597904" Nov 03 22:44:14.574818 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:14.574803 5196 scope.go:122] "RemoveContainer" containerID="bac6b676c2ff28f05520593ed372758812778294eaacb203f01e7afd94597904" Nov 03 22:44:14.575109 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:44:14.575086 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bac6b676c2ff28f05520593ed372758812778294eaacb203f01e7afd94597904\": container with ID starting with bac6b676c2ff28f05520593ed372758812778294eaacb203f01e7afd94597904 not found: ID does not exist" containerID="bac6b676c2ff28f05520593ed372758812778294eaacb203f01e7afd94597904" Nov 03 22:44:14.575177 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:14.575120 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bac6b676c2ff28f05520593ed372758812778294eaacb203f01e7afd94597904"} err="failed to get container status \"bac6b676c2ff28f05520593ed372758812778294eaacb203f01e7afd94597904\": rpc error: code = NotFound desc = could not find container \"bac6b676c2ff28f05520593ed372758812778294eaacb203f01e7afd94597904\": container with ID starting with bac6b676c2ff28f05520593ed372758812778294eaacb203f01e7afd94597904 not found: ID does not exist" Nov 03 22:44:14.773431 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:14.773351 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt"] Nov 03 22:44:14.824632 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:14.824605 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wtfpt"] Nov 03 22:44:14.892001 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:14.891965 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn"] Nov 03 22:44:14.897279 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:14.897255 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vs7zn"] Nov 03 22:44:14.922100 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:14.922074 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:16.817290 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:16.817253 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26ad9218-689e-4989-8fb5-4f311773e668" path="/var/lib/kubelet/pods/26ad9218-689e-4989-8fb5-4f311773e668/volumes" Nov 03 22:44:16.817804 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:16.817788 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b095b36-a492-414f-92bd-f1e3ac60fe7c" path="/var/lib/kubelet/pods/6b095b36-a492-414f-92bd-f1e3ac60fe7c/volumes" Nov 03 22:44:19.728970 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:19.728939 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:21.832831 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:21.832788 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" podUID="26fa51c2-0f2e-482f-9c67-0ea7924d7b3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:21.856165 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:21.856029 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" podUID="9d8b4fed-b4b0-49ce-9238-e23175e90b70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:21.937715 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:21.937678 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" podUID="82749c55-0fca-4a14-b6ba-7df122a6a407" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 03 22:44:22.255027 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:22.250062 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" podUID="abb461c4-9707-4b5c-aba0-793f7bb993e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:22.255027 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:22.250632 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" podUID="34e9cf3a-3456-4835-8933-102ce735cccc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 03 22:44:22.877882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:22.873872 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" podUID="26fa51c2-0f2e-482f-9c67-0ea7924d7b3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:22.877882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:22.874894 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" Nov 03 22:44:22.906882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:22.894226 5196 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" podUID="26fa51c2-0f2e-482f-9c67-0ea7924d7b3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded" Nov 03 22:44:22.957883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:22.935440 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" podUID="9f2e3f52-0664-434f-919b-32736cbf0602" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 22:44:23.329120 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:23.328799 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" podUID="9d8b4fed-b4b0-49ce-9238-e23175e90b70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 03 22:44:23.330602 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:23.330569 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" podUID="82749c55-0fca-4a14-b6ba-7df122a6a407" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 03 22:44:23.345487 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:23.345449 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" podUID="34e9cf3a-3456-4835-8933-102ce735cccc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:23.399884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:23.399424 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" podUID="abb461c4-9707-4b5c-aba0-793f7bb993e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 03 22:44:23.489398 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:23.489363 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" Nov 03 22:44:23.552960 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:23.552932 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" Nov 03 22:44:23.562010 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:23.561986 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" Nov 03 22:44:23.590205 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:23.588962 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" Nov 03 22:44:23.680893 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:23.680537 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" Nov 03 22:44:23.754881 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:23.752688 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" Nov 03 22:44:23.877047 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:23.876934 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" podUID="26fa51c2-0f2e-482f-9c67-0ea7924d7b3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:23.932218 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:23.932191 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" Nov 03 22:44:23.982598 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:23.980477 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" Nov 03 22:44:24.056067 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:24.046647 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" Nov 03 22:44:25.919180 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:44:25.919135 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:44:25.927327 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:25.927300 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:29.980101 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:29.980061 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6974-eead3b"] Nov 03 22:44:29.980810 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:29.980788 5196 cpu_manager.go:414] "containerMap: removing container" podUID="26ad9218-689e-4989-8fb5-4f311773e668" containerName="istio-proxy" Nov 03 22:44:29.980928 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:29.980812 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26ad9218-689e-4989-8fb5-4f311773e668" containerName="istio-proxy" Nov 03 22:44:29.980928 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:29.980850 5196 cpu_manager.go:414] "containerMap: removing container" podUID="6b095b36-a492-414f-92bd-f1e3ac60fe7c" containerName="istio-proxy" Nov 03 22:44:29.980928 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:29.980876 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b095b36-a492-414f-92bd-f1e3ac60fe7c" containerName="istio-proxy" Nov 03 22:44:29.981092 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:29.981033 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="26ad9218-689e-4989-8fb5-4f311773e668" containerName="istio-proxy" Nov 03 22:44:29.981092 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:29.981053 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b095b36-a492-414f-92bd-f1e3ac60fe7c" containerName="istio-proxy" Nov 03 22:44:29.986321 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:29.986300 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6974-eead3b" Nov 03 22:44:30.042178 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:30.042149 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6974-eead3b"] Nov 03 22:44:30.088876 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:30.088828 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jgmv2\"" Nov 03 22:44:30.091630 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:30.091585 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9j2b\" (UniqueName: \"kubernetes.io/projected/64739e84-da6c-45e8-b93c-583e27d841b7-kube-api-access-k9j2b\") pod \"test-curl-6974-eead3b\" (UID: \"64739e84-da6c-45e8-b93c-583e27d841b7\") " pod="opendatahub/test-curl-6974-eead3b" Nov 03 22:44:30.192678 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:30.192646 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9j2b\" (UniqueName: \"kubernetes.io/projected/64739e84-da6c-45e8-b93c-583e27d841b7-kube-api-access-k9j2b\") pod \"test-curl-6974-eead3b\" (UID: \"64739e84-da6c-45e8-b93c-583e27d841b7\") " pod="opendatahub/test-curl-6974-eead3b" Nov 03 22:44:30.254525 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:30.254457 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9j2b\" (UniqueName: \"kubernetes.io/projected/64739e84-da6c-45e8-b93c-583e27d841b7-kube-api-access-k9j2b\") pod \"test-curl-6974-eead3b\" (UID: \"64739e84-da6c-45e8-b93c-583e27d841b7\") " pod="opendatahub/test-curl-6974-eead3b" Nov 03 22:44:30.298678 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:30.298650 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6974-eead3b" Nov 03 22:44:30.682364 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:30.682320 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6974-eead3b"] Nov 03 22:44:30.683495 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:44:30.683467 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64739e84_da6c_45e8_b93c_583e27d841b7.slice/crio-5a7775d387b316db8b001b100b9332ecc635b8741f18e560786d67a541c4c76f WatchSource:0}: Error finding container 5a7775d387b316db8b001b100b9332ecc635b8741f18e560786d67a541c4c76f: Status 404 returned error can't find the container with id 5a7775d387b316db8b001b100b9332ecc635b8741f18e560786d67a541c4c76f Nov 03 22:44:30.685492 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:30.685473 5196 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:44:30.712401 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:30.712369 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:30.997100 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:30.997026 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6974-eead3b" event={"ID":"64739e84-da6c-45e8-b93c-583e27d841b7","Type":"ContainerStarted","Data":"5a7775d387b316db8b001b100b9332ecc635b8741f18e560786d67a541c4c76f"} Nov 03 22:44:34.024389 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:34.024217 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6974-eead3b" event={"ID":"64739e84-da6c-45e8-b93c-583e27d841b7","Type":"ContainerStarted","Data":"6ac4548ef7deb43f1aa913ceeb56ad606a51247a3c2e814bfd17a777cc0cedb9"} Nov 03 22:44:34.140835 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:34.140783 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6974-eead3b" podStartSLOduration=2.6921133619999997 podStartE2EDuration="5.140767573s" totalImagesPullingTime="2.448654211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:34.139678895 +0000 UTC m=+1303.449238188" watchObservedRunningTime="2026-11-03 22:44:34.140767573 +0000 UTC m=+1303.450326903" Nov 03 22:44:35.522183 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:35.522154 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:35.618407 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:35.618368 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6974-eead3b"] Nov 03 22:44:35.623042 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:35.623007 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6974-eead3b"] Nov 03 22:44:35.781815 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:35.781735 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6974-f5f35e"] Nov 03 22:44:35.782370 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:35.782349 5196 cpu_manager.go:414] "containerMap: removing container" podUID="64739e84-da6c-45e8-b93c-583e27d841b7" containerName="test-curl-6974-eead3b" Nov 03 22:44:35.782478 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:35.782371 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64739e84-da6c-45e8-b93c-583e27d841b7" containerName="test-curl-6974-eead3b" Nov 03 22:44:35.782549 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:35.782535 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="64739e84-da6c-45e8-b93c-583e27d841b7" containerName="test-curl-6974-eead3b" Nov 03 22:44:35.785967 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:35.785847 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6974-f5f35e" Nov 03 22:44:35.839733 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:35.839705 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6974-f5f35e"] Nov 03 22:44:35.974513 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:35.974473 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxdwh\" (UniqueName: \"kubernetes.io/projected/4d7be2ea-6e0c-4176-ab80-71744dc21ca4-kube-api-access-mxdwh\") pod \"test-curl-6974-f5f35e\" (UID: \"4d7be2ea-6e0c-4176-ab80-71744dc21ca4\") " pod="opendatahub/test-curl-6974-f5f35e" Nov 03 22:44:36.038902 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:36.038771 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6974-eead3b" podUID="64739e84-da6c-45e8-b93c-583e27d841b7" containerName="test-curl-6974-eead3b" containerID="cri-o://6ac4548ef7deb43f1aa913ceeb56ad606a51247a3c2e814bfd17a777cc0cedb9" gracePeriod=2 Nov 03 22:44:36.075611 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:36.075573 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxdwh\" (UniqueName: \"kubernetes.io/projected/4d7be2ea-6e0c-4176-ab80-71744dc21ca4-kube-api-access-mxdwh\") pod \"test-curl-6974-f5f35e\" (UID: \"4d7be2ea-6e0c-4176-ab80-71744dc21ca4\") " pod="opendatahub/test-curl-6974-f5f35e" Nov 03 22:44:36.091266 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:44:36.091232 5196 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6974-eead3b\" is forbidden: User \"system:node:ip-10-0-143-132.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-132.us-west-1.compute.internal' and this object" podUID="64739e84-da6c-45e8-b93c-583e27d841b7" pod="opendatahub/test-curl-6974-eead3b" Nov 03 22:44:36.137647 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:36.137625 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxdwh\" (UniqueName: \"kubernetes.io/projected/4d7be2ea-6e0c-4176-ab80-71744dc21ca4-kube-api-access-mxdwh\") pod \"test-curl-6974-f5f35e\" (UID: \"4d7be2ea-6e0c-4176-ab80-71744dc21ca4\") " pod="opendatahub/test-curl-6974-f5f35e" Nov 03 22:44:36.398204 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:36.398176 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6974-f5f35e" Nov 03 22:44:36.775093 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:36.775071 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6974-f5f35e"] Nov 03 22:44:36.776910 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:44:36.776878 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d7be2ea_6e0c_4176_ab80_71744dc21ca4.slice/crio-852f77e32bd912858a3ed55e211f6ba41caa3e2fa6dd6047c437a3c05b703cb1 WatchSource:0}: Error finding container 852f77e32bd912858a3ed55e211f6ba41caa3e2fa6dd6047c437a3c05b703cb1: Status 404 returned error can't find the container with id 852f77e32bd912858a3ed55e211f6ba41caa3e2fa6dd6047c437a3c05b703cb1 Nov 03 22:44:37.048375 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:37.048270 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6974-f5f35e" event={"ID":"4d7be2ea-6e0c-4176-ab80-71744dc21ca4","Type":"ContainerStarted","Data":"7edf7881f4742b1910e126990db2b8ba49a8679ae166ce3190f9cddc8a332501"} Nov 03 22:44:37.048375 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:37.048326 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6974-f5f35e" event={"ID":"4d7be2ea-6e0c-4176-ab80-71744dc21ca4","Type":"ContainerStarted","Data":"852f77e32bd912858a3ed55e211f6ba41caa3e2fa6dd6047c437a3c05b703cb1"} Nov 03 22:44:37.167259 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:37.167205 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6974-f5f35e" podStartSLOduration=2.167191565 podStartE2EDuration="2.167191565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:37.163574118 +0000 UTC m=+1306.473133413" watchObservedRunningTime="2026-11-03 22:44:37.167191565 +0000 UTC m=+1306.476750857" Nov 03 22:44:37.661695 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:37.661654 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:37.825292 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:37.825251 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:38.313214 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:38.313160 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6974-eead3b" Nov 03 22:44:38.399226 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:38.399189 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64739e84-da6c-45e8-b93c-583e27d841b7-kube-api-access-k9j2b\" (UniqueName: \"kubernetes.io/projected/64739e84-da6c-45e8-b93c-583e27d841b7-kube-api-access-k9j2b\") pod \"64739e84-da6c-45e8-b93c-583e27d841b7\" (UID: \"64739e84-da6c-45e8-b93c-583e27d841b7\") " Nov 03 22:44:38.401303 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:38.401270 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64739e84-da6c-45e8-b93c-583e27d841b7-kube-api-access-k9j2b" pod "64739e84-da6c-45e8-b93c-583e27d841b7" (UID: "64739e84-da6c-45e8-b93c-583e27d841b7"). InnerVolumeSpecName "kube-api-access-k9j2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:38.426740 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:38.426685 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:38.500802 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:38.500773 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9j2b\" (UniqueName: \"kubernetes.io/projected/64739e84-da6c-45e8-b93c-583e27d841b7-kube-api-access-k9j2b\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:38.712768 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:38.712735 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6974-f5f35e"] Nov 03 22:44:38.715599 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:38.715575 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6974-f5f35e"] Nov 03 22:44:38.822352 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:38.822321 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64739e84-da6c-45e8-b93c-583e27d841b7" path="/var/lib/kubelet/pods/64739e84-da6c-45e8-b93c-583e27d841b7/volumes" Nov 03 22:44:39.063984 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:39.063896 5196 generic.go:356] "Generic (PLEG): container finished" podID="64739e84-da6c-45e8-b93c-583e27d841b7" containerID="6ac4548ef7deb43f1aa913ceeb56ad606a51247a3c2e814bfd17a777cc0cedb9" exitCode=137 Nov 03 22:44:39.063984 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:39.063963 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6974-eead3b" Nov 03 22:44:39.063984 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:39.063974 5196 scope.go:122] "RemoveContainer" containerID="6ac4548ef7deb43f1aa913ceeb56ad606a51247a3c2e814bfd17a777cc0cedb9" Nov 03 22:44:39.064498 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:39.064169 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6974-f5f35e" podUID="4d7be2ea-6e0c-4176-ab80-71744dc21ca4" containerName="test-curl-6974-f5f35e" containerID="cri-o://7edf7881f4742b1910e126990db2b8ba49a8679ae166ce3190f9cddc8a332501" gracePeriod=2 Nov 03 22:44:39.073749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:39.073727 5196 scope.go:122] "RemoveContainer" containerID="6ac4548ef7deb43f1aa913ceeb56ad606a51247a3c2e814bfd17a777cc0cedb9" Nov 03 22:44:39.074043 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:44:39.074022 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ac4548ef7deb43f1aa913ceeb56ad606a51247a3c2e814bfd17a777cc0cedb9\": container with ID starting with 6ac4548ef7deb43f1aa913ceeb56ad606a51247a3c2e814bfd17a777cc0cedb9 not found: ID does not exist" containerID="6ac4548ef7deb43f1aa913ceeb56ad606a51247a3c2e814bfd17a777cc0cedb9" Nov 03 22:44:39.074100 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:39.074051 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ac4548ef7deb43f1aa913ceeb56ad606a51247a3c2e814bfd17a777cc0cedb9"} err="failed to get container status \"6ac4548ef7deb43f1aa913ceeb56ad606a51247a3c2e814bfd17a777cc0cedb9\": rpc error: code = NotFound desc = could not find container \"6ac4548ef7deb43f1aa913ceeb56ad606a51247a3c2e814bfd17a777cc0cedb9\": container with ID starting with 6ac4548ef7deb43f1aa913ceeb56ad606a51247a3c2e814bfd17a777cc0cedb9 not found: ID does not exist" Nov 03 22:44:39.546185 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:39.546153 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:40.258709 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:40.256606 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:40.410216 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:40.410182 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:40.569662 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:40.569570 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:40.918234 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:40.918200 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:41.060765 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:41.060717 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:41.350368 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:41.350343 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6974-f5f35e" Nov 03 22:44:41.507576 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:41.507489 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d7be2ea-6e0c-4176-ab80-71744dc21ca4-kube-api-access-mxdwh\" (UniqueName: \"kubernetes.io/projected/4d7be2ea-6e0c-4176-ab80-71744dc21ca4-kube-api-access-mxdwh\") pod \"4d7be2ea-6e0c-4176-ab80-71744dc21ca4\" (UID: \"4d7be2ea-6e0c-4176-ab80-71744dc21ca4\") " Nov 03 22:44:41.509938 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:41.509905 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d7be2ea-6e0c-4176-ab80-71744dc21ca4-kube-api-access-mxdwh" pod "4d7be2ea-6e0c-4176-ab80-71744dc21ca4" (UID: "4d7be2ea-6e0c-4176-ab80-71744dc21ca4"). InnerVolumeSpecName "kube-api-access-mxdwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:41.609245 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:41.609211 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxdwh\" (UniqueName: \"kubernetes.io/projected/4d7be2ea-6e0c-4176-ab80-71744dc21ca4-kube-api-access-mxdwh\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:44:42.091847 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:42.091812 5196 generic.go:356] "Generic (PLEG): container finished" podID="4d7be2ea-6e0c-4176-ab80-71744dc21ca4" containerID="7edf7881f4742b1910e126990db2b8ba49a8679ae166ce3190f9cddc8a332501" exitCode=137 Nov 03 22:44:42.092042 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:42.091874 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6974-f5f35e" Nov 03 22:44:42.092042 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:42.091908 5196 scope.go:122] "RemoveContainer" containerID="7edf7881f4742b1910e126990db2b8ba49a8679ae166ce3190f9cddc8a332501" Nov 03 22:44:42.102259 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:42.102243 5196 scope.go:122] "RemoveContainer" containerID="7edf7881f4742b1910e126990db2b8ba49a8679ae166ce3190f9cddc8a332501" Nov 03 22:44:42.102524 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:44:42.102506 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7edf7881f4742b1910e126990db2b8ba49a8679ae166ce3190f9cddc8a332501\": container with ID starting with 7edf7881f4742b1910e126990db2b8ba49a8679ae166ce3190f9cddc8a332501 not found: ID does not exist" containerID="7edf7881f4742b1910e126990db2b8ba49a8679ae166ce3190f9cddc8a332501" Nov 03 22:44:42.102574 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:42.102543 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7edf7881f4742b1910e126990db2b8ba49a8679ae166ce3190f9cddc8a332501"} err="failed to get container status \"7edf7881f4742b1910e126990db2b8ba49a8679ae166ce3190f9cddc8a332501\": rpc error: code = NotFound desc = could not find container \"7edf7881f4742b1910e126990db2b8ba49a8679ae166ce3190f9cddc8a332501\": container with ID starting with 7edf7881f4742b1910e126990db2b8ba49a8679ae166ce3190f9cddc8a332501 not found: ID does not exist" Nov 03 22:44:42.152978 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:44:42.152950 5196 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6974-f5f35e\" is forbidden: User \"system:node:ip-10-0-143-132.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-132.us-west-1.compute.internal' and this object" podUID="4d7be2ea-6e0c-4176-ab80-71744dc21ca4" pod="opendatahub/test-curl-6974-f5f35e" Nov 03 22:44:42.814781 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:42.814746 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d7be2ea-6e0c-4176-ab80-71744dc21ca4" path="/var/lib/kubelet/pods/4d7be2ea-6e0c-4176-ab80-71744dc21ca4/volumes" Nov 03 22:44:43.754316 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:43.754287 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:43.919183 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:43.919149 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:44.073697 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:44.073614 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:45.252040 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:45.252008 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:45.402242 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:45.402213 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:45.578669 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:45.578573 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:45.719524 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:45.719473 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:45.868110 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:45.868081 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:46.579098 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:46.579062 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:46.979894 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:46.979847 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:50.092834 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:50.092800 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:50.240058 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:50.240023 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:50.384345 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:50.384263 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:50.617547 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:50.617509 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:53.408737 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:53.408707 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:53.702500 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:53.702413 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:56.208140 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:44:56.208099 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:44:56.639426 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:56.639360 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:56.735886 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:56.720442 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" podUID="9d8b4fed-b4b0-49ce-9238-e23175e90b70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 03 22:44:56.792739 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:56.788957 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" podUID="abb461c4-9707-4b5c-aba0-793f7bb993e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:56.792739 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:56.789956 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" podUID="9f2e3f52-0664-434f-919b-32736cbf0602" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 22:44:56.951051 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:56.940227 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:56.955522 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:56.953731 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" podUID="26fa51c2-0f2e-482f-9c67-0ea7924d7b3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:57.075881 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:57.075057 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" podUID="34e9cf3a-3456-4835-8933-102ce735cccc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 03 22:44:57.118208 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:57.116385 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:44:57.138915 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:57.129266 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" podUID="82749c55-0fca-4a14-b6ba-7df122a6a407" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:57.741017 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:57.740967 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" podUID="9d8b4fed-b4b0-49ce-9238-e23175e90b70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 03 22:44:57.760164 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:57.760126 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" Nov 03 22:44:57.792051 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:57.791999 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" podUID="abb461c4-9707-4b5c-aba0-793f7bb993e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 03 22:44:57.798764 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:57.798724 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" Nov 03 22:44:57.897143 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:57.798991 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" podUID="9f2e3f52-0664-434f-919b-32736cbf0602" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:57.968521 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:57.967188 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" podUID="26fa51c2-0f2e-482f-9c67-0ea7924d7b3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:58.004311 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:58.004224 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" Nov 03 22:44:58.004588 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:58.004571 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" Nov 03 22:44:58.022123 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:58.021967 5196 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:44:58.022123 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:58.022036 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" podUID="b0aafab2-abe8-4dd8-81ea-71278482df2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:58.034886 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:58.033965 5196 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:44:58.034886 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:58.034031 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" podUID="03244edf-87f5-4237-be87-8a523c4d96b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:58.215088 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:58.150110 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" podUID="82749c55-0fca-4a14-b6ba-7df122a6a407" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:58.215088 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:58.151244 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" Nov 03 22:44:58.329461 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:58.293084 5196 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" podUID="82749c55-0fca-4a14-b6ba-7df122a6a407" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:58.658929 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:58.607134 5196 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:44:58.658929 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:58.607202 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" podUID="c9bf76ea-2e4e-46ee-9c23-3f084e9f8475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:58.748890 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:58.725121 5196 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" podUID="9f2e3f52-0664-434f-919b-32736cbf0602" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded" Nov 03 22:44:58.748890 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:58.742985 5196 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:44:58.748890 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:58.743044 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" podUID="c974ac3a-2c08-40df-a4a1-c62b3d3fa040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:58.905211 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:58.885170 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" podUID="9d8b4fed-b4b0-49ce-9238-e23175e90b70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:58.923201 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:58.905502 5196 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" podUID="34e9cf3a-3456-4835-8933-102ce735cccc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:58.923201 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:58.916425 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" podUID="9f2e3f52-0664-434f-919b-32736cbf0602" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 22:44:58.929528 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:58.929487 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" podUID="abb461c4-9707-4b5c-aba0-793f7bb993e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 03 22:44:59.008063 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:59.004316 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" podUID="26fa51c2-0f2e-482f-9c67-0ea7924d7b3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 03 22:44:59.120593 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:59.120550 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" podUID="34e9cf3a-3456-4835-8933-102ce735cccc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:59.153515 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:59.153463 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" podUID="82749c55-0fca-4a14-b6ba-7df122a6a407" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:59.278757 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:59.278664 5196 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:44:59.279151 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:59.279108 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" podUID="cea0ff59-b9f4-49e6-995e-0c568ca8dd9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:59.397486 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:59.397455 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" Nov 03 22:44:59.759106 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:59.759058 5196 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:44:59.759800 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:59.759693 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" podUID="734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:59.939895 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:59.938251 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" podUID="9d8b4fed-b4b0-49ce-9238-e23175e90b70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:59.971896 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:44:59.969687 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" podUID="abb461c4-9707-4b5c-aba0-793f7bb993e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:00.030939 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:00.030785 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" podUID="26fa51c2-0f2e-482f-9c67-0ea7924d7b3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:00.146894 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:00.142339 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" podUID="34e9cf3a-3456-4835-8933-102ce735cccc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:00.146894 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:00.142669 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" Nov 03 22:45:00.179896 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:00.177786 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" podUID="82749c55-0fca-4a14-b6ba-7df122a6a407" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 03 22:45:00.233887 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:00.231652 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895765-jdlrt"] Nov 03 22:45:00.233887 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:00.232517 5196 cpu_manager.go:414] "containerMap: removing container" podUID="4d7be2ea-6e0c-4176-ab80-71744dc21ca4" containerName="test-curl-6974-f5f35e" Nov 03 22:45:00.233887 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:00.232538 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d7be2ea-6e0c-4176-ab80-71744dc21ca4" containerName="test-curl-6974-f5f35e" Nov 03 22:45:00.233887 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:00.232668 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d7be2ea-6e0c-4176-ab80-71744dc21ca4" containerName="test-curl-6974-f5f35e" Nov 03 22:45:00.327899 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:00.327787 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895765-jdlrt" Nov 03 22:45:00.400893 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:00.389757 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p2s2g\"" Nov 03 22:45:00.474913 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:00.461156 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88a9a7b-033c-403d-b1a5-5a2f71d0c918-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-7540da-29895765-jdlrt\" (UID: \"b88a9a7b-033c-403d-b1a5-5a2f71d0c918\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895765-jdlrt" Nov 03 22:45:00.474913 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:00.461202 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqxwg\" (UniqueName: \"kubernetes.io/projected/b88a9a7b-033c-403d-b1a5-5a2f71d0c918-kube-api-access-nqxwg\") pod \"maas-api-key-cleanup-e2e-worker-w6-7540da-29895765-jdlrt\" (UID: \"b88a9a7b-033c-403d-b1a5-5a2f71d0c918\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895765-jdlrt" Nov 03 22:45:00.534333 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:00.533907 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895765-jdlrt"] Nov 03 22:45:00.594878 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:00.593708 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88a9a7b-033c-403d-b1a5-5a2f71d0c918-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-7540da-29895765-jdlrt\" (UID: \"b88a9a7b-033c-403d-b1a5-5a2f71d0c918\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895765-jdlrt" Nov 03 22:45:00.594878 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:00.593756 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqxwg\" (UniqueName: \"kubernetes.io/projected/b88a9a7b-033c-403d-b1a5-5a2f71d0c918-kube-api-access-nqxwg\") pod \"maas-api-key-cleanup-e2e-worker-w6-7540da-29895765-jdlrt\" (UID: \"b88a9a7b-033c-403d-b1a5-5a2f71d0c918\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895765-jdlrt" Nov 03 22:45:00.595194 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:00.595133 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88a9a7b-033c-403d-b1a5-5a2f71d0c918-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-7540da-29895765-jdlrt\" (UID: \"b88a9a7b-033c-403d-b1a5-5a2f71d0c918\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895765-jdlrt" Nov 03 22:45:00.722583 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:00.722551 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqxwg\" (UniqueName: \"kubernetes.io/projected/b88a9a7b-033c-403d-b1a5-5a2f71d0c918-kube-api-access-nqxwg\") pod \"maas-api-key-cleanup-e2e-worker-w6-7540da-29895765-jdlrt\" (UID: \"b88a9a7b-033c-403d-b1a5-5a2f71d0c918\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895765-jdlrt" Nov 03 22:45:01.010942 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:00.984032 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" podUID="9f2e3f52-0664-434f-919b-32736cbf0602" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:01.010942 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:00.986227 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" podUID="abb461c4-9707-4b5c-aba0-793f7bb993e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:01.010942 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:00.987050 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" podUID="9d8b4fed-b4b0-49ce-9238-e23175e90b70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 03 22:45:01.054118 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:01.054084 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895765-jdlrt" Nov 03 22:45:01.207769 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:01.207723 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" podUID="34e9cf3a-3456-4835-8933-102ce735cccc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 03 22:45:01.304804 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:01.304707 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" podUID="82749c55-0fca-4a14-b6ba-7df122a6a407" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 03 22:45:01.331717 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:01.331676 5196 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" podUID="abb461c4-9707-4b5c-aba0-793f7bb993e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded" Nov 03 22:45:01.368158 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:01.368107 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" podUID="26fa51c2-0f2e-482f-9c67-0ea7924d7b3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:01.483605 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:01.483572 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" Nov 03 22:45:01.496598 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:01.496572 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" Nov 03 22:45:01.679065 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:01.678679 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" Nov 03 22:45:01.681577 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:01.681549 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" Nov 03 22:45:01.684998 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:01.684439 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" Nov 03 22:45:02.089324 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:02.089294 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895765-jdlrt"] Nov 03 22:45:02.090279 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:45:02.090248 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb88a9a7b_033c_403d_b1a5_5a2f71d0c918.slice/crio-5388aa6eca0fc186567f6fb1f5b379a1de092b151d694a53611ec5321be8b99b WatchSource:0}: Error finding container 5388aa6eca0fc186567f6fb1f5b379a1de092b151d694a53611ec5321be8b99b: Status 404 returned error can't find the container with id 5388aa6eca0fc186567f6fb1f5b379a1de092b151d694a53611ec5321be8b99b Nov 03 22:45:02.890318 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:02.890276 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895765-jdlrt" event={"ID":"b88a9a7b-033c-403d-b1a5-5a2f71d0c918","Type":"ContainerStarted","Data":"538a13d1e56efb40aa7d171d9567e19a69ab987823b1b5e8bddcac44d9b5ba17"} Nov 03 22:45:02.890593 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:02.890563 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895765-jdlrt" event={"ID":"b88a9a7b-033c-403d-b1a5-5a2f71d0c918","Type":"ContainerStarted","Data":"5388aa6eca0fc186567f6fb1f5b379a1de092b151d694a53611ec5321be8b99b"} Nov 03 22:45:03.008186 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:03.008137 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895765-jdlrt" podStartSLOduration=3.008116962 podStartE2EDuration="3.008116962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:03.006283748 +0000 UTC m=+1332.315843042" watchObservedRunningTime="2026-11-03 22:45:03.008116962 +0000 UTC m=+1332.317676255" Nov 03 22:45:03.142787 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:03.142723 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:03.645925 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:03.645892 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:03.903638 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:03.903558 5196 generic.go:356] "Generic (PLEG): container finished" podID="b88a9a7b-033c-403d-b1a5-5a2f71d0c918" containerID="538a13d1e56efb40aa7d171d9567e19a69ab987823b1b5e8bddcac44d9b5ba17" exitCode=60 Nov 03 22:45:03.903803 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:03.903635 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895765-jdlrt" event={"ID":"b88a9a7b-033c-403d-b1a5-5a2f71d0c918","Type":"ContainerDied","Data":"538a13d1e56efb40aa7d171d9567e19a69ab987823b1b5e8bddcac44d9b5ba17"} Nov 03 22:45:03.904249 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:03.904233 5196 scope.go:122] "RemoveContainer" containerID="538a13d1e56efb40aa7d171d9567e19a69ab987823b1b5e8bddcac44d9b5ba17" Nov 03 22:45:04.058231 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:04.058201 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:04.217434 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:04.217400 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:04.367083 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:04.367024 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:04.807569 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:04.807538 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:04.912949 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:04.912917 5196 generic.go:356] "Generic (PLEG): container finished" podID="b88a9a7b-033c-403d-b1a5-5a2f71d0c918" containerID="31ce203290525bd35f57e8e12d28379e251a763a447f359915187e61a2186e4b" exitCode=60 Nov 03 22:45:04.913078 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:04.913030 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895765-jdlrt" event={"ID":"b88a9a7b-033c-403d-b1a5-5a2f71d0c918","Type":"ContainerDied","Data":"31ce203290525bd35f57e8e12d28379e251a763a447f359915187e61a2186e4b"} Nov 03 22:45:04.913153 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:04.913092 5196 scope.go:122] "RemoveContainer" containerID="538a13d1e56efb40aa7d171d9567e19a69ab987823b1b5e8bddcac44d9b5ba17" Nov 03 22:45:04.913335 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:04.913319 5196 scope.go:122] "RemoveContainer" containerID="31ce203290525bd35f57e8e12d28379e251a763a447f359915187e61a2186e4b" Nov 03 22:45:04.913705 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:45:04.913676 5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-7540da-29895765-jdlrt_odh-ai-gateway-infra(b88a9a7b-033c-403d-b1a5-5a2f71d0c918)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895765-jdlrt" podUID="b88a9a7b-033c-403d-b1a5-5a2f71d0c918" Nov 03 22:45:04.996579 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:04.996542 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:05.150385 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:05.150355 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:05.453348 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:05.453259 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:05.602582 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:05.602551 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:05.922765 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:05.922741 5196 scope.go:122] "RemoveContainer" containerID="31ce203290525bd35f57e8e12d28379e251a763a447f359915187e61a2186e4b" Nov 03 22:45:05.923150 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:45:05.923126 5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-7540da-29895765-jdlrt_odh-ai-gateway-infra(b88a9a7b-033c-403d-b1a5-5a2f71d0c918)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895765-jdlrt" podUID="b88a9a7b-033c-403d-b1a5-5a2f71d0c918" Nov 03 22:45:06.822383 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:06.822350 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:06.967163 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:06.967137 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:08.219274 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:08.219236 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:09.017568 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:09.017538 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:13.004015 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:13.003982 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:18.808478 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:18.808445 5196 scope.go:122] "RemoveContainer" containerID="31ce203290525bd35f57e8e12d28379e251a763a447f359915187e61a2186e4b" Nov 03 22:45:19.642179 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:19.642117 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:19.792993 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:19.792962 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:20.012404 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.012314 5196 generic.go:356] "Generic (PLEG): container finished" podID="b88a9a7b-033c-403d-b1a5-5a2f71d0c918" containerID="c4dbd33f58916b21be36281dba5e31a380f7c58d7b1eab34b560fccb2a1ff23e" exitCode=60 Nov 03 22:45:20.012404 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.012392 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895765-jdlrt" event={"ID":"b88a9a7b-033c-403d-b1a5-5a2f71d0c918","Type":"ContainerDied","Data":"c4dbd33f58916b21be36281dba5e31a380f7c58d7b1eab34b560fccb2a1ff23e"} Nov 03 22:45:20.012911 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.012457 5196 scope.go:122] "RemoveContainer" containerID="31ce203290525bd35f57e8e12d28379e251a763a447f359915187e61a2186e4b" Nov 03 22:45:20.012911 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.012893 5196 scope.go:122] "RemoveContainer" containerID="c4dbd33f58916b21be36281dba5e31a380f7c58d7b1eab34b560fccb2a1ff23e" Nov 03 22:45:20.013262 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:45:20.013233 5196 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-7540da-29895765-jdlrt_odh-ai-gateway-infra(b88a9a7b-033c-403d-b1a5-5a2f71d0c918)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895765-jdlrt" podUID="b88a9a7b-033c-403d-b1a5-5a2f71d0c918" Nov 03 22:45:20.188024 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.187993 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g"] Nov 03 22:45:20.191059 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.191040 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" Nov 03 22:45:20.206390 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.206366 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq"] Nov 03 22:45:20.209064 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.209045 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" Nov 03 22:45:20.254368 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.254334 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g"] Nov 03 22:45:20.320741 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.320651 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmkb6\" (UniqueName: \"kubernetes.io/projected/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-kube-api-access-gmkb6\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq\" (UID: \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" Nov 03 22:45:20.320741 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.320718 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cacf0851-d190-4ef3-8df0-e181b4444430-istio-data\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g\" (UID: \"cacf0851-d190-4ef3-8df0-e181b4444430\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" Nov 03 22:45:20.321009 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.320758 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-istio-token\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq\" (UID: \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" Nov 03 22:45:20.321009 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.320800 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cacf0851-d190-4ef3-8df0-e181b4444430-istiod-ca-cert\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g\" (UID: \"cacf0851-d190-4ef3-8df0-e181b4444430\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" Nov 03 22:45:20.321009 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.320922 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacf0851-d190-4ef3-8df0-e181b4444430-credential-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g\" (UID: \"cacf0851-d190-4ef3-8df0-e181b4444430\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" Nov 03 22:45:20.321009 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.320986 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cacf0851-d190-4ef3-8df0-e181b4444430-istio-podinfo\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g\" (UID: \"cacf0851-d190-4ef3-8df0-e181b4444430\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" Nov 03 22:45:20.321204 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.321078 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbs2j\" (UniqueName: \"kubernetes.io/projected/cacf0851-d190-4ef3-8df0-e181b4444430-kube-api-access-jbs2j\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g\" (UID: \"cacf0851-d190-4ef3-8df0-e181b4444430\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" Nov 03 22:45:20.321204 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.321134 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-istio-envoy\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq\" (UID: \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" Nov 03 22:45:20.321204 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.321174 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-istiod-ca-cert\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq\" (UID: \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" Nov 03 22:45:20.321339 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.321232 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cacf0851-d190-4ef3-8df0-e181b4444430-istio-envoy\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g\" (UID: \"cacf0851-d190-4ef3-8df0-e181b4444430\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" Nov 03 22:45:20.321339 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.321265 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cacf0851-d190-4ef3-8df0-e181b4444430-istio-token\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g\" (UID: \"cacf0851-d190-4ef3-8df0-e181b4444430\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" Nov 03 22:45:20.321339 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.321325 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-workload-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq\" (UID: \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" Nov 03 22:45:20.321481 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.321375 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacf0851-d190-4ef3-8df0-e181b4444430-workload-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g\" (UID: \"cacf0851-d190-4ef3-8df0-e181b4444430\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" Nov 03 22:45:20.321481 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.321422 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-credential-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq\" (UID: \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" Nov 03 22:45:20.321580 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.321479 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cacf0851-d190-4ef3-8df0-e181b4444430-workload-certs\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g\" (UID: \"cacf0851-d190-4ef3-8df0-e181b4444430\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" Nov 03 22:45:20.321580 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.321529 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-workload-certs\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq\" (UID: \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" Nov 03 22:45:20.321688 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.321576 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-istio-data\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq\" (UID: \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" Nov 03 22:45:20.321688 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.321657 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-istio-podinfo\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq\" (UID: \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" Nov 03 22:45:20.327426 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.327395 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq"] Nov 03 22:45:20.423016 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.422988 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-istio-envoy\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq\" (UID: \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" Nov 03 22:45:20.423176 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.423026 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-istiod-ca-cert\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq\" (UID: \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" Nov 03 22:45:20.423176 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.423070 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cacf0851-d190-4ef3-8df0-e181b4444430-istio-envoy\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g\" (UID: \"cacf0851-d190-4ef3-8df0-e181b4444430\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" Nov 03 22:45:20.423176 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.423088 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cacf0851-d190-4ef3-8df0-e181b4444430-istio-token\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g\" (UID: \"cacf0851-d190-4ef3-8df0-e181b4444430\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" Nov 03 22:45:20.423176 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.423124 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-workload-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq\" (UID: \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" Nov 03 22:45:20.423176 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.423168 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacf0851-d190-4ef3-8df0-e181b4444430-workload-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g\" (UID: \"cacf0851-d190-4ef3-8df0-e181b4444430\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" Nov 03 22:45:20.423459 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.423218 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-credential-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq\" (UID: \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" Nov 03 22:45:20.423459 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.423269 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cacf0851-d190-4ef3-8df0-e181b4444430-workload-certs\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g\" (UID: \"cacf0851-d190-4ef3-8df0-e181b4444430\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" Nov 03 22:45:20.423459 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.423312 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-workload-certs\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq\" (UID: \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" Nov 03 22:45:20.423459 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.423356 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-istio-data\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq\" (UID: \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" Nov 03 22:45:20.423459 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.423394 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-istio-podinfo\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq\" (UID: \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" Nov 03 22:45:20.423459 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.423431 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmkb6\" (UniqueName: \"kubernetes.io/projected/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-kube-api-access-gmkb6\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq\" (UID: \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" Nov 03 22:45:20.423786 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.423475 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cacf0851-d190-4ef3-8df0-e181b4444430-istio-data\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g\" (UID: \"cacf0851-d190-4ef3-8df0-e181b4444430\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" Nov 03 22:45:20.423786 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.423506 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-istio-token\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq\" (UID: \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" Nov 03 22:45:20.423786 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.423546 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-workload-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq\" (UID: \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" Nov 03 22:45:20.423786 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.423557 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cacf0851-d190-4ef3-8df0-e181b4444430-istiod-ca-cert\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g\" (UID: \"cacf0851-d190-4ef3-8df0-e181b4444430\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" Nov 03 22:45:20.423786 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.423609 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacf0851-d190-4ef3-8df0-e181b4444430-credential-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g\" (UID: \"cacf0851-d190-4ef3-8df0-e181b4444430\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" Nov 03 22:45:20.423786 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.423638 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cacf0851-d190-4ef3-8df0-e181b4444430-istio-podinfo\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g\" (UID: \"cacf0851-d190-4ef3-8df0-e181b4444430\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" Nov 03 22:45:20.423786 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.423678 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbs2j\" (UniqueName: \"kubernetes.io/projected/cacf0851-d190-4ef3-8df0-e181b4444430-kube-api-access-jbs2j\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g\" (UID: \"cacf0851-d190-4ef3-8df0-e181b4444430\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" Nov 03 22:45:20.423786 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.423674 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-credential-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq\" (UID: \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" Nov 03 22:45:20.423786 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.423716 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-workload-certs\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq\" (UID: \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" Nov 03 22:45:20.423786 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.423786 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacf0851-d190-4ef3-8df0-e181b4444430-workload-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g\" (UID: \"cacf0851-d190-4ef3-8df0-e181b4444430\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" Nov 03 22:45:20.424360 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.423947 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-istiod-ca-cert\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq\" (UID: \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" Nov 03 22:45:20.424360 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.423998 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacf0851-d190-4ef3-8df0-e181b4444430-credential-socket\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g\" (UID: \"cacf0851-d190-4ef3-8df0-e181b4444430\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" Nov 03 22:45:20.424360 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.424188 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-istio-data\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq\" (UID: \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" Nov 03 22:45:20.424360 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.424265 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cacf0851-d190-4ef3-8df0-e181b4444430-workload-certs\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g\" (UID: \"cacf0851-d190-4ef3-8df0-e181b4444430\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" Nov 03 22:45:20.424570 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.424378 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cacf0851-d190-4ef3-8df0-e181b4444430-istio-data\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g\" (UID: \"cacf0851-d190-4ef3-8df0-e181b4444430\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" Nov 03 22:45:20.424727 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.424705 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cacf0851-d190-4ef3-8df0-e181b4444430-istiod-ca-cert\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g\" (UID: \"cacf0851-d190-4ef3-8df0-e181b4444430\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" Nov 03 22:45:20.425800 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.425779 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-istio-envoy\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq\" (UID: \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" Nov 03 22:45:20.426120 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.426102 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cacf0851-d190-4ef3-8df0-e181b4444430-istio-envoy\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g\" (UID: \"cacf0851-d190-4ef3-8df0-e181b4444430\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" Nov 03 22:45:20.426809 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.426791 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-istio-podinfo\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq\" (UID: \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" Nov 03 22:45:20.426925 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.426820 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cacf0851-d190-4ef3-8df0-e181b4444430-istio-podinfo\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g\" (UID: \"cacf0851-d190-4ef3-8df0-e181b4444430\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" Nov 03 22:45:20.486479 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.486451 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cacf0851-d190-4ef3-8df0-e181b4444430-istio-token\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g\" (UID: \"cacf0851-d190-4ef3-8df0-e181b4444430\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" Nov 03 22:45:20.486656 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.486637 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmkb6\" (UniqueName: \"kubernetes.io/projected/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-kube-api-access-gmkb6\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq\" (UID: \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" Nov 03 22:45:20.486699 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.486640 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbs2j\" (UniqueName: \"kubernetes.io/projected/cacf0851-d190-4ef3-8df0-e181b4444430-kube-api-access-jbs2j\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g\" (UID: \"cacf0851-d190-4ef3-8df0-e181b4444430\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" Nov 03 22:45:20.486699 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.486685 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-istio-token\") pod \"e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq\" (UID: \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\") " pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" Nov 03 22:45:20.504118 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.504096 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" Nov 03 22:45:20.520746 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.520722 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" Nov 03 22:45:20.773525 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.773492 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv"] Nov 03 22:45:20.777096 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.777073 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" Nov 03 22:45:20.844588 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.844558 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv"] Nov 03 22:45:20.929889 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.929830 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdc73375-64fa-42d8-8ba8-23a936fd1f06-credential-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv\" (UID: \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" Nov 03 22:45:20.929889 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.929882 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdc73375-64fa-42d8-8ba8-23a936fd1f06-istio-data\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv\" (UID: \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" Nov 03 22:45:20.930099 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.929992 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdc73375-64fa-42d8-8ba8-23a936fd1f06-istio-token\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv\" (UID: \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" Nov 03 22:45:20.930099 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.930013 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdc73375-64fa-42d8-8ba8-23a936fd1f06-workload-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv\" (UID: \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" Nov 03 22:45:20.930276 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.930257 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdc73375-64fa-42d8-8ba8-23a936fd1f06-istiod-ca-cert\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv\" (UID: \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" Nov 03 22:45:20.930312 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.930287 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdc73375-64fa-42d8-8ba8-23a936fd1f06-workload-certs\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv\" (UID: \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" Nov 03 22:45:20.930370 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.930359 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdc73375-64fa-42d8-8ba8-23a936fd1f06-istio-envoy\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv\" (UID: \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" Nov 03 22:45:20.930406 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.930382 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfxgj\" (UniqueName: \"kubernetes.io/projected/bdc73375-64fa-42d8-8ba8-23a936fd1f06-kube-api-access-xfxgj\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv\" (UID: \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" Nov 03 22:45:20.930437 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.930422 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdc73375-64fa-42d8-8ba8-23a936fd1f06-istio-podinfo\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv\" (UID: \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" Nov 03 22:45:20.959050 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.959023 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq"] Nov 03 22:45:20.959622 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:45:20.959603 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ee2347e_d2c8_4319_a5f8_f926e83ae1c6.slice/crio-3fb06711fab6c410a8e31c049e8aaf028eaca5026adc4d55d2eadff0f797391b WatchSource:0}: Error finding container 3fb06711fab6c410a8e31c049e8aaf028eaca5026adc4d55d2eadff0f797391b: Status 404 returned error can't find the container with id 3fb06711fab6c410a8e31c049e8aaf028eaca5026adc4d55d2eadff0f797391b Nov 03 22:45:20.962066 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:45:20.962042 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcacf0851_d190_4ef3_8df0_e181b4444430.slice/crio-7a9cd663f8332eb7a691ecfeb05dc9dd52fe4e18e8833e251f433f03c33ce7d5 WatchSource:0}: Error finding container 7a9cd663f8332eb7a691ecfeb05dc9dd52fe4e18e8833e251f433f03c33ce7d5: Status 404 returned error can't find the container with id 7a9cd663f8332eb7a691ecfeb05dc9dd52fe4e18e8833e251f433f03c33ce7d5 Nov 03 22:45:20.962550 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.962513 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:45:20.962613 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.962597 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:45:20.962657 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.962642 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:45:20.962908 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.962887 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g"] Nov 03 22:45:20.964693 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.964666 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:45:20.964770 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.964724 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:45:20.964770 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:20.964753 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:45:21.022647 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.022616 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" event={"ID":"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6","Type":"ContainerStarted","Data":"3fb06711fab6c410a8e31c049e8aaf028eaca5026adc4d55d2eadff0f797391b"} Nov 03 22:45:21.024766 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.024484 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" event={"ID":"cacf0851-d190-4ef3-8df0-e181b4444430","Type":"ContainerStarted","Data":"7a9cd663f8332eb7a691ecfeb05dc9dd52fe4e18e8833e251f433f03c33ce7d5"} Nov 03 22:45:21.031783 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.031758 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdc73375-64fa-42d8-8ba8-23a936fd1f06-istio-envoy\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv\" (UID: \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" Nov 03 22:45:21.031903 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.031789 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfxgj\" (UniqueName: \"kubernetes.io/projected/bdc73375-64fa-42d8-8ba8-23a936fd1f06-kube-api-access-xfxgj\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv\" (UID: \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" Nov 03 22:45:21.031903 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.031809 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdc73375-64fa-42d8-8ba8-23a936fd1f06-istio-podinfo\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv\" (UID: \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" Nov 03 22:45:21.031903 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.031849 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdc73375-64fa-42d8-8ba8-23a936fd1f06-credential-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv\" (UID: \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" Nov 03 22:45:21.031903 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.031891 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdc73375-64fa-42d8-8ba8-23a936fd1f06-istio-data\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv\" (UID: \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" Nov 03 22:45:21.032185 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.031982 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdc73375-64fa-42d8-8ba8-23a936fd1f06-istio-token\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv\" (UID: \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" Nov 03 22:45:21.032185 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.032013 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdc73375-64fa-42d8-8ba8-23a936fd1f06-workload-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv\" (UID: \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" Nov 03 22:45:21.032302 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.032233 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdc73375-64fa-42d8-8ba8-23a936fd1f06-istiod-ca-cert\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv\" (UID: \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" Nov 03 22:45:21.032302 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.032276 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdc73375-64fa-42d8-8ba8-23a936fd1f06-workload-certs\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv\" (UID: \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" Nov 03 22:45:21.032410 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.032274 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdc73375-64fa-42d8-8ba8-23a936fd1f06-credential-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv\" (UID: \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" Nov 03 22:45:21.032410 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.032366 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdc73375-64fa-42d8-8ba8-23a936fd1f06-istio-data\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv\" (UID: \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" Nov 03 22:45:21.032604 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.032584 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdc73375-64fa-42d8-8ba8-23a936fd1f06-workload-certs\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv\" (UID: \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" Nov 03 22:45:21.032851 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.032829 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdc73375-64fa-42d8-8ba8-23a936fd1f06-workload-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv\" (UID: \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" Nov 03 22:45:21.032964 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.032924 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdc73375-64fa-42d8-8ba8-23a936fd1f06-istiod-ca-cert\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv\" (UID: \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" Nov 03 22:45:21.034329 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.034309 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdc73375-64fa-42d8-8ba8-23a936fd1f06-istio-podinfo\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv\" (UID: \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" Nov 03 22:45:21.034417 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.034402 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdc73375-64fa-42d8-8ba8-23a936fd1f06-istio-envoy\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv\" (UID: \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" Nov 03 22:45:21.094421 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.094378 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdc73375-64fa-42d8-8ba8-23a936fd1f06-istio-token\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv\" (UID: \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" Nov 03 22:45:21.095238 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.095183 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfxgj\" (UniqueName: \"kubernetes.io/projected/bdc73375-64fa-42d8-8ba8-23a936fd1f06-kube-api-access-xfxgj\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv\" (UID: \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" Nov 03 22:45:21.145260 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.145230 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895765-jdlrt"] Nov 03 22:45:21.168303 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.168253 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr"] Nov 03 22:45:21.172253 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.172225 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" Nov 03 22:45:21.242769 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.242739 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr"] Nov 03 22:45:21.313052 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.313025 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895765-jdlrt" Nov 03 22:45:21.342578 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.342546 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcc0cca0-e8aa-49c0-a502-25badef479aa-istio-data\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr\" (UID: \"dcc0cca0-e8aa-49c0-a502-25badef479aa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" Nov 03 22:45:21.342746 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.342595 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcc0cca0-e8aa-49c0-a502-25badef479aa-istio-podinfo\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr\" (UID: \"dcc0cca0-e8aa-49c0-a502-25badef479aa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" Nov 03 22:45:21.342746 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.342729 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc0cca0-e8aa-49c0-a502-25badef479aa-credential-socket\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr\" (UID: \"dcc0cca0-e8aa-49c0-a502-25badef479aa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" Nov 03 22:45:21.342822 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.342762 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcc0cca0-e8aa-49c0-a502-25badef479aa-workload-certs\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr\" (UID: \"dcc0cca0-e8aa-49c0-a502-25badef479aa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" Nov 03 22:45:21.342932 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.342911 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcc0cca0-e8aa-49c0-a502-25badef479aa-istio-envoy\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr\" (UID: \"dcc0cca0-e8aa-49c0-a502-25badef479aa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" Nov 03 22:45:21.342976 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.342942 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcc0cca0-e8aa-49c0-a502-25badef479aa-istio-token\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr\" (UID: \"dcc0cca0-e8aa-49c0-a502-25badef479aa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" Nov 03 22:45:21.342976 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.342970 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcc0cca0-e8aa-49c0-a502-25badef479aa-istiod-ca-cert\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr\" (UID: \"dcc0cca0-e8aa-49c0-a502-25badef479aa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" Nov 03 22:45:21.343098 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.343085 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc0cca0-e8aa-49c0-a502-25badef479aa-workload-socket\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr\" (UID: \"dcc0cca0-e8aa-49c0-a502-25badef479aa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" Nov 03 22:45:21.343137 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.343111 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgtlz\" (UniqueName: \"kubernetes.io/projected/dcc0cca0-e8aa-49c0-a502-25badef479aa-kube-api-access-hgtlz\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr\" (UID: \"dcc0cca0-e8aa-49c0-a502-25badef479aa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" Nov 03 22:45:21.390133 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.389984 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" Nov 03 22:45:21.445354 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.444511 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b88a9a7b-033c-403d-b1a5-5a2f71d0c918-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88a9a7b-033c-403d-b1a5-5a2f71d0c918-maas-api-service-ca\") pod \"b88a9a7b-033c-403d-b1a5-5a2f71d0c918\" (UID: \"b88a9a7b-033c-403d-b1a5-5a2f71d0c918\") " Nov 03 22:45:21.445354 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.444675 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b88a9a7b-033c-403d-b1a5-5a2f71d0c918-kube-api-access-nqxwg\" (UniqueName: \"kubernetes.io/projected/b88a9a7b-033c-403d-b1a5-5a2f71d0c918-kube-api-access-nqxwg\") pod \"b88a9a7b-033c-403d-b1a5-5a2f71d0c918\" (UID: \"b88a9a7b-033c-403d-b1a5-5a2f71d0c918\") " Nov 03 22:45:21.445354 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.445083 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcc0cca0-e8aa-49c0-a502-25badef479aa-istiod-ca-cert\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr\" (UID: \"dcc0cca0-e8aa-49c0-a502-25badef479aa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" Nov 03 22:45:21.445354 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.445213 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc0cca0-e8aa-49c0-a502-25badef479aa-workload-socket\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr\" (UID: \"dcc0cca0-e8aa-49c0-a502-25badef479aa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" Nov 03 22:45:21.445354 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.445249 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgtlz\" (UniqueName: \"kubernetes.io/projected/dcc0cca0-e8aa-49c0-a502-25badef479aa-kube-api-access-hgtlz\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr\" (UID: \"dcc0cca0-e8aa-49c0-a502-25badef479aa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" Nov 03 22:45:21.445740 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.445383 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcc0cca0-e8aa-49c0-a502-25badef479aa-istio-data\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr\" (UID: \"dcc0cca0-e8aa-49c0-a502-25badef479aa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" Nov 03 22:45:21.445740 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.445421 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcc0cca0-e8aa-49c0-a502-25badef479aa-istio-podinfo\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr\" (UID: \"dcc0cca0-e8aa-49c0-a502-25badef479aa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" Nov 03 22:45:21.445740 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.445511 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc0cca0-e8aa-49c0-a502-25badef479aa-credential-socket\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr\" (UID: \"dcc0cca0-e8aa-49c0-a502-25badef479aa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" Nov 03 22:45:21.445740 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.445546 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcc0cca0-e8aa-49c0-a502-25badef479aa-workload-certs\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr\" (UID: \"dcc0cca0-e8aa-49c0-a502-25badef479aa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" Nov 03 22:45:21.445740 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.445660 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcc0cca0-e8aa-49c0-a502-25badef479aa-istio-envoy\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr\" (UID: \"dcc0cca0-e8aa-49c0-a502-25badef479aa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" Nov 03 22:45:21.445740 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.445690 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcc0cca0-e8aa-49c0-a502-25badef479aa-istio-token\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr\" (UID: \"dcc0cca0-e8aa-49c0-a502-25badef479aa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" Nov 03 22:45:21.448114 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.446449 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b88a9a7b-033c-403d-b1a5-5a2f71d0c918-maas-api-service-ca" pod "b88a9a7b-033c-403d-b1a5-5a2f71d0c918" (UID: "b88a9a7b-033c-403d-b1a5-5a2f71d0c918"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:21.448114 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.447208 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc0cca0-e8aa-49c0-a502-25badef479aa-credential-socket\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr\" (UID: \"dcc0cca0-e8aa-49c0-a502-25badef479aa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" Nov 03 22:45:21.448114 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.447433 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcc0cca0-e8aa-49c0-a502-25badef479aa-istio-data\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr\" (UID: \"dcc0cca0-e8aa-49c0-a502-25badef479aa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" Nov 03 22:45:21.448114 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.447641 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc0cca0-e8aa-49c0-a502-25badef479aa-workload-socket\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr\" (UID: \"dcc0cca0-e8aa-49c0-a502-25badef479aa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" Nov 03 22:45:21.448114 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.448064 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcc0cca0-e8aa-49c0-a502-25badef479aa-workload-certs\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr\" (UID: \"dcc0cca0-e8aa-49c0-a502-25badef479aa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" Nov 03 22:45:21.448532 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.448357 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcc0cca0-e8aa-49c0-a502-25badef479aa-istiod-ca-cert\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr\" (UID: \"dcc0cca0-e8aa-49c0-a502-25badef479aa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" Nov 03 22:45:21.451544 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.451274 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcc0cca0-e8aa-49c0-a502-25badef479aa-istio-envoy\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr\" (UID: \"dcc0cca0-e8aa-49c0-a502-25badef479aa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" Nov 03 22:45:21.458302 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.458122 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcc0cca0-e8aa-49c0-a502-25badef479aa-istio-podinfo\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr\" (UID: \"dcc0cca0-e8aa-49c0-a502-25badef479aa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" Nov 03 22:45:21.461746 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.461719 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b88a9a7b-033c-403d-b1a5-5a2f71d0c918-kube-api-access-nqxwg" pod "b88a9a7b-033c-403d-b1a5-5a2f71d0c918" (UID: "b88a9a7b-033c-403d-b1a5-5a2f71d0c918"). InnerVolumeSpecName "kube-api-access-nqxwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:21.513240 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.510711 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgtlz\" (UniqueName: \"kubernetes.io/projected/dcc0cca0-e8aa-49c0-a502-25badef479aa-kube-api-access-hgtlz\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr\" (UID: \"dcc0cca0-e8aa-49c0-a502-25badef479aa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" Nov 03 22:45:21.525595 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.525545 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcc0cca0-e8aa-49c0-a502-25badef479aa-istio-token\") pod \"e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr\" (UID: \"dcc0cca0-e8aa-49c0-a502-25badef479aa\") " pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" Nov 03 22:45:21.547400 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.547369 5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88a9a7b-033c-403d-b1a5-5a2f71d0c918-maas-api-service-ca\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:45:21.547400 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.547399 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqxwg\" (UniqueName: \"kubernetes.io/projected/b88a9a7b-033c-403d-b1a5-5a2f71d0c918-kube-api-access-nqxwg\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:45:21.786415 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.786384 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv"] Nov 03 22:45:21.789096 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.789071 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" Nov 03 22:45:21.812054 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.806824 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:45:21.812054 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.806949 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:45:21.812054 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:21.806993 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:45:22.083910 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:22.083810 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" event={"ID":"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6","Type":"ContainerStarted","Data":"fcb540b2641ca06fc9dbc1019e9965a288bbf186c22bbf5e1be3b12d40a55646"} Nov 03 22:45:22.087187 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:22.087158 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" event={"ID":"cacf0851-d190-4ef3-8df0-e181b4444430","Type":"ContainerStarted","Data":"edb6ad9aabcee8b78073613b96b985deef1a5a72c4d2e4b1672a97edb6406383"} Nov 03 22:45:22.089340 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:22.089319 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895765-jdlrt" Nov 03 22:45:22.089446 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:22.089365 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895765-jdlrt" event={"ID":"b88a9a7b-033c-403d-b1a5-5a2f71d0c918","Type":"ContainerDied","Data":"5388aa6eca0fc186567f6fb1f5b379a1de092b151d694a53611ec5321be8b99b"} Nov 03 22:45:22.089446 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:22.089420 5196 scope.go:122] "RemoveContainer" containerID="c4dbd33f58916b21be36281dba5e31a380f7c58d7b1eab34b560fccb2a1ff23e" Nov 03 22:45:22.255747 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:22.255690 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" event={"ID":"bdc73375-64fa-42d8-8ba8-23a936fd1f06","Type":"ContainerStarted","Data":"6c5a5d87185c8163c7d8735001be10125282042d09f99a6850bd6d1e40e77b97"} Nov 03 22:45:22.255951 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:22.255821 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" event={"ID":"bdc73375-64fa-42d8-8ba8-23a936fd1f06","Type":"ContainerStarted","Data":"60ba4966aa1aa33d56d2c654df7f58c9d590a3d4e5f33ffe7a9edd08f6bed3df"} Nov 03 22:45:22.269916 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:22.269804 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" podStartSLOduration=2.262693243 podStartE2EDuration="2.262693243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:22.257531393 +0000 UTC m=+1351.567090698" watchObservedRunningTime="2026-11-03 22:45:22.262693243 +0000 UTC m=+1351.572252552" Nov 03 22:45:22.350745 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:22.350707 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr"] Nov 03 22:45:22.361655 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:45:22.361293 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcc0cca0_e8aa_49c0_a502_25badef479aa.slice/crio-df1e5ebe7aecbb73f9c14742e6b53d8c70702b04b9f8065bf30d5c050a386c08 WatchSource:0}: Error finding container df1e5ebe7aecbb73f9c14742e6b53d8c70702b04b9f8065bf30d5c050a386c08: Status 404 returned error can't find the container with id df1e5ebe7aecbb73f9c14742e6b53d8c70702b04b9f8065bf30d5c050a386c08 Nov 03 22:45:22.367720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:22.367673 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:45:22.367914 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:22.367767 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:45:22.367914 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:22.367811 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:45:22.451882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:22.451418 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" Nov 03 22:45:22.454458 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:22.454396 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" podStartSLOduration=2.454375117 podStartE2EDuration="2.454375117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:22.451956499 +0000 UTC m=+1351.761515803" watchObservedRunningTime="2026-11-03 22:45:22.454375117 +0000 UTC m=+1351.763934411" Nov 03 22:45:22.545883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:22.529035 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" Nov 03 22:45:22.545883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:22.529114 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" Nov 03 22:45:22.557884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:22.555991 5196 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 22:45:22.557884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:22.556048 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" podUID="cacf0851-d190-4ef3-8df0-e181b4444430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 22:45:22.557884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:22.556001 5196 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 22:45:22.557884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:22.556104 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" podUID="5ee2347e-d2c8-4319-a5f8-f926e83ae1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 22:45:22.655785 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:22.655729 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" podStartSLOduration=2.655714933 podStartE2EDuration="2.655714933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:22.655096989 +0000 UTC m=+1351.964656297" watchObservedRunningTime="2026-11-03 22:45:22.655714933 +0000 UTC m=+1351.965274226" Nov 03 22:45:22.783744 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:22.783706 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895765-jdlrt"] Nov 03 22:45:22.786768 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:22.786748 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7540da-29895765-jdlrt"] Nov 03 22:45:22.856703 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:22.856602 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:22.859345 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:22.858771 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b88a9a7b-033c-403d-b1a5-5a2f71d0c918" path="/var/lib/kubelet/pods/b88a9a7b-033c-403d-b1a5-5a2f71d0c918/volumes" Nov 03 22:45:23.330502 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:23.330412 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" event={"ID":"dcc0cca0-e8aa-49c0-a502-25badef479aa","Type":"ContainerStarted","Data":"2d8e84c9f92d492a9a24402f80d9a79e32721350f3b1e272e933542ee06eb32c"} Nov 03 22:45:23.331079 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:23.331054 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" event={"ID":"dcc0cca0-e8aa-49c0-a502-25badef479aa","Type":"ContainerStarted","Data":"df1e5ebe7aecbb73f9c14742e6b53d8c70702b04b9f8065bf30d5c050a386c08"} Nov 03 22:45:23.403494 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:23.403458 5196 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 22:45:23.403830 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:23.403801 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" podUID="bdc73375-64fa-42d8-8ba8-23a936fd1f06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 22:45:23.469083 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:23.468918 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" podStartSLOduration=2.46889343 podStartE2EDuration="2.46889343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:23.461561807 +0000 UTC m=+1352.771121101" watchObservedRunningTime="2026-11-03 22:45:23.46889343 +0000 UTC m=+1352.778452747" Nov 03 22:45:23.514325 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:23.514267 5196 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 22:45:23.514530 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:23.514334 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" podUID="cacf0851-d190-4ef3-8df0-e181b4444430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 22:45:23.537882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:23.528546 5196 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 22:45:23.537882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:23.528606 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" podUID="5ee2347e-d2c8-4319-a5f8-f926e83ae1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 22:45:23.825077 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:23.825027 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" Nov 03 22:45:23.860912 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:23.855085 5196 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:45:23.868895 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:23.865937 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" podUID="dcc0cca0-e8aa-49c0-a502-25badef479aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 22:45:24.393883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:24.393049 5196 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 22:45:24.393883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:24.393127 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" podUID="bdc73375-64fa-42d8-8ba8-23a936fd1f06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 22:45:24.511883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:24.505152 5196 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 22:45:24.511883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:24.505212 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" podUID="cacf0851-d190-4ef3-8df0-e181b4444430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 22:45:24.537221 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:24.537187 5196 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 22:45:24.569921 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:24.560955 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" podUID="5ee2347e-d2c8-4319-a5f8-f926e83ae1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 22:45:24.700093 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:24.699986 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" podUID="26fa51c2-0f2e-482f-9c67-0ea7924d7b3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:24.791258 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:24.791223 5196 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:45:24.791530 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:24.791503 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" podUID="dcc0cca0-e8aa-49c0-a502-25badef479aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 22:45:24.952566 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:24.952468 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" podUID="9f2e3f52-0664-434f-919b-32736cbf0602" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:24.983013 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:24.982951 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" podUID="9d8b4fed-b4b0-49ce-9238-e23175e90b70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:25.295936 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:25.287469 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:25.319884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:25.308170 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" podUID="82749c55-0fca-4a14-b6ba-7df122a6a407" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:25.413178 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:25.402360 5196 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 22:45:25.413178 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:25.402418 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" podUID="bdc73375-64fa-42d8-8ba8-23a936fd1f06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 22:45:25.509884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:25.509245 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" podUID="34e9cf3a-3456-4835-8933-102ce735cccc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:25.510087 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:25.510048 5196 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 22:45:25.510160 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:25.510090 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" podUID="cacf0851-d190-4ef3-8df0-e181b4444430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 22:45:25.533883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:25.524435 5196 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 22:45:25.533883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:25.524490 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" podUID="5ee2347e-d2c8-4319-a5f8-f926e83ae1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 22:45:25.700883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:25.700541 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" podUID="26fa51c2-0f2e-482f-9c67-0ea7924d7b3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 03 22:45:25.701101 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:25.700911 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" Nov 03 22:45:25.800279 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:25.800241 5196 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:45:25.800545 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:25.800519 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" podUID="dcc0cca0-e8aa-49c0-a502-25badef479aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 22:45:25.983978 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:25.956432 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" podUID="abb461c4-9707-4b5c-aba0-793f7bb993e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:26.022920 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:25.966969 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" podUID="9f2e3f52-0664-434f-919b-32736cbf0602" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 22:45:26.022920 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:25.967028 5196 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" podUID="9d8b4fed-b4b0-49ce-9238-e23175e90b70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:26.022920 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:25.984991 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" podUID="9d8b4fed-b4b0-49ce-9238-e23175e90b70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:26.022920 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:26.021050 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" Nov 03 22:45:26.022920 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:26.021110 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" Nov 03 22:45:26.311951 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:26.310955 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" podUID="82749c55-0fca-4a14-b6ba-7df122a6a407" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:26.311951 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:26.311431 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" Nov 03 22:45:26.573251 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:26.573115 5196 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 22:45:26.608510 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:26.608470 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" podUID="5ee2347e-d2c8-4319-a5f8-f926e83ae1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 22:45:26.608774 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:26.608399 5196 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 22:45:26.608976 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:26.608952 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" podUID="bdc73375-64fa-42d8-8ba8-23a936fd1f06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 22:45:26.638936 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:26.630694 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" podUID="34e9cf3a-3456-4835-8933-102ce735cccc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:26.638936 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:26.637958 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" Nov 03 22:45:26.739925 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:26.726399 5196 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 22:45:26.740259 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:26.740207 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" podUID="cacf0851-d190-4ef3-8df0-e181b4444430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 22:45:26.740582 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:26.740558 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" podUID="26fa51c2-0f2e-482f-9c67-0ea7924d7b3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 03 22:45:26.784910 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:45:26.774483 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:45:26.874005 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:26.873974 5196 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:45:26.908299 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:26.894722 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" podUID="dcc0cca0-e8aa-49c0-a502-25badef479aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 22:45:27.140947 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:27.116929 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" podUID="abb461c4-9707-4b5c-aba0-793f7bb993e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:27.398941 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:27.136880 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" podUID="9d8b4fed-b4b0-49ce-9238-e23175e90b70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 03 22:45:27.398941 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:27.137802 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" podUID="9f2e3f52-0664-434f-919b-32736cbf0602" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 22:45:27.398941 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:27.334376 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" podUID="82749c55-0fca-4a14-b6ba-7df122a6a407" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:27.437883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:27.400878 5196 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 22:45:27.437883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:27.400941 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" podUID="bdc73375-64fa-42d8-8ba8-23a936fd1f06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 22:45:27.437883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:27.412547 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" Nov 03 22:45:27.529892 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:27.528931 5196 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 22:45:27.529892 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:27.528984 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" podUID="cacf0851-d190-4ef3-8df0-e181b4444430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 22:45:27.529892 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:27.529084 5196 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 22:45:27.529892 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:27.529105 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" podUID="5ee2347e-d2c8-4319-a5f8-f926e83ae1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 22:45:27.724940 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:27.724439 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" podUID="34e9cf3a-3456-4835-8933-102ce735cccc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:27.770009 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:27.769961 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" podUID="26fa51c2-0f2e-482f-9c67-0ea7924d7b3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:27.811882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:27.811767 5196 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:45:27.811882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:27.811824 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" podUID="dcc0cca0-e8aa-49c0-a502-25badef479aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 22:45:28.179877 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:28.173012 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" podUID="abb461c4-9707-4b5c-aba0-793f7bb993e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:28.188881 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:28.180824 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" Nov 03 22:45:28.188881 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:28.182982 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" Nov 03 22:45:28.273881 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:28.227386 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" podUID="9d8b4fed-b4b0-49ce-9238-e23175e90b70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 03 22:45:28.353896 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:28.343877 5196 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" podUID="82749c55-0fca-4a14-b6ba-7df122a6a407" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:28.551953 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:28.536210 5196 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 22:45:28.551953 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:28.536272 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" podUID="cacf0851-d190-4ef3-8df0-e181b4444430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 22:45:28.572468 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:28.571673 5196 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 22:45:28.572468 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:28.571731 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" podUID="bdc73375-64fa-42d8-8ba8-23a936fd1f06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 22:45:28.580817 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:28.580792 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v"] Nov 03 22:45:28.581932 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:28.581908 5196 cpu_manager.go:414] "containerMap: removing container" podUID="b88a9a7b-033c-403d-b1a5-5a2f71d0c918" containerName="cleanup" Nov 03 22:45:28.582098 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:28.582083 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b88a9a7b-033c-403d-b1a5-5a2f71d0c918" containerName="cleanup" Nov 03 22:45:28.582190 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:28.582179 5196 cpu_manager.go:414] "containerMap: removing container" podUID="b88a9a7b-033c-403d-b1a5-5a2f71d0c918" containerName="cleanup" Nov 03 22:45:28.582274 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:28.582262 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b88a9a7b-033c-403d-b1a5-5a2f71d0c918" containerName="cleanup" Nov 03 22:45:28.582370 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:28.582359 5196 cpu_manager.go:414] "containerMap: removing container" podUID="b88a9a7b-033c-403d-b1a5-5a2f71d0c918" containerName="cleanup" Nov 03 22:45:28.582445 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:28.582434 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b88a9a7b-033c-403d-b1a5-5a2f71d0c918" containerName="cleanup" Nov 03 22:45:28.582727 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:28.582710 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="b88a9a7b-033c-403d-b1a5-5a2f71d0c918" containerName="cleanup" Nov 03 22:45:28.582870 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:28.582829 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="b88a9a7b-033c-403d-b1a5-5a2f71d0c918" containerName="cleanup" Nov 03 22:45:28.582988 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:28.582976 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="b88a9a7b-033c-403d-b1a5-5a2f71d0c918" containerName="cleanup" Nov 03 22:45:28.608949 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:28.596083 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" Nov 03 22:45:28.614887 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:28.609189 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v"] Nov 03 22:45:28.636888 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:28.623993 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" Nov 03 22:45:28.675654 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:28.675625 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" Nov 03 22:45:28.675961 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:28.674440 5196 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 22:45:28.676140 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:28.676116 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" podUID="5ee2347e-d2c8-4319-a5f8-f926e83ae1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 22:45:28.676256 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:28.632031 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v" Nov 03 22:45:28.688585 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:28.688507 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhfv2\" (UniqueName: \"kubernetes.io/projected/5054337e-aaf6-42ca-a75c-ab1fd174afeb-kube-api-access-hhfv2\") pod \"maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v\" (UID: \"5054337e-aaf6-42ca-a75c-ab1fd174afeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v" Nov 03 22:45:28.688585 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:28.688580 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5054337e-aaf6-42ca-a75c-ab1fd174afeb-openshift-service-ca\") pod \"maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v\" (UID: \"5054337e-aaf6-42ca-a75c-ab1fd174afeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v" Nov 03 22:45:28.688780 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:28.688727 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5054337e-aaf6-42ca-a75c-ab1fd174afeb-metrics-certs\") pod \"maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v\" (UID: \"5054337e-aaf6-42ca-a75c-ab1fd174afeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v" Nov 03 22:45:28.688847 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:28.688823 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5054337e-aaf6-42ca-a75c-ab1fd174afeb-maas-api-tls\") pod \"maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v\" (UID: \"5054337e-aaf6-42ca-a75c-ab1fd174afeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v" Nov 03 22:45:28.757906 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:28.744575 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:28.757906 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:28.744970 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-468d8d1b-serving-cert\"" Nov 03 22:45:28.840936 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:28.814552 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" Nov 03 22:45:28.865515 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:28.865477 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5054337e-aaf6-42ca-a75c-ab1fd174afeb-metrics-certs\") pod \"maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v\" (UID: \"5054337e-aaf6-42ca-a75c-ab1fd174afeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v" Nov 03 22:45:29.043911 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:29.036543 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5054337e-aaf6-42ca-a75c-ab1fd174afeb-maas-api-tls\") pod \"maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v\" (UID: \"5054337e-aaf6-42ca-a75c-ab1fd174afeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v" Nov 03 22:45:29.066882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:29.044326 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhfv2\" (UniqueName: \"kubernetes.io/projected/5054337e-aaf6-42ca-a75c-ab1fd174afeb-kube-api-access-hhfv2\") pod \"maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v\" (UID: \"5054337e-aaf6-42ca-a75c-ab1fd174afeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v" Nov 03 22:45:29.068473 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:29.068439 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5054337e-aaf6-42ca-a75c-ab1fd174afeb-openshift-service-ca\") pod \"maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v\" (UID: \"5054337e-aaf6-42ca-a75c-ab1fd174afeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v" Nov 03 22:45:29.084555 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:29.084486 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5054337e-aaf6-42ca-a75c-ab1fd174afeb-metrics-certs\") pod \"maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v\" (UID: \"5054337e-aaf6-42ca-a75c-ab1fd174afeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v" Nov 03 22:45:29.085401 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:28.843519 5196 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:45:29.085691 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:29.085662 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" podUID="dcc0cca0-e8aa-49c0-a502-25badef479aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 22:45:29.086514 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:29.086481 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:29.123967 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:29.109502 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5054337e-aaf6-42ca-a75c-ab1fd174afeb-maas-api-tls\") pod \"maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v\" (UID: \"5054337e-aaf6-42ca-a75c-ab1fd174afeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v" Nov 03 22:45:29.179064 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:29.179026 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5054337e-aaf6-42ca-a75c-ab1fd174afeb-openshift-service-ca\") pod \"maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v\" (UID: \"5054337e-aaf6-42ca-a75c-ab1fd174afeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v" Nov 03 22:45:29.252993 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:29.252957 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:29.360892 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:29.360804 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhfv2\" (UniqueName: \"kubernetes.io/projected/5054337e-aaf6-42ca-a75c-ab1fd174afeb-kube-api-access-hhfv2\") pod \"maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v\" (UID: \"5054337e-aaf6-42ca-a75c-ab1fd174afeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v" Nov 03 22:45:29.373609 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:29.373344 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v" Nov 03 22:45:29.447209 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:29.447158 5196 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 22:45:29.447524 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:29.447495 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" podUID="bdc73375-64fa-42d8-8ba8-23a936fd1f06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 22:45:29.506565 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:29.505985 5196 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 22:45:29.506565 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:29.506041 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" podUID="cacf0851-d190-4ef3-8df0-e181b4444430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 22:45:29.823938 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:29.818474 5196 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:45:29.823938 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:29.818533 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" podUID="dcc0cca0-e8aa-49c0-a502-25badef479aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 22:45:30.004415 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:45:30.004367 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5054337e_aaf6_42ca_a75c_ab1fd174afeb.slice/crio-5fb1e44613971cbe150f4a832cfb8f7b98dabed764fba0cd84a154c4ac6fadf6 WatchSource:0}: Error finding container 5fb1e44613971cbe150f4a832cfb8f7b98dabed764fba0cd84a154c4ac6fadf6: Status 404 returned error can't find the container with id 5fb1e44613971cbe150f4a832cfb8f7b98dabed764fba0cd84a154c4ac6fadf6 Nov 03 22:45:30.012019 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:30.011991 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v"] Nov 03 22:45:30.025879 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:30.025821 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:30.128817 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:30.126475 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" Nov 03 22:45:30.128817 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:30.128233 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" Nov 03 22:45:30.144883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:30.139285 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" Nov 03 22:45:30.196465 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:30.196430 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:30.390437 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:30.390330 5196 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 22:45:30.390437 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:30.390380 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" podUID="bdc73375-64fa-42d8-8ba8-23a936fd1f06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 22:45:30.511799 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:30.511768 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" Nov 03 22:45:30.512754 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:30.512663 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" Nov 03 22:45:30.513444 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:30.513424 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" Nov 03 22:45:30.608494 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:30.608449 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v" event={"ID":"5054337e-aaf6-42ca-a75c-ab1fd174afeb","Type":"ContainerStarted","Data":"5fb1e44613971cbe150f4a832cfb8f7b98dabed764fba0cd84a154c4ac6fadf6"} Nov 03 22:45:30.793237 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:30.790097 5196 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 22:45:30.793237 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:30.790156 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" podUID="dcc0cca0-e8aa-49c0-a502-25badef479aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 22:45:31.399882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:31.398134 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" Nov 03 22:45:31.399882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:31.399627 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" Nov 03 22:45:31.400470 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:31.400435 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" Nov 03 22:45:31.619152 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:31.619113 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v" event={"ID":"5054337e-aaf6-42ca-a75c-ab1fd174afeb","Type":"ContainerStarted","Data":"467b6779293474097a0053d7be46f64a86e8c524bef407df0bcaf56229d74ef4"} Nov 03 22:45:31.619347 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:31.619191 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v" Nov 03 22:45:31.775048 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:31.774928 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v" podStartSLOduration=2.943202813 podStartE2EDuration="3.774913024s" totalImagesPullingTime="831.710211ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:31.770594451 +0000 UTC m=+1361.080153761" watchObservedRunningTime="2026-11-03 22:45:31.774913024 +0000 UTC m=+1361.084472317" Nov 03 22:45:31.793519 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:31.793494 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" Nov 03 22:45:31.793777 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:31.793759 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" Nov 03 22:45:31.794543 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:31.794525 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" Nov 03 22:45:34.213563 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:34.213530 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:37.666220 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:37.665023 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v" Nov 03 22:45:37.730018 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:37.729972 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" podUID="34e9cf3a-3456-4835-8933-102ce735cccc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:37.756103 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:37.756057 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" podUID="26fa51c2-0f2e-482f-9c67-0ea7924d7b3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 03 22:45:38.228314 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:38.226986 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" podUID="9d8b4fed-b4b0-49ce-9238-e23175e90b70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:38.341593 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:38.340075 5196 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" podUID="82749c55-0fca-4a14-b6ba-7df122a6a407" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:38.341593 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:38.340410 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" podUID="82749c55-0fca-4a14-b6ba-7df122a6a407" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:38.343028 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:38.341943 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" podUID="9f2e3f52-0664-434f-919b-32736cbf0602" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:38.368908 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:38.344140 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" podUID="abb461c4-9707-4b5c-aba0-793f7bb993e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:38.702096 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:38.702046 5196 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" podUID="9f2e3f52-0664-434f-919b-32736cbf0602" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded" Nov 03 22:45:38.815920 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:38.756892 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" podUID="26fa51c2-0f2e-482f-9c67-0ea7924d7b3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:38.815920 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:38.779254 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" Nov 03 22:45:38.815920 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:38.815677 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" podUID="34e9cf3a-3456-4835-8933-102ce735cccc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 03 22:45:38.837906 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:38.817183 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" Nov 03 22:45:38.912883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:38.862026 5196 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" podUID="34e9cf3a-3456-4835-8933-102ce735cccc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:39.058942 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:39.048777 5196 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:45:39.058942 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:39.048919 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" podUID="5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:39.436885 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:39.427242 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" podUID="abb461c4-9707-4b5c-aba0-793f7bb993e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 03 22:45:39.436885 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:39.429805 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" podUID="82749c55-0fca-4a14-b6ba-7df122a6a407" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 03 22:45:39.436885 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:39.429950 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" podUID="9d8b4fed-b4b0-49ce-9238-e23175e90b70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:39.467315 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:39.467256 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" podUID="9f2e3f52-0664-434f-919b-32736cbf0602" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:39.551109 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:39.550309 5196 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:45:39.551109 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:39.550623 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" podUID="aa304409-0e6c-4543-a6ae-3db66214e80d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:39.896895 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:39.896313 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" podUID="34e9cf3a-3456-4835-8933-102ce735cccc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:39.896895 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:39.896416 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" podUID="26fa51c2-0f2e-482f-9c67-0ea7924d7b3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 03 22:45:40.288945 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:45:40.221384 5196 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.359s" Nov 03 22:45:40.313631 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:40.313602 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" Nov 03 22:45:40.313975 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:40.313955 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" Nov 03 22:45:40.314304 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:40.314288 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" Nov 03 22:45:40.314482 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:40.314469 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" Nov 03 22:45:40.448106 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:40.448056 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" podUID="9d8b4fed-b4b0-49ce-9238-e23175e90b70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:40.477019 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:40.448507 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" podUID="abb461c4-9707-4b5c-aba0-793f7bb993e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 03 22:45:40.486100 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:40.482965 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" podUID="9f2e3f52-0664-434f-919b-32736cbf0602" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:40.783514 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:40.783482 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" Nov 03 22:45:40.961299 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:40.961256 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" podUID="34e9cf3a-3456-4835-8933-102ce735cccc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:41.038632 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:41.037430 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" podUID="26fa51c2-0f2e-482f-9c67-0ea7924d7b3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:41.278883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:41.260439 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" Nov 03 22:45:41.301931 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:41.301766 5196 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" podUID="abb461c4-9707-4b5c-aba0-793f7bb993e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:41.336498 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:41.332703 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:41.336833 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:41.336801 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx"] Nov 03 22:45:41.518884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:41.503687 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" podUID="9d8b4fed-b4b0-49ce-9238-e23175e90b70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 03 22:45:41.702884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:41.699498 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:41.874883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:41.854737 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx"] Nov 03 22:45:41.875215 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:41.875192 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" Nov 03 22:45:41.895236 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:41.878977 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" Nov 03 22:45:41.895236 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:41.862354 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx" Nov 03 22:45:41.928929 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:41.928896 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg89h\" (UniqueName: \"kubernetes.io/projected/02c2015e-ae6f-49d2-bc3c-934aa8c07f55-kube-api-access-jg89h\") pod \"maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx\" (UID: \"02c2015e-ae6f-49d2-bc3c-934aa8c07f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx" Nov 03 22:45:41.929284 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:41.929261 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02c2015e-ae6f-49d2-bc3c-934aa8c07f55-metrics-certs\") pod \"maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx\" (UID: \"02c2015e-ae6f-49d2-bc3c-934aa8c07f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx" Nov 03 22:45:41.929504 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:41.929481 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02c2015e-ae6f-49d2-bc3c-934aa8c07f55-maas-api-tls\") pod \"maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx\" (UID: \"02c2015e-ae6f-49d2-bc3c-934aa8c07f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx" Nov 03 22:45:41.929838 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:41.929817 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c2015e-ae6f-49d2-bc3c-934aa8c07f55-openshift-service-ca\") pod \"maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx\" (UID: \"02c2015e-ae6f-49d2-bc3c-934aa8c07f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx" Nov 03 22:45:41.975882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:41.970241 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-fc129ccf-serving-cert\"" Nov 03 22:45:42.009893 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:41.978613 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:42.080443 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:42.066330 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" podUID="34e9cf3a-3456-4835-8933-102ce735cccc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:42.080443 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:42.071455 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c2015e-ae6f-49d2-bc3c-934aa8c07f55-openshift-service-ca\") pod \"maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx\" (UID: \"02c2015e-ae6f-49d2-bc3c-934aa8c07f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx" Nov 03 22:45:42.080443 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:42.071519 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg89h\" (UniqueName: \"kubernetes.io/projected/02c2015e-ae6f-49d2-bc3c-934aa8c07f55-kube-api-access-jg89h\") pod \"maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx\" (UID: \"02c2015e-ae6f-49d2-bc3c-934aa8c07f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx" Nov 03 22:45:42.080443 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:42.071624 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02c2015e-ae6f-49d2-bc3c-934aa8c07f55-metrics-certs\") pod \"maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx\" (UID: \"02c2015e-ae6f-49d2-bc3c-934aa8c07f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx" Nov 03 22:45:42.080443 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:42.071689 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02c2015e-ae6f-49d2-bc3c-934aa8c07f55-maas-api-tls\") pod \"maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx\" (UID: \"02c2015e-ae6f-49d2-bc3c-934aa8c07f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx" Nov 03 22:45:42.122880 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:42.102140 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c2015e-ae6f-49d2-bc3c-934aa8c07f55-openshift-service-ca\") pod \"maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx\" (UID: \"02c2015e-ae6f-49d2-bc3c-934aa8c07f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx" Nov 03 22:45:42.136158 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:42.136133 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" Nov 03 22:45:42.138763 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:42.138721 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-nqmgk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:45:42.138996 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:42.138972 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" podUID="aa3310c5-bd35-4bef-b838-1722c7143cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:42.153711 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:42.153679 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" podUID="9f2e3f52-0664-434f-919b-32736cbf0602" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 22:45:42.248713 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:42.247568 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg89h\" (UniqueName: \"kubernetes.io/projected/02c2015e-ae6f-49d2-bc3c-934aa8c07f55-kube-api-access-jg89h\") pod \"maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx\" (UID: \"02c2015e-ae6f-49d2-bc3c-934aa8c07f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx" Nov 03 22:45:42.278016 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:42.277981 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02c2015e-ae6f-49d2-bc3c-934aa8c07f55-maas-api-tls\") pod \"maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx\" (UID: \"02c2015e-ae6f-49d2-bc3c-934aa8c07f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx" Nov 03 22:45:42.405173 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:42.405136 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02c2015e-ae6f-49d2-bc3c-934aa8c07f55-metrics-certs\") pod \"maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx\" (UID: \"02c2015e-ae6f-49d2-bc3c-934aa8c07f55\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx" Nov 03 22:45:42.411894 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:42.405664 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" Nov 03 22:45:42.708882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:42.707397 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx" Nov 03 22:45:43.205246 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:45:43.205208 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c2015e_ae6f_49d2_bc3c_934aa8c07f55.slice/crio-69e0383ed83f52c9fa1b3fa8c94f5c4f8341d08587282df6ec76a73113daba86 WatchSource:0}: Error finding container 69e0383ed83f52c9fa1b3fa8c94f5c4f8341d08587282df6ec76a73113daba86: Status 404 returned error can't find the container with id 69e0383ed83f52c9fa1b3fa8c94f5c4f8341d08587282df6ec76a73113daba86 Nov 03 22:45:43.214004 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:43.213967 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx"] Nov 03 22:45:44.083395 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:44.083363 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx" event={"ID":"02c2015e-ae6f-49d2-bc3c-934aa8c07f55","Type":"ContainerStarted","Data":"69e0383ed83f52c9fa1b3fa8c94f5c4f8341d08587282df6ec76a73113daba86"} Nov 03 22:45:45.093628 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:45.093593 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx" event={"ID":"02c2015e-ae6f-49d2-bc3c-934aa8c07f55","Type":"ContainerStarted","Data":"f7c1c661a86ee16b0c13b3241100a68641c4b105debbce731b3eb6dc01953d67"} Nov 03 22:45:45.094083 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:45.093748 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx" Nov 03 22:45:45.214973 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:45.214849 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx" podStartSLOduration=4.345962982 podStartE2EDuration="5.214823575s" totalImagesPullingTime="868.860593ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:45.213127308 +0000 UTC m=+1374.522686601" watchObservedRunningTime="2026-11-03 22:45:45.214823575 +0000 UTC m=+1374.524382869" Nov 03 22:45:46.240262 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:46.240221 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:46.413878 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:46.413834 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:51.026669 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:51.026633 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:51.106784 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:51.106760 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx" Nov 03 22:45:52.325709 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:52.325678 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:55.740055 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:55.740023 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:55.925527 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:45:55.925488 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:45:57.820312 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:57.820281 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:58.406167 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:58.406134 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:45:58.696317 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:45:58.696237 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:00.517946 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:00.517914 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:02.482852 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:02.482806 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:02.656031 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:02.655984 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:02.816037 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:02.815961 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:04.003091 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:04.003057 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:05.623491 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:05.623462 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:06.112015 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:06.111983 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm"] Nov 03 22:46:06.117246 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:06.117225 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" Nov 03 22:46:06.225161 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:06.225130 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm"] Nov 03 22:46:06.264835 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:06.264803 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a1c6229-58c7-412d-b67d-9a31e460a59c-istiod-ca-cert\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm\" (UID: \"3a1c6229-58c7-412d-b67d-9a31e460a59c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" Nov 03 22:46:06.265059 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:06.264846 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a1c6229-58c7-412d-b67d-9a31e460a59c-istio-token\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm\" (UID: \"3a1c6229-58c7-412d-b67d-9a31e460a59c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" Nov 03 22:46:06.265134 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:06.265057 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a1c6229-58c7-412d-b67d-9a31e460a59c-istio-podinfo\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm\" (UID: \"3a1c6229-58c7-412d-b67d-9a31e460a59c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" Nov 03 22:46:06.265196 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:06.265133 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a1c6229-58c7-412d-b67d-9a31e460a59c-istio-envoy\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm\" (UID: \"3a1c6229-58c7-412d-b67d-9a31e460a59c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" Nov 03 22:46:06.265196 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:06.265188 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1c6229-58c7-412d-b67d-9a31e460a59c-workload-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm\" (UID: \"3a1c6229-58c7-412d-b67d-9a31e460a59c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" Nov 03 22:46:06.265338 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:06.265321 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a1c6229-58c7-412d-b67d-9a31e460a59c-workload-certs\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm\" (UID: \"3a1c6229-58c7-412d-b67d-9a31e460a59c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" Nov 03 22:46:06.265383 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:06.265348 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a1c6229-58c7-412d-b67d-9a31e460a59c-istio-data\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm\" (UID: \"3a1c6229-58c7-412d-b67d-9a31e460a59c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" Nov 03 22:46:06.265383 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:06.265370 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlv9m\" (UniqueName: \"kubernetes.io/projected/3a1c6229-58c7-412d-b67d-9a31e460a59c-kube-api-access-nlv9m\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm\" (UID: \"3a1c6229-58c7-412d-b67d-9a31e460a59c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" Nov 03 22:46:06.265514 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:06.265502 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1c6229-58c7-412d-b67d-9a31e460a59c-credential-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm\" (UID: \"3a1c6229-58c7-412d-b67d-9a31e460a59c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" Nov 03 22:46:06.367001 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:06.366914 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a1c6229-58c7-412d-b67d-9a31e460a59c-istio-podinfo\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm\" (UID: \"3a1c6229-58c7-412d-b67d-9a31e460a59c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" Nov 03 22:46:06.367001 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:06.366958 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a1c6229-58c7-412d-b67d-9a31e460a59c-istio-envoy\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm\" (UID: \"3a1c6229-58c7-412d-b67d-9a31e460a59c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" Nov 03 22:46:06.367001 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:06.366992 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1c6229-58c7-412d-b67d-9a31e460a59c-workload-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm\" (UID: \"3a1c6229-58c7-412d-b67d-9a31e460a59c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" Nov 03 22:46:06.367287 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:06.367060 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a1c6229-58c7-412d-b67d-9a31e460a59c-workload-certs\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm\" (UID: \"3a1c6229-58c7-412d-b67d-9a31e460a59c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" Nov 03 22:46:06.367287 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:06.367087 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a1c6229-58c7-412d-b67d-9a31e460a59c-istio-data\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm\" (UID: \"3a1c6229-58c7-412d-b67d-9a31e460a59c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" Nov 03 22:46:06.367287 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:06.367111 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlv9m\" (UniqueName: \"kubernetes.io/projected/3a1c6229-58c7-412d-b67d-9a31e460a59c-kube-api-access-nlv9m\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm\" (UID: \"3a1c6229-58c7-412d-b67d-9a31e460a59c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" Nov 03 22:46:06.367287 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:06.367173 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1c6229-58c7-412d-b67d-9a31e460a59c-credential-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm\" (UID: \"3a1c6229-58c7-412d-b67d-9a31e460a59c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" Nov 03 22:46:06.367287 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:06.367246 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a1c6229-58c7-412d-b67d-9a31e460a59c-istiod-ca-cert\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm\" (UID: \"3a1c6229-58c7-412d-b67d-9a31e460a59c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" Nov 03 22:46:06.367287 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:06.367280 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a1c6229-58c7-412d-b67d-9a31e460a59c-istio-token\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm\" (UID: \"3a1c6229-58c7-412d-b67d-9a31e460a59c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" Nov 03 22:46:06.367645 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:06.367552 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1c6229-58c7-412d-b67d-9a31e460a59c-workload-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm\" (UID: \"3a1c6229-58c7-412d-b67d-9a31e460a59c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" Nov 03 22:46:06.367645 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:06.367573 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a1c6229-58c7-412d-b67d-9a31e460a59c-istio-data\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm\" (UID: \"3a1c6229-58c7-412d-b67d-9a31e460a59c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" Nov 03 22:46:06.367645 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:06.367624 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a1c6229-58c7-412d-b67d-9a31e460a59c-workload-certs\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm\" (UID: \"3a1c6229-58c7-412d-b67d-9a31e460a59c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" Nov 03 22:46:06.368000 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:06.367974 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1c6229-58c7-412d-b67d-9a31e460a59c-credential-socket\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm\" (UID: \"3a1c6229-58c7-412d-b67d-9a31e460a59c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" Nov 03 22:46:06.368182 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:06.368160 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a1c6229-58c7-412d-b67d-9a31e460a59c-istiod-ca-cert\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm\" (UID: \"3a1c6229-58c7-412d-b67d-9a31e460a59c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" Nov 03 22:46:06.369477 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:06.369452 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a1c6229-58c7-412d-b67d-9a31e460a59c-istio-envoy\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm\" (UID: \"3a1c6229-58c7-412d-b67d-9a31e460a59c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" Nov 03 22:46:06.369661 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:06.369646 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a1c6229-58c7-412d-b67d-9a31e460a59c-istio-podinfo\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm\" (UID: \"3a1c6229-58c7-412d-b67d-9a31e460a59c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" Nov 03 22:46:06.429094 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:06.429066 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a1c6229-58c7-412d-b67d-9a31e460a59c-istio-token\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm\" (UID: \"3a1c6229-58c7-412d-b67d-9a31e460a59c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" Nov 03 22:46:06.429094 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:06.429084 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlv9m\" (UniqueName: \"kubernetes.io/projected/3a1c6229-58c7-412d-b67d-9a31e460a59c-kube-api-access-nlv9m\") pod \"e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm\" (UID: \"3a1c6229-58c7-412d-b67d-9a31e460a59c\") " pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" Nov 03 22:46:06.431935 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:06.431917 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" Nov 03 22:46:06.872919 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:06.872896 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm"] Nov 03 22:46:06.873741 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:46:06.873714 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a1c6229_58c7_412d_b67d_9a31e460a59c.slice/crio-d2efb22d25e904c3dfc9cdcde5ff66e5ebba8bdf35fb429f623ad393d6a6c31e WatchSource:0}: Error finding container d2efb22d25e904c3dfc9cdcde5ff66e5ebba8bdf35fb429f623ad393d6a6c31e: Status 404 returned error can't find the container with id d2efb22d25e904c3dfc9cdcde5ff66e5ebba8bdf35fb429f623ad393d6a6c31e Nov 03 22:46:06.876461 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:06.876423 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:46:06.876566 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:06.876502 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:46:06.876566 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:06.876550 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:46:07.259166 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:07.259072 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" event={"ID":"3a1c6229-58c7-412d-b67d-9a31e460a59c","Type":"ContainerStarted","Data":"ca502e0ebe562a6264e655b1f9b99bdef227a9ffd7b50f3623e30c3a9a273f13"} Nov 03 22:46:07.259166 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:07.259145 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" event={"ID":"3a1c6229-58c7-412d-b67d-9a31e460a59c","Type":"ContainerStarted","Data":"d2efb22d25e904c3dfc9cdcde5ff66e5ebba8bdf35fb429f623ad393d6a6c31e"} Nov 03 22:46:07.383629 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:07.383581 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" podStartSLOduration=1.383567242 podStartE2EDuration="1.383567242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:46:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:46:07.380307013 +0000 UTC m=+1396.689866328" watchObservedRunningTime="2026-11-03 22:46:07.383567242 +0000 UTC m=+1396.693126587" Nov 03 22:46:07.432931 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:07.432900 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" Nov 03 22:46:07.434424 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:07.434399 5196 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 22:46:07.434555 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:07.434445 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" podUID="3a1c6229-58c7-412d-b67d-9a31e460a59c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 22:46:08.432438 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:08.432404 5196 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 22:46:08.432844 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:08.432453 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" podUID="3a1c6229-58c7-412d-b67d-9a31e460a59c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 22:46:09.418457 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:09.418429 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:09.432503 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:09.432480 5196 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 22:46:09.432835 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:09.432525 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" podUID="3a1c6229-58c7-412d-b67d-9a31e460a59c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 22:46:10.432362 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:10.432332 5196 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 22:46:10.432517 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:10.432394 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" podUID="3a1c6229-58c7-412d-b67d-9a31e460a59c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 22:46:10.811722 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:10.811634 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:11.446047 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:11.446011 5196 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 22:46:11.451098 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:11.451056 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" podUID="3a1c6229-58c7-412d-b67d-9a31e460a59c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 22:46:11.453007 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:11.452981 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:11.732946 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:11.730439 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:12.445435 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:12.445378 5196 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 22:46:12.445724 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:12.445692 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" podUID="3a1c6229-58c7-412d-b67d-9a31e460a59c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 22:46:13.433879 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:13.432844 5196 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 22:46:13.439879 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:13.436451 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" podUID="3a1c6229-58c7-412d-b67d-9a31e460a59c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 22:46:15.098030 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:15.097991 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" Nov 03 22:46:15.098411 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:15.098363 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" Nov 03 22:46:15.099527 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:15.099499 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" Nov 03 22:46:15.937269 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:15.937235 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:17.653339 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:17.653306 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:20.814926 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:20.814892 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:23.661163 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:23.661135 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:24.663850 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.663819 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98"] Nov 03 22:46:24.668205 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.668188 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" Nov 03 22:46:24.714882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.714834 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz"] Nov 03 22:46:24.718818 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.718802 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" Nov 03 22:46:24.772664 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.772623 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98"] Nov 03 22:46:24.790127 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.790100 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxr98\" (UID: \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" Nov 03 22:46:24.790303 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.790135 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxr98\" (UID: \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" Nov 03 22:46:24.790303 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.790167 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjv82\" (UniqueName: \"kubernetes.io/projected/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-kube-api-access-jjv82\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxr98\" (UID: \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" Nov 03 22:46:24.790303 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.790275 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxr98\" (UID: \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" Nov 03 22:46:24.790470 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.790348 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxr98\" (UID: \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" Nov 03 22:46:24.790470 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.790383 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxr98\" (UID: \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" Nov 03 22:46:24.790470 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.790410 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxr98\" (UID: \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" Nov 03 22:46:24.790622 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.790553 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxr98\" (UID: \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" Nov 03 22:46:24.790675 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.790639 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxr98\" (UID: \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" Nov 03 22:46:24.827186 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.827156 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz"] Nov 03 22:46:24.892355 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.892316 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jrgz\" (UID: \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" Nov 03 22:46:24.892553 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.892431 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxr98\" (UID: \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" Nov 03 22:46:24.892553 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.892471 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxr98\" (UID: \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" Nov 03 22:46:24.892553 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.892497 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf5x7\" (UniqueName: \"kubernetes.io/projected/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-kube-api-access-lf5x7\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jrgz\" (UID: \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" Nov 03 22:46:24.892553 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.892548 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjv82\" (UniqueName: \"kubernetes.io/projected/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-kube-api-access-jjv82\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxr98\" (UID: \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" Nov 03 22:46:24.892774 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.892585 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxr98\" (UID: \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" Nov 03 22:46:24.892774 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.892614 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxr98\" (UID: \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" Nov 03 22:46:24.892774 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.892642 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxr98\" (UID: \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" Nov 03 22:46:24.892774 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.892699 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxr98\" (UID: \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" Nov 03 22:46:24.893019 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.892794 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jrgz\" (UID: \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" Nov 03 22:46:24.893019 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.892903 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxr98\" (UID: \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" Nov 03 22:46:24.893019 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.892929 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jrgz\" (UID: \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" Nov 03 22:46:24.893019 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.892989 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jrgz\" (UID: \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" Nov 03 22:46:24.893019 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.893015 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jrgz\" (UID: \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" Nov 03 22:46:24.893273 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.893059 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jrgz\" (UID: \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" Nov 03 22:46:24.893273 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.893102 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jrgz\" (UID: \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" Nov 03 22:46:24.893273 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.893182 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxr98\" (UID: \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" Nov 03 22:46:24.893273 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.893212 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jrgz\" (UID: \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" Nov 03 22:46:24.894257 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.894232 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxr98\" (UID: \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" Nov 03 22:46:24.895137 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.895117 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxr98\" (UID: \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" Nov 03 22:46:24.895496 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.895479 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxr98\" (UID: \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" Nov 03 22:46:24.896292 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.896267 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxr98\" (UID: \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" Nov 03 22:46:24.896635 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.896618 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxr98\" (UID: \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" Nov 03 22:46:24.897868 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.896730 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxr98\" (UID: \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" Nov 03 22:46:24.958180 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.957685 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxr98\" (UID: \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" Nov 03 22:46:24.996009 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.993996 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjv82\" (UniqueName: \"kubernetes.io/projected/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-kube-api-access-jjv82\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxr98\" (UID: \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" Nov 03 22:46:24.996009 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.994979 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf5x7\" (UniqueName: \"kubernetes.io/projected/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-kube-api-access-lf5x7\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jrgz\" (UID: \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" Nov 03 22:46:24.996009 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.995123 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jrgz\" (UID: \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" Nov 03 22:46:24.996009 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.995203 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jrgz\" (UID: \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" Nov 03 22:46:24.996009 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.995260 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jrgz\" (UID: \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" Nov 03 22:46:24.996009 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.995286 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jrgz\" (UID: \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" Nov 03 22:46:24.996009 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.995344 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jrgz\" (UID: \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" Nov 03 22:46:24.996009 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.995379 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jrgz\" (UID: \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" Nov 03 22:46:24.996009 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.995439 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jrgz\" (UID: \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" Nov 03 22:46:24.996009 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.995504 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jrgz\" (UID: \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" Nov 03 22:46:24.998957 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.998629 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jrgz\" (UID: \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" Nov 03 22:46:24.998957 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.998641 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jrgz\" (UID: \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" Nov 03 22:46:24.998957 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.998905 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jrgz\" (UID: \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" Nov 03 22:46:24.998957 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.998922 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jrgz\" (UID: \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" Nov 03 22:46:24.999272 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:24.999137 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jrgz\" (UID: \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" Nov 03 22:46:25.001071 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:25.001026 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jrgz\" (UID: \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" Nov 03 22:46:25.001570 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:25.001527 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jrgz\" (UID: \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" Nov 03 22:46:25.012271 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:25.011959 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxr98\" (UID: \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" Nov 03 22:46:25.060044 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:25.060016 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jrgz\" (UID: \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" Nov 03 22:46:25.061018 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:25.060997 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf5x7\" (UniqueName: \"kubernetes.io/projected/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-kube-api-access-lf5x7\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6jrgz\" (UID: \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" Nov 03 22:46:25.283673 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:25.283592 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" Nov 03 22:46:25.331478 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:25.331450 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" Nov 03 22:46:25.703975 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:25.703943 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:25.769666 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:25.769635 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98"] Nov 03 22:46:25.770205 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:46:25.770175 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3386155e_a5bb_4de6_aaf3_0cb3e1dacc4c.slice/crio-9d47737941cdca3d6a97c2342be7f9a93b1245be4de3fb3b2e09e0aa8f0d0559 WatchSource:0}: Error finding container 9d47737941cdca3d6a97c2342be7f9a93b1245be4de3fb3b2e09e0aa8f0d0559: Status 404 returned error can't find the container with id 9d47737941cdca3d6a97c2342be7f9a93b1245be4de3fb3b2e09e0aa8f0d0559 Nov 03 22:46:25.773873 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:25.773815 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:46:25.773989 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:25.773953 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:46:25.774081 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:25.774000 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:46:25.818379 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:25.818350 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz"] Nov 03 22:46:25.818843 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:46:25.818810 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec68f4cb_7b6d_4693_9fbb_9a10cf2604a2.slice/crio-ee8a32c08bbc7223bc0775cd74a7c6b17fef67a8b5cc0ef99893d9741446b281 WatchSource:0}: Error finding container ee8a32c08bbc7223bc0775cd74a7c6b17fef67a8b5cc0ef99893d9741446b281: Status 404 returned error can't find the container with id ee8a32c08bbc7223bc0775cd74a7c6b17fef67a8b5cc0ef99893d9741446b281 Nov 03 22:46:25.822224 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:25.822186 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:46:25.822292 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:25.822273 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:46:25.822342 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:25.822313 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:46:25.952162 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:46:25.952100 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:46:26.492929 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:26.492896 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" event={"ID":"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2","Type":"ContainerStarted","Data":"2faa7a1c976f7a0dc8b989b7d7e728293c13b15c6013e4222315347d36d2468a"} Nov 03 22:46:26.493131 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:26.492949 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" event={"ID":"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2","Type":"ContainerStarted","Data":"ee8a32c08bbc7223bc0775cd74a7c6b17fef67a8b5cc0ef99893d9741446b281"} Nov 03 22:46:26.494700 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:26.494674 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" event={"ID":"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c","Type":"ContainerStarted","Data":"0e6e2cb774a6795ad59b759598e1e51148d8c5b389b3816409eff65ded329859"} Nov 03 22:46:26.494787 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:26.494715 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" event={"ID":"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c","Type":"ContainerStarted","Data":"9d47737941cdca3d6a97c2342be7f9a93b1245be4de3fb3b2e09e0aa8f0d0559"} Nov 03 22:46:26.749567 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:26.749447 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" podStartSLOduration=2.7494271919999997 podStartE2EDuration="2.749427192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:46:26.623601693 +0000 UTC m=+1415.933161030" watchObservedRunningTime="2026-11-03 22:46:26.749427192 +0000 UTC m=+1416.058986486" Nov 03 22:46:26.750433 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:26.750381 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" podStartSLOduration=2.750369573 podStartE2EDuration="2.750369573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:46:26.746080701 +0000 UTC m=+1416.055640020" watchObservedRunningTime="2026-11-03 22:46:26.750369573 +0000 UTC m=+1416.059928866" Nov 03 22:46:27.284666 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:27.284635 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" Nov 03 22:46:27.286034 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:27.286008 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lxr98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 22:46:27.286141 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:27.286056 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" podUID="3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 22:46:27.339079 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:27.339050 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" Nov 03 22:46:27.341230 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:27.341085 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-6jrgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 22:46:27.341230 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:27.341146 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" podUID="ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 22:46:28.284078 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:28.284036 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lxr98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 22:46:28.284573 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:28.284099 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" podUID="3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 22:46:28.332340 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:28.332302 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-6jrgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 22:46:28.332924 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:28.332885 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" podUID="ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 22:46:29.284485 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:29.284449 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lxr98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 22:46:29.284902 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:29.284535 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" podUID="3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 22:46:29.332884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:29.332234 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-6jrgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 22:46:29.332884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:29.332299 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" podUID="ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 22:46:30.284442 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:30.284403 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lxr98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 22:46:30.284613 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:30.284474 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" podUID="3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 22:46:30.336248 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:30.336210 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-6jrgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 22:46:30.336502 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:30.336475 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" podUID="ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 22:46:31.128831 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:31.128797 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:31.284489 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:31.284437 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lxr98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 22:46:31.284711 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:31.284521 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" podUID="3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 22:46:31.340884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:31.340288 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-6jrgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 22:46:31.340884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:31.340371 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" podUID="ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 22:46:32.284732 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:32.284700 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lxr98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 22:46:32.285143 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:32.284756 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" podUID="3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 22:46:32.332310 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:32.332274 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-6jrgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 22:46:32.332500 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:32.332343 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" podUID="ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 22:46:33.284765 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:33.284729 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lxr98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 22:46:33.285183 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:33.284786 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" podUID="3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 22:46:33.332014 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:33.331978 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-6jrgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 22:46:33.332217 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:33.332037 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" podUID="ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 22:46:34.798835 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:34.798796 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" Nov 03 22:46:34.799883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:34.799833 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" Nov 03 22:46:34.801529 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:34.801505 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" Nov 03 22:46:34.982583 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:34.981340 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" Nov 03 22:46:34.982583 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:34.981793 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" Nov 03 22:46:34.983178 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:34.983158 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" Nov 03 22:46:36.041269 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:36.041233 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:40.803269 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:40.803232 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:41.205642 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:41.205611 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:43.956911 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:43.956838 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:44.103111 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:44.103068 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:46.023596 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:46.023555 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:47.046623 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:47.046588 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:49.381435 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:49.381406 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:49.527140 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:49.527109 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:50.803290 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:50.803249 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:55.259050 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:55.259013 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1cb89500-56959c5b65-nx997"] Nov 03 22:46:55.263775 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:55.263749 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cb89500-56959c5b65-nx997" Nov 03 22:46:55.316959 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:55.316937 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1cb89500-serving-cert\"" Nov 03 22:46:55.350388 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:55.350356 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecc35923-4d08-43be-91e7-6e905385fc13-openshift-service-ca\") pod \"maas-api-e2e-ait-1cb89500-56959c5b65-nx997\" (UID: \"ecc35923-4d08-43be-91e7-6e905385fc13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cb89500-56959c5b65-nx997" Nov 03 22:46:55.350519 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:55.350396 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xghb4\" (UniqueName: \"kubernetes.io/projected/ecc35923-4d08-43be-91e7-6e905385fc13-kube-api-access-xghb4\") pod \"maas-api-e2e-ait-1cb89500-56959c5b65-nx997\" (UID: \"ecc35923-4d08-43be-91e7-6e905385fc13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cb89500-56959c5b65-nx997" Nov 03 22:46:55.350692 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:55.350665 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecc35923-4d08-43be-91e7-6e905385fc13-maas-api-tls\") pod \"maas-api-e2e-ait-1cb89500-56959c5b65-nx997\" (UID: \"ecc35923-4d08-43be-91e7-6e905385fc13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cb89500-56959c5b65-nx997" Nov 03 22:46:55.350765 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:55.350750 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecc35923-4d08-43be-91e7-6e905385fc13-metrics-certs\") pod \"maas-api-e2e-ait-1cb89500-56959c5b65-nx997\" (UID: \"ecc35923-4d08-43be-91e7-6e905385fc13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cb89500-56959c5b65-nx997" Nov 03 22:46:55.371778 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:55.371748 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1cb89500-56959c5b65-nx997"] Nov 03 22:46:55.446760 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:55.446726 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:55.452565 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:55.452539 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecc35923-4d08-43be-91e7-6e905385fc13-maas-api-tls\") pod \"maas-api-e2e-ait-1cb89500-56959c5b65-nx997\" (UID: \"ecc35923-4d08-43be-91e7-6e905385fc13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cb89500-56959c5b65-nx997" Nov 03 22:46:55.452707 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:55.452587 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecc35923-4d08-43be-91e7-6e905385fc13-metrics-certs\") pod \"maas-api-e2e-ait-1cb89500-56959c5b65-nx997\" (UID: \"ecc35923-4d08-43be-91e7-6e905385fc13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cb89500-56959c5b65-nx997" Nov 03 22:46:55.452707 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:46:55.452696 5196 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-1cb89500-serving-cert: secret "maas-api-e2e-ait-1cb89500-serving-cert" not found Nov 03 22:46:55.452800 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:55.452703 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecc35923-4d08-43be-91e7-6e905385fc13-openshift-service-ca\") pod \"maas-api-e2e-ait-1cb89500-56959c5b65-nx997\" (UID: \"ecc35923-4d08-43be-91e7-6e905385fc13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cb89500-56959c5b65-nx997" Nov 03 22:46:55.452800 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:46:55.452778 5196 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecc35923-4d08-43be-91e7-6e905385fc13-maas-api-tls podName:ecc35923-4d08-43be-91e7-6e905385fc13 nodeName:}" failed. No retries permitted until 2026-11-03 22:46:55.952756508 +0000 UTC m=+1445.262315793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ecc35923-4d08-43be-91e7-6e905385fc13-maas-api-tls") pod "maas-api-e2e-ait-1cb89500-56959c5b65-nx997" (UID: "ecc35923-4d08-43be-91e7-6e905385fc13") : secret "maas-api-e2e-ait-1cb89500-serving-cert" not found Nov 03 22:46:55.452936 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:55.452835 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xghb4\" (UniqueName: \"kubernetes.io/projected/ecc35923-4d08-43be-91e7-6e905385fc13-kube-api-access-xghb4\") pod \"maas-api-e2e-ait-1cb89500-56959c5b65-nx997\" (UID: \"ecc35923-4d08-43be-91e7-6e905385fc13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cb89500-56959c5b65-nx997" Nov 03 22:46:55.453399 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:55.453368 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecc35923-4d08-43be-91e7-6e905385fc13-openshift-service-ca\") pod \"maas-api-e2e-ait-1cb89500-56959c5b65-nx997\" (UID: \"ecc35923-4d08-43be-91e7-6e905385fc13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cb89500-56959c5b65-nx997" Nov 03 22:46:55.455217 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:55.455196 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecc35923-4d08-43be-91e7-6e905385fc13-metrics-certs\") pod \"maas-api-e2e-ait-1cb89500-56959c5b65-nx997\" (UID: \"ecc35923-4d08-43be-91e7-6e905385fc13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cb89500-56959c5b65-nx997" Nov 03 22:46:55.527746 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:55.527645 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xghb4\" (UniqueName: \"kubernetes.io/projected/ecc35923-4d08-43be-91e7-6e905385fc13-kube-api-access-xghb4\") pod \"maas-api-e2e-ait-1cb89500-56959c5b65-nx997\" (UID: \"ecc35923-4d08-43be-91e7-6e905385fc13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cb89500-56959c5b65-nx997" Nov 03 22:46:55.612236 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:55.612203 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:55.764221 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:55.764150 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:55.921808 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:46:55.921769 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:46:55.963344 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:55.963318 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecc35923-4d08-43be-91e7-6e905385fc13-maas-api-tls\") pod \"maas-api-e2e-ait-1cb89500-56959c5b65-nx997\" (UID: \"ecc35923-4d08-43be-91e7-6e905385fc13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cb89500-56959c5b65-nx997" Nov 03 22:46:55.965666 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:55.965649 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecc35923-4d08-43be-91e7-6e905385fc13-maas-api-tls\") pod \"maas-api-e2e-ait-1cb89500-56959c5b65-nx997\" (UID: \"ecc35923-4d08-43be-91e7-6e905385fc13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cb89500-56959c5b65-nx997" Nov 03 22:46:56.157302 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:56.157264 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:46:56.177618 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:56.177546 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cb89500-56959c5b65-nx997" Nov 03 22:46:56.560833 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:56.560806 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1cb89500-56959c5b65-nx997"] Nov 03 22:46:56.561646 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:46:56.561614 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc35923_4d08_43be_91e7_6e905385fc13.slice/crio-9d87794c322baefab2610423c3651cc4153b79d2053b945874d90c9d29b0ab7c WatchSource:0}: Error finding container 9d87794c322baefab2610423c3651cc4153b79d2053b945874d90c9d29b0ab7c: Status 404 returned error can't find the container with id 9d87794c322baefab2610423c3651cc4153b79d2053b945874d90c9d29b0ab7c Nov 03 22:46:56.780423 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:56.780386 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cb89500-56959c5b65-nx997" event={"ID":"ecc35923-4d08-43be-91e7-6e905385fc13","Type":"ContainerStarted","Data":"9d87794c322baefab2610423c3651cc4153b79d2053b945874d90c9d29b0ab7c"} Nov 03 22:46:57.789416 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:57.789333 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cb89500-56959c5b65-nx997" event={"ID":"ecc35923-4d08-43be-91e7-6e905385fc13","Type":"ContainerStarted","Data":"c540476fb78413c110fe33d282e0a57d12d4fd6da39a6477f340568c402c998f"} Nov 03 22:46:57.789799 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:57.789464 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cb89500-56959c5b65-nx997" Nov 03 22:46:57.910706 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:46:57.910658 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cb89500-56959c5b65-nx997" podStartSLOduration=2.096931598 podStartE2EDuration="2.910643574s" totalImagesPullingTime="813.711976ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:46:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:46:57.90792612 +0000 UTC m=+1447.217485414" watchObservedRunningTime="2026-11-03 22:46:57.910643574 +0000 UTC m=+1447.220202929" Nov 03 22:47:00.036311 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:00.036220 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:00.528309 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:00.528262 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:02.729785 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:02.726711 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd"] Nov 03 22:47:02.739176 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:02.739148 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" Nov 03 22:47:02.792677 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:02.792640 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be26434-d0d6-43e2-a89f-39d6533660d8-workload-certs\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd\" (UID: \"4be26434-d0d6-43e2-a89f-39d6533660d8\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" Nov 03 22:47:02.793078 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:02.793054 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4be26434-d0d6-43e2-a89f-39d6533660d8-istio-token\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd\" (UID: \"4be26434-d0d6-43e2-a89f-39d6533660d8\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" Nov 03 22:47:02.793408 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:02.793389 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be26434-d0d6-43e2-a89f-39d6533660d8-istio-podinfo\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd\" (UID: \"4be26434-d0d6-43e2-a89f-39d6533660d8\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" Nov 03 22:47:02.804818 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:02.804153 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-fc129ccf-gw-openshift-default-dockercfg-fjr2t\"" Nov 03 22:47:02.805098 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:02.805011 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be26434-d0d6-43e2-a89f-39d6533660d8-credential-socket\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd\" (UID: \"4be26434-d0d6-43e2-a89f-39d6533660d8\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" Nov 03 22:47:02.805392 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:02.805373 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be26434-d0d6-43e2-a89f-39d6533660d8-istio-envoy\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd\" (UID: \"4be26434-d0d6-43e2-a89f-39d6533660d8\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" Nov 03 22:47:02.805628 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:02.805562 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be26434-d0d6-43e2-a89f-39d6533660d8-istiod-ca-cert\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd\" (UID: \"4be26434-d0d6-43e2-a89f-39d6533660d8\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" Nov 03 22:47:02.805869 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:02.805785 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be26434-d0d6-43e2-a89f-39d6533660d8-istio-data\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd\" (UID: \"4be26434-d0d6-43e2-a89f-39d6533660d8\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" Nov 03 22:47:02.806177 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:02.806112 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-668m5\" (UniqueName: \"kubernetes.io/projected/4be26434-d0d6-43e2-a89f-39d6533660d8-kube-api-access-668m5\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd\" (UID: \"4be26434-d0d6-43e2-a89f-39d6533660d8\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" Nov 03 22:47:02.806410 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:02.806393 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be26434-d0d6-43e2-a89f-39d6533660d8-workload-socket\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd\" (UID: \"4be26434-d0d6-43e2-a89f-39d6533660d8\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" Nov 03 22:47:02.867353 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:02.867313 5196 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" podUID="26fa51c2-0f2e-482f-9c67-0ea7924d7b3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:47:02.921957 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:02.921921 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be26434-d0d6-43e2-a89f-39d6533660d8-workload-socket\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd\" (UID: \"4be26434-d0d6-43e2-a89f-39d6533660d8\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" Nov 03 22:47:02.922310 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:02.922284 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be26434-d0d6-43e2-a89f-39d6533660d8-workload-certs\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd\" (UID: \"4be26434-d0d6-43e2-a89f-39d6533660d8\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" Nov 03 22:47:02.922493 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:02.922477 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4be26434-d0d6-43e2-a89f-39d6533660d8-istio-token\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd\" (UID: \"4be26434-d0d6-43e2-a89f-39d6533660d8\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" Nov 03 22:47:02.937896 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:02.922614 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be26434-d0d6-43e2-a89f-39d6533660d8-istio-podinfo\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd\" (UID: \"4be26434-d0d6-43e2-a89f-39d6533660d8\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" Nov 03 22:47:02.938172 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:02.938146 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be26434-d0d6-43e2-a89f-39d6533660d8-credential-socket\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd\" (UID: \"4be26434-d0d6-43e2-a89f-39d6533660d8\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" Nov 03 22:47:02.938312 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:02.938297 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be26434-d0d6-43e2-a89f-39d6533660d8-istio-envoy\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd\" (UID: \"4be26434-d0d6-43e2-a89f-39d6533660d8\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" Nov 03 22:47:02.938436 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:02.938420 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be26434-d0d6-43e2-a89f-39d6533660d8-istiod-ca-cert\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd\" (UID: \"4be26434-d0d6-43e2-a89f-39d6533660d8\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" Nov 03 22:47:02.938569 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:02.938554 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be26434-d0d6-43e2-a89f-39d6533660d8-istio-data\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd\" (UID: \"4be26434-d0d6-43e2-a89f-39d6533660d8\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" Nov 03 22:47:02.938816 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:02.938797 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-668m5\" (UniqueName: \"kubernetes.io/projected/4be26434-d0d6-43e2-a89f-39d6533660d8-kube-api-access-668m5\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd\" (UID: \"4be26434-d0d6-43e2-a89f-39d6533660d8\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" Nov 03 22:47:02.967882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:02.940064 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be26434-d0d6-43e2-a89f-39d6533660d8-workload-socket\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd\" (UID: \"4be26434-d0d6-43e2-a89f-39d6533660d8\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" Nov 03 22:47:02.989804 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:02.989731 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be26434-d0d6-43e2-a89f-39d6533660d8-workload-certs\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd\" (UID: \"4be26434-d0d6-43e2-a89f-39d6533660d8\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" Nov 03 22:47:03.044878 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:03.044194 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be26434-d0d6-43e2-a89f-39d6533660d8-credential-socket\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd\" (UID: \"4be26434-d0d6-43e2-a89f-39d6533660d8\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" Nov 03 22:47:03.049876 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:03.048222 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be26434-d0d6-43e2-a89f-39d6533660d8-istio-envoy\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd\" (UID: \"4be26434-d0d6-43e2-a89f-39d6533660d8\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" Nov 03 22:47:03.049876 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:03.048925 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be26434-d0d6-43e2-a89f-39d6533660d8-istiod-ca-cert\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd\" (UID: \"4be26434-d0d6-43e2-a89f-39d6533660d8\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" Nov 03 22:47:03.049876 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:03.049216 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be26434-d0d6-43e2-a89f-39d6533660d8-istio-data\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd\" (UID: \"4be26434-d0d6-43e2-a89f-39d6533660d8\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" Nov 03 22:47:03.061885 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:03.050206 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be26434-d0d6-43e2-a89f-39d6533660d8-istio-podinfo\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd\" (UID: \"4be26434-d0d6-43e2-a89f-39d6533660d8\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" Nov 03 22:47:03.096884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:03.096827 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd"] Nov 03 22:47:03.097778 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:03.097518 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" podUID="34e9cf3a-3456-4835-8933-102ce735cccc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:47:03.128510 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:03.104061 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4be26434-d0d6-43e2-a89f-39d6533660d8-istio-token\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd\" (UID: \"4be26434-d0d6-43e2-a89f-39d6533660d8\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" Nov 03 22:47:03.128510 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:03.121689 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-668m5\" (UniqueName: \"kubernetes.io/projected/4be26434-d0d6-43e2-a89f-39d6533660d8-kube-api-access-668m5\") pod \"e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd\" (UID: \"4be26434-d0d6-43e2-a89f-39d6533660d8\") " pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" Nov 03 22:47:03.407350 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:03.407317 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" Nov 03 22:47:03.801547 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:03.801469 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cb89500-56959c5b65-nx997" Nov 03 22:47:03.845804 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:03.845778 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd"] Nov 03 22:47:03.846355 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:47:03.846328 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be26434_d0d6_43e2_a89f_39d6533660d8.slice/crio-827adc0b738f1ae16019aadb448f3c106cd6f1e3dad62240a02ca3d286c43f70 WatchSource:0}: Error finding container 827adc0b738f1ae16019aadb448f3c106cd6f1e3dad62240a02ca3d286c43f70: Status 404 returned error can't find the container with id 827adc0b738f1ae16019aadb448f3c106cd6f1e3dad62240a02ca3d286c43f70 Nov 03 22:47:03.848908 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:03.848878 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:47:03.848984 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:03.848943 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:47:03.848984 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:03.848970 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:47:04.106319 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:04.106285 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" event={"ID":"4be26434-d0d6-43e2-a89f-39d6533660d8","Type":"ContainerStarted","Data":"ee56c17d9c5f4e64993c0113c5269e222fc2f171fcd5e49625c279abc2980143"} Nov 03 22:47:04.106529 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:04.106342 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" event={"ID":"4be26434-d0d6-43e2-a89f-39d6533660d8","Type":"ContainerStarted","Data":"827adc0b738f1ae16019aadb448f3c106cd6f1e3dad62240a02ca3d286c43f70"} Nov 03 22:47:04.229586 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:04.229510 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" podStartSLOduration=2.229482787 podStartE2EDuration="2.229482787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:47:04.227749875 +0000 UTC m=+1453.537309169" watchObservedRunningTime="2026-11-03 22:47:04.229482787 +0000 UTC m=+1453.539042082" Nov 03 22:47:04.358365 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:04.358276 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:04.407899 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:04.407871 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" Nov 03 22:47:04.409459 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:04.409435 5196 patch_prober.go:29] interesting pod/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 22:47:04.409597 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:04.409490 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" podUID="4be26434-d0d6-43e2-a89f-39d6533660d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 22:47:05.416018 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:05.415975 5196 patch_prober.go:29] interesting pod/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 22:47:05.416530 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:05.416057 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" podUID="4be26434-d0d6-43e2-a89f-39d6533660d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 22:47:06.305465 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:06.305432 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:06.415297 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:06.415254 5196 patch_prober.go:29] interesting pod/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 22:47:06.415484 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:06.415330 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" podUID="4be26434-d0d6-43e2-a89f-39d6533660d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 22:47:06.914884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:06.914465 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:07.410883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:07.410234 5196 patch_prober.go:29] interesting pod/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 22:47:07.410883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:07.410299 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" podUID="4be26434-d0d6-43e2-a89f-39d6533660d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 22:47:08.415364 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:08.415320 5196 patch_prober.go:29] interesting pod/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 22:47:08.415899 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:08.415391 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" podUID="4be26434-d0d6-43e2-a89f-39d6533660d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 22:47:09.411883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:09.408190 5196 patch_prober.go:29] interesting pod/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 22:47:09.411883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:09.408248 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" podUID="4be26434-d0d6-43e2-a89f-39d6533660d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 22:47:09.814405 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:09.814320 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:10.412522 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:10.412496 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" Nov 03 22:47:10.413004 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:10.412991 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" Nov 03 22:47:10.413850 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:10.413835 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" Nov 03 22:47:10.425209 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:10.425182 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:12.833004 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:12.832972 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:12.979849 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:12.979822 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:14.477094 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:14.477064 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v"] Nov 03 22:47:14.477552 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:14.477260 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v" podUID="5054337e-aaf6-42ca-a75c-ab1fd174afeb" containerName="maas-api" containerID="cri-o://467b6779293474097a0053d7be46f64a86e8c524bef407df0bcaf56229d74ef4" gracePeriod=30 Nov 03 22:47:14.581187 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:14.581124 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:14.928839 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:14.928814 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v" Nov 03 22:47:15.054558 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:15.054464 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5054337e-aaf6-42ca-a75c-ab1fd174afeb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5054337e-aaf6-42ca-a75c-ab1fd174afeb-openshift-service-ca\") pod \"5054337e-aaf6-42ca-a75c-ab1fd174afeb\" (UID: \"5054337e-aaf6-42ca-a75c-ab1fd174afeb\") " Nov 03 22:47:15.054919 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:15.054895 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5054337e-aaf6-42ca-a75c-ab1fd174afeb-kube-api-access-hhfv2\" (UniqueName: \"kubernetes.io/projected/5054337e-aaf6-42ca-a75c-ab1fd174afeb-kube-api-access-hhfv2\") pod \"5054337e-aaf6-42ca-a75c-ab1fd174afeb\" (UID: \"5054337e-aaf6-42ca-a75c-ab1fd174afeb\") " Nov 03 22:47:15.055307 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:15.055290 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5054337e-aaf6-42ca-a75c-ab1fd174afeb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5054337e-aaf6-42ca-a75c-ab1fd174afeb-maas-api-tls\") pod \"5054337e-aaf6-42ca-a75c-ab1fd174afeb\" (UID: \"5054337e-aaf6-42ca-a75c-ab1fd174afeb\") " Nov 03 22:47:15.055557 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:15.055538 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5054337e-aaf6-42ca-a75c-ab1fd174afeb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5054337e-aaf6-42ca-a75c-ab1fd174afeb-metrics-certs\") pod \"5054337e-aaf6-42ca-a75c-ab1fd174afeb\" (UID: \"5054337e-aaf6-42ca-a75c-ab1fd174afeb\") " Nov 03 22:47:15.065257 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:15.065210 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5054337e-aaf6-42ca-a75c-ab1fd174afeb-openshift-service-ca" pod "5054337e-aaf6-42ca-a75c-ab1fd174afeb" (UID: "5054337e-aaf6-42ca-a75c-ab1fd174afeb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:15.067764 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:15.067734 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5054337e-aaf6-42ca-a75c-ab1fd174afeb-metrics-certs" pod "5054337e-aaf6-42ca-a75c-ab1fd174afeb" (UID: "5054337e-aaf6-42ca-a75c-ab1fd174afeb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:47:15.069311 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:15.069275 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5054337e-aaf6-42ca-a75c-ab1fd174afeb-maas-api-tls" pod "5054337e-aaf6-42ca-a75c-ab1fd174afeb" (UID: "5054337e-aaf6-42ca-a75c-ab1fd174afeb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:47:15.071016 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:15.070995 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5054337e-aaf6-42ca-a75c-ab1fd174afeb-kube-api-access-hhfv2" pod "5054337e-aaf6-42ca-a75c-ab1fd174afeb" (UID: "5054337e-aaf6-42ca-a75c-ab1fd174afeb"). InnerVolumeSpecName "kube-api-access-hhfv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:15.157361 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:15.157328 5196 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5054337e-aaf6-42ca-a75c-ab1fd174afeb-openshift-service-ca\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:47:15.157361 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:15.157357 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhfv2\" (UniqueName: \"kubernetes.io/projected/5054337e-aaf6-42ca-a75c-ab1fd174afeb-kube-api-access-hhfv2\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:47:15.157361 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:15.157369 5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5054337e-aaf6-42ca-a75c-ab1fd174afeb-maas-api-tls\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:47:15.157640 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:15.157378 5196 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5054337e-aaf6-42ca-a75c-ab1fd174afeb-metrics-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:47:15.188913 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:15.188882 5196 generic.go:356] "Generic (PLEG): container finished" podID="5054337e-aaf6-42ca-a75c-ab1fd174afeb" containerID="467b6779293474097a0053d7be46f64a86e8c524bef407df0bcaf56229d74ef4" exitCode=0 Nov 03 22:47:15.189067 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:15.188930 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v" event={"ID":"5054337e-aaf6-42ca-a75c-ab1fd174afeb","Type":"ContainerDied","Data":"467b6779293474097a0053d7be46f64a86e8c524bef407df0bcaf56229d74ef4"} Nov 03 22:47:15.189067 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:15.188950 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v" Nov 03 22:47:15.189067 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:15.188998 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v" event={"ID":"5054337e-aaf6-42ca-a75c-ab1fd174afeb","Type":"ContainerDied","Data":"5fb1e44613971cbe150f4a832cfb8f7b98dabed764fba0cd84a154c4ac6fadf6"} Nov 03 22:47:15.189067 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:15.189020 5196 scope.go:122] "RemoveContainer" containerID="467b6779293474097a0053d7be46f64a86e8c524bef407df0bcaf56229d74ef4" Nov 03 22:47:15.199871 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:15.199840 5196 scope.go:122] "RemoveContainer" containerID="467b6779293474097a0053d7be46f64a86e8c524bef407df0bcaf56229d74ef4" Nov 03 22:47:15.200129 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:47:15.200111 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"467b6779293474097a0053d7be46f64a86e8c524bef407df0bcaf56229d74ef4\": container with ID starting with 467b6779293474097a0053d7be46f64a86e8c524bef407df0bcaf56229d74ef4 not found: ID does not exist" containerID="467b6779293474097a0053d7be46f64a86e8c524bef407df0bcaf56229d74ef4" Nov 03 22:47:15.200184 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:15.200138 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"467b6779293474097a0053d7be46f64a86e8c524bef407df0bcaf56229d74ef4"} err="failed to get container status \"467b6779293474097a0053d7be46f64a86e8c524bef407df0bcaf56229d74ef4\": rpc error: code = NotFound desc = could not find container \"467b6779293474097a0053d7be46f64a86e8c524bef407df0bcaf56229d74ef4\": container with ID starting with 467b6779293474097a0053d7be46f64a86e8c524bef407df0bcaf56229d74ef4 not found: ID does not exist" Nov 03 22:47:15.358700 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:15.358674 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v"] Nov 03 22:47:15.361988 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:15.361962 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-468d8d1b-5ffc65b784-d665v"] Nov 03 22:47:16.127153 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:16.127120 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:16.351444 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:16.351417 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:16.503175 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:16.503070 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:16.656212 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:16.656177 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:16.810602 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:16.810509 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:16.818602 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:16.818564 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5054337e-aaf6-42ca-a75c-ab1fd174afeb" path="/var/lib/kubelet/pods/5054337e-aaf6-42ca-a75c-ab1fd174afeb/volumes" Nov 03 22:47:17.011499 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:17.011473 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:17.377049 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:17.377013 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:18.142682 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:18.139425 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:18.477220 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:18.477126 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:20.742290 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:20.742258 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx"] Nov 03 22:47:20.742682 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:20.742513 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx" podUID="02c2015e-ae6f-49d2-bc3c-934aa8c07f55" containerName="maas-api" containerID="cri-o://f7c1c661a86ee16b0c13b3241100a68641c4b105debbce731b3eb6dc01953d67" gracePeriod=30 Nov 03 22:47:20.849620 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:20.849587 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:21.132201 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:21.132153 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx" Nov 03 22:47:21.205790 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:21.203665 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02c2015e-ae6f-49d2-bc3c-934aa8c07f55-kube-api-access-jg89h\" (UniqueName: \"kubernetes.io/projected/02c2015e-ae6f-49d2-bc3c-934aa8c07f55-kube-api-access-jg89h\") pod \"02c2015e-ae6f-49d2-bc3c-934aa8c07f55\" (UID: \"02c2015e-ae6f-49d2-bc3c-934aa8c07f55\") " Nov 03 22:47:21.205790 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:21.204042 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02c2015e-ae6f-49d2-bc3c-934aa8c07f55-metrics-certs\" (UniqueName: \"kubernetes.io/secret/02c2015e-ae6f-49d2-bc3c-934aa8c07f55-metrics-certs\") pod \"02c2015e-ae6f-49d2-bc3c-934aa8c07f55\" (UID: \"02c2015e-ae6f-49d2-bc3c-934aa8c07f55\") " Nov 03 22:47:21.205790 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:21.204132 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02c2015e-ae6f-49d2-bc3c-934aa8c07f55-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c2015e-ae6f-49d2-bc3c-934aa8c07f55-openshift-service-ca\") pod \"02c2015e-ae6f-49d2-bc3c-934aa8c07f55\" (UID: \"02c2015e-ae6f-49d2-bc3c-934aa8c07f55\") " Nov 03 22:47:21.205790 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:21.204536 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02c2015e-ae6f-49d2-bc3c-934aa8c07f55-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02c2015e-ae6f-49d2-bc3c-934aa8c07f55-maas-api-tls\") pod \"02c2015e-ae6f-49d2-bc3c-934aa8c07f55\" (UID: \"02c2015e-ae6f-49d2-bc3c-934aa8c07f55\") " Nov 03 22:47:21.205790 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:21.205711 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02c2015e-ae6f-49d2-bc3c-934aa8c07f55-openshift-service-ca" pod "02c2015e-ae6f-49d2-bc3c-934aa8c07f55" (UID: "02c2015e-ae6f-49d2-bc3c-934aa8c07f55"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:21.208057 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:21.208014 5196 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c2015e-ae6f-49d2-bc3c-934aa8c07f55-openshift-service-ca\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:47:21.213378 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:21.213351 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02c2015e-ae6f-49d2-bc3c-934aa8c07f55-kube-api-access-jg89h" pod "02c2015e-ae6f-49d2-bc3c-934aa8c07f55" (UID: "02c2015e-ae6f-49d2-bc3c-934aa8c07f55"). InnerVolumeSpecName "kube-api-access-jg89h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:21.225694 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:21.215191 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02c2015e-ae6f-49d2-bc3c-934aa8c07f55-maas-api-tls" pod "02c2015e-ae6f-49d2-bc3c-934aa8c07f55" (UID: "02c2015e-ae6f-49d2-bc3c-934aa8c07f55"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:47:21.225790 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:21.223999 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02c2015e-ae6f-49d2-bc3c-934aa8c07f55-metrics-certs" pod "02c2015e-ae6f-49d2-bc3c-934aa8c07f55" (UID: "02c2015e-ae6f-49d2-bc3c-934aa8c07f55"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:47:21.277901 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:21.277403 5196 generic.go:356] "Generic (PLEG): container finished" podID="02c2015e-ae6f-49d2-bc3c-934aa8c07f55" containerID="f7c1c661a86ee16b0c13b3241100a68641c4b105debbce731b3eb6dc01953d67" exitCode=0 Nov 03 22:47:21.277901 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:21.277644 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx" event={"ID":"02c2015e-ae6f-49d2-bc3c-934aa8c07f55","Type":"ContainerDied","Data":"f7c1c661a86ee16b0c13b3241100a68641c4b105debbce731b3eb6dc01953d67"} Nov 03 22:47:21.277901 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:21.277707 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx" event={"ID":"02c2015e-ae6f-49d2-bc3c-934aa8c07f55","Type":"ContainerDied","Data":"69e0383ed83f52c9fa1b3fa8c94f5c4f8341d08587282df6ec76a73113daba86"} Nov 03 22:47:21.277901 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:21.277742 5196 scope.go:122] "RemoveContainer" containerID="f7c1c661a86ee16b0c13b3241100a68641c4b105debbce731b3eb6dc01953d67" Nov 03 22:47:21.278245 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:21.277928 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx" Nov 03 22:47:21.300844 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:21.300823 5196 scope.go:122] "RemoveContainer" containerID="f7c1c661a86ee16b0c13b3241100a68641c4b105debbce731b3eb6dc01953d67" Nov 03 22:47:21.301320 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:47:21.301292 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7c1c661a86ee16b0c13b3241100a68641c4b105debbce731b3eb6dc01953d67\": container with ID starting with f7c1c661a86ee16b0c13b3241100a68641c4b105debbce731b3eb6dc01953d67 not found: ID does not exist" containerID="f7c1c661a86ee16b0c13b3241100a68641c4b105debbce731b3eb6dc01953d67" Nov 03 22:47:21.301472 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:21.301444 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7c1c661a86ee16b0c13b3241100a68641c4b105debbce731b3eb6dc01953d67"} err="failed to get container status \"f7c1c661a86ee16b0c13b3241100a68641c4b105debbce731b3eb6dc01953d67\": rpc error: code = NotFound desc = could not find container \"f7c1c661a86ee16b0c13b3241100a68641c4b105debbce731b3eb6dc01953d67\": container with ID starting with f7c1c661a86ee16b0c13b3241100a68641c4b105debbce731b3eb6dc01953d67 not found: ID does not exist" Nov 03 22:47:21.310502 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:21.309474 5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02c2015e-ae6f-49d2-bc3c-934aa8c07f55-maas-api-tls\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:47:21.310502 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:21.309508 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg89h\" (UniqueName: \"kubernetes.io/projected/02c2015e-ae6f-49d2-bc3c-934aa8c07f55-kube-api-access-jg89h\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:47:21.310502 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:21.309525 5196 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02c2015e-ae6f-49d2-bc3c-934aa8c07f55-metrics-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:47:21.453793 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:21.453761 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx"] Nov 03 22:47:21.457230 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:21.457203 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx"] Nov 03 22:47:21.797244 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:21.797165 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:21.963428 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:21.963393 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:22.263591 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:22.263562 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:22.592148 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:22.592117 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:22.761582 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:22.761543 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:22.827561 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:22.827522 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02c2015e-ae6f-49d2-bc3c-934aa8c07f55" path="/var/lib/kubelet/pods/02c2015e-ae6f-49d2-bc3c-934aa8c07f55/volumes" Nov 03 22:47:22.914085 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:22.913990 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:23.226617 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:23.226533 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd"] Nov 03 22:47:23.233494 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:23.233420 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" podUID="4be26434-d0d6-43e2-a89f-39d6533660d8" containerName="istio-proxy" containerID="cri-o://ee56c17d9c5f4e64993c0113c5269e222fc2f171fcd5e49625c279abc2980143" gracePeriod=30 Nov 03 22:47:23.234323 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:23.234303 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:23.808129 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:23.808088 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:24.103254 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.103216 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc129ccf-6c65fb67ff-5r5qx" podUID="02c2015e-ae6f-49d2-bc3c-934aa8c07f55" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.80:8443/health\": context deadline exceeded" Nov 03 22:47:24.328078 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.328040 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td"] Nov 03 22:47:24.328676 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.328661 5196 cpu_manager.go:414] "containerMap: removing container" podUID="5054337e-aaf6-42ca-a75c-ab1fd174afeb" containerName="maas-api" Nov 03 22:47:24.328676 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.328676 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5054337e-aaf6-42ca-a75c-ab1fd174afeb" containerName="maas-api" Nov 03 22:47:24.328813 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.328686 5196 cpu_manager.go:414] "containerMap: removing container" podUID="02c2015e-ae6f-49d2-bc3c-934aa8c07f55" containerName="maas-api" Nov 03 22:47:24.328813 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.328691 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02c2015e-ae6f-49d2-bc3c-934aa8c07f55" containerName="maas-api" Nov 03 22:47:24.328813 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.328799 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="5054337e-aaf6-42ca-a75c-ab1fd174afeb" containerName="maas-api" Nov 03 22:47:24.328813 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.328806 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="02c2015e-ae6f-49d2-bc3c-934aa8c07f55" containerName="maas-api" Nov 03 22:47:24.333872 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.333828 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:47:24.374521 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.374428 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae839647-c999-4479-9174-3e4c389d24b8-istio-data\") pod \"e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td\" (UID: \"ae839647-c999-4479-9174-3e4c389d24b8\") " pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:47:24.374698 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.374569 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae839647-c999-4479-9174-3e4c389d24b8-istio-token\") pod \"e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td\" (UID: \"ae839647-c999-4479-9174-3e4c389d24b8\") " pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:47:24.374698 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.374682 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae839647-c999-4479-9174-3e4c389d24b8-istio-envoy\") pod \"e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td\" (UID: \"ae839647-c999-4479-9174-3e4c389d24b8\") " pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:47:24.374929 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.374905 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae839647-c999-4479-9174-3e4c389d24b8-workload-certs\") pod \"e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td\" (UID: \"ae839647-c999-4479-9174-3e4c389d24b8\") " pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:47:24.375028 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.374948 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae839647-c999-4479-9174-3e4c389d24b8-workload-socket\") pod \"e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td\" (UID: \"ae839647-c999-4479-9174-3e4c389d24b8\") " pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:47:24.375089 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.375022 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzrn9\" (UniqueName: \"kubernetes.io/projected/ae839647-c999-4479-9174-3e4c389d24b8-kube-api-access-kzrn9\") pod \"e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td\" (UID: \"ae839647-c999-4479-9174-3e4c389d24b8\") " pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:47:24.375153 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.375118 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae839647-c999-4479-9174-3e4c389d24b8-credential-socket\") pod \"e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td\" (UID: \"ae839647-c999-4479-9174-3e4c389d24b8\") " pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:47:24.375208 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.375185 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae839647-c999-4479-9174-3e4c389d24b8-istio-podinfo\") pod \"e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td\" (UID: \"ae839647-c999-4479-9174-3e4c389d24b8\") " pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:47:24.375277 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.375262 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae839647-c999-4479-9174-3e4c389d24b8-istiod-ca-cert\") pod \"e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td\" (UID: \"ae839647-c999-4479-9174-3e4c389d24b8\") " pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:47:24.400695 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.400659 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td"] Nov 03 22:47:24.425994 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.425960 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:24.477056 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.477017 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzrn9\" (UniqueName: \"kubernetes.io/projected/ae839647-c999-4479-9174-3e4c389d24b8-kube-api-access-kzrn9\") pod \"e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td\" (UID: \"ae839647-c999-4479-9174-3e4c389d24b8\") " pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:47:24.477273 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.477085 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae839647-c999-4479-9174-3e4c389d24b8-credential-socket\") pod \"e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td\" (UID: \"ae839647-c999-4479-9174-3e4c389d24b8\") " pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:47:24.477273 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.477150 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae839647-c999-4479-9174-3e4c389d24b8-istio-podinfo\") pod \"e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td\" (UID: \"ae839647-c999-4479-9174-3e4c389d24b8\") " pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:47:24.477273 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.477227 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae839647-c999-4479-9174-3e4c389d24b8-istiod-ca-cert\") pod \"e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td\" (UID: \"ae839647-c999-4479-9174-3e4c389d24b8\") " pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:47:24.477456 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.477286 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae839647-c999-4479-9174-3e4c389d24b8-istio-data\") pod \"e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td\" (UID: \"ae839647-c999-4479-9174-3e4c389d24b8\") " pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:47:24.477456 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.477337 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae839647-c999-4479-9174-3e4c389d24b8-istio-token\") pod \"e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td\" (UID: \"ae839647-c999-4479-9174-3e4c389d24b8\") " pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:47:24.477456 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.477446 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae839647-c999-4479-9174-3e4c389d24b8-istio-envoy\") pod \"e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td\" (UID: \"ae839647-c999-4479-9174-3e4c389d24b8\") " pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:47:24.477613 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.477555 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae839647-c999-4479-9174-3e4c389d24b8-workload-certs\") pod \"e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td\" (UID: \"ae839647-c999-4479-9174-3e4c389d24b8\") " pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:47:24.477613 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.477587 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae839647-c999-4479-9174-3e4c389d24b8-workload-socket\") pod \"e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td\" (UID: \"ae839647-c999-4479-9174-3e4c389d24b8\") " pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:47:24.480238 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.478106 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae839647-c999-4479-9174-3e4c389d24b8-workload-socket\") pod \"e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td\" (UID: \"ae839647-c999-4479-9174-3e4c389d24b8\") " pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:47:24.480238 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.478356 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae839647-c999-4479-9174-3e4c389d24b8-credential-socket\") pod \"e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td\" (UID: \"ae839647-c999-4479-9174-3e4c389d24b8\") " pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:47:24.480238 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.479566 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae839647-c999-4479-9174-3e4c389d24b8-istiod-ca-cert\") pod \"e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td\" (UID: \"ae839647-c999-4479-9174-3e4c389d24b8\") " pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:47:24.480238 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.480148 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae839647-c999-4479-9174-3e4c389d24b8-istio-data\") pod \"e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td\" (UID: \"ae839647-c999-4479-9174-3e4c389d24b8\") " pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:47:24.480798 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.480730 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae839647-c999-4479-9174-3e4c389d24b8-workload-certs\") pod \"e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td\" (UID: \"ae839647-c999-4479-9174-3e4c389d24b8\") " pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:47:24.482379 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.482356 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae839647-c999-4479-9174-3e4c389d24b8-istio-podinfo\") pod \"e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td\" (UID: \"ae839647-c999-4479-9174-3e4c389d24b8\") " pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:47:24.482718 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.482699 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-9d7aa682-openshift-default-dockercfg-rxljv\"" Nov 03 22:47:24.544060 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.519807 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae839647-c999-4479-9174-3e4c389d24b8-istio-envoy\") pod \"e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td\" (UID: \"ae839647-c999-4479-9174-3e4c389d24b8\") " pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:47:24.547916 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.547214 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzrn9\" (UniqueName: \"kubernetes.io/projected/ae839647-c999-4479-9174-3e4c389d24b8-kube-api-access-kzrn9\") pod \"e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td\" (UID: \"ae839647-c999-4479-9174-3e4c389d24b8\") " pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:47:24.584887 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.572512 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae839647-c999-4479-9174-3e4c389d24b8-istio-token\") pod \"e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td\" (UID: \"ae839647-c999-4479-9174-3e4c389d24b8\") " pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:47:24.658929 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.650138 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:47:24.757888 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.757305 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:24.916827 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:24.916227 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:25.172529 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:25.172456 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td"] Nov 03 22:47:25.230071 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:47:25.227158 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae839647_c999_4479_9174_3e4c389d24b8.slice/crio-d646e0fbf8c57ce5564e89d5d481f322bd6d9d31a9d7c0eb9f1eb2a0b798e2ff WatchSource:0}: Error finding container d646e0fbf8c57ce5564e89d5d481f322bd6d9d31a9d7c0eb9f1eb2a0b798e2ff: Status 404 returned error can't find the container with id d646e0fbf8c57ce5564e89d5d481f322bd6d9d31a9d7c0eb9f1eb2a0b798e2ff Nov 03 22:47:25.231881 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:25.231806 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:47:25.232130 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:25.232106 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:47:25.232292 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:25.232273 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:47:25.409933 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:25.396712 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:25.409933 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:25.399125 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" event={"ID":"ae839647-c999-4479-9174-3e4c389d24b8","Type":"ContainerStarted","Data":"d646e0fbf8c57ce5564e89d5d481f322bd6d9d31a9d7c0eb9f1eb2a0b798e2ff"} Nov 03 22:47:25.441932 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:25.433517 5196 patch_prober.go:29] interesting pod/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:47:25.441932 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:25.433575 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" podUID="4be26434-d0d6-43e2-a89f-39d6533660d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:47:25.564885 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:25.564332 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:26.127456 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:47:26.127415 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:47:26.612092 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:26.610932 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" event={"ID":"ae839647-c999-4479-9174-3e4c389d24b8","Type":"ContainerStarted","Data":"72554492ecab9434f7b00cc6dfdc65372965387acd8c498e27dc31ff42e14c3a"} Nov 03 22:47:26.653896 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:26.653432 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:47:26.653896 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:26.653552 5196 patch_prober.go:29] interesting pod/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 22:47:26.653896 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:26.653608 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" podUID="ae839647-c999-4479-9174-3e4c389d24b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 22:47:26.885732 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:26.885614 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" podStartSLOduration=2.885594433 podStartE2EDuration="2.885594433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:47:26.793725131 +0000 UTC m=+1476.103284455" watchObservedRunningTime="2026-11-03 22:47:26.885594433 +0000 UTC m=+1476.195153730" Nov 03 22:47:27.004993 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:27.004957 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-nqmgk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:47:27.005263 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:27.005233 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" podUID="aa3310c5-bd35-4bef-b838-1722c7143cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:47:27.683885 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:27.683141 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:47:27.932002 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:27.929972 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" podUID="26fa51c2-0f2e-482f-9c67-0ea7924d7b3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:47:28.322010 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:28.321961 5196 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" podUID="82749c55-0fca-4a14-b6ba-7df122a6a407" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:47:28.532940 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:28.532228 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" podUID="abb461c4-9707-4b5c-aba0-793f7bb993e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:47:28.760315 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:28.760276 5196 generic.go:356] "Generic (PLEG): container finished" podID="4be26434-d0d6-43e2-a89f-39d6533660d8" containerID="ee56c17d9c5f4e64993c0113c5269e222fc2f171fcd5e49625c279abc2980143" exitCode=0 Nov 03 22:47:28.762259 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:28.762233 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" event={"ID":"4be26434-d0d6-43e2-a89f-39d6533660d8","Type":"ContainerDied","Data":"ee56c17d9c5f4e64993c0113c5269e222fc2f171fcd5e49625c279abc2980143"} Nov 03 22:47:28.762498 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:28.762484 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:47:28.772261 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:28.772234 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:47:29.046054 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:29.045964 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:29.053229 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:29.053193 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" Nov 03 22:47:29.073981 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:29.073945 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4be26434-d0d6-43e2-a89f-39d6533660d8-istio-token\" (UniqueName: \"kubernetes.io/projected/4be26434-d0d6-43e2-a89f-39d6533660d8-istio-token\") pod \"4be26434-d0d6-43e2-a89f-39d6533660d8\" (UID: \"4be26434-d0d6-43e2-a89f-39d6533660d8\") " Nov 03 22:47:29.074160 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:29.074000 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be26434-d0d6-43e2-a89f-39d6533660d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be26434-d0d6-43e2-a89f-39d6533660d8-workload-socket\") pod \"4be26434-d0d6-43e2-a89f-39d6533660d8\" (UID: \"4be26434-d0d6-43e2-a89f-39d6533660d8\") " Nov 03 22:47:29.074160 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:29.074036 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4be26434-d0d6-43e2-a89f-39d6533660d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be26434-d0d6-43e2-a89f-39d6533660d8-istiod-ca-cert\") pod \"4be26434-d0d6-43e2-a89f-39d6533660d8\" (UID: \"4be26434-d0d6-43e2-a89f-39d6533660d8\") " Nov 03 22:47:29.074160 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:29.074093 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4be26434-d0d6-43e2-a89f-39d6533660d8-kube-api-access-668m5\" (UniqueName: \"kubernetes.io/projected/4be26434-d0d6-43e2-a89f-39d6533660d8-kube-api-access-668m5\") pod \"4be26434-d0d6-43e2-a89f-39d6533660d8\" (UID: \"4be26434-d0d6-43e2-a89f-39d6533660d8\") " Nov 03 22:47:29.074160 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:29.074137 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be26434-d0d6-43e2-a89f-39d6533660d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be26434-d0d6-43e2-a89f-39d6533660d8-credential-socket\") pod \"4be26434-d0d6-43e2-a89f-39d6533660d8\" (UID: \"4be26434-d0d6-43e2-a89f-39d6533660d8\") " Nov 03 22:47:29.074907 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:29.074278 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be26434-d0d6-43e2-a89f-39d6533660d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be26434-d0d6-43e2-a89f-39d6533660d8-istio-envoy\") pod \"4be26434-d0d6-43e2-a89f-39d6533660d8\" (UID: \"4be26434-d0d6-43e2-a89f-39d6533660d8\") " Nov 03 22:47:29.074907 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:29.074360 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be26434-d0d6-43e2-a89f-39d6533660d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be26434-d0d6-43e2-a89f-39d6533660d8-istio-data\") pod \"4be26434-d0d6-43e2-a89f-39d6533660d8\" (UID: \"4be26434-d0d6-43e2-a89f-39d6533660d8\") " Nov 03 22:47:29.074907 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:29.074398 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be26434-d0d6-43e2-a89f-39d6533660d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be26434-d0d6-43e2-a89f-39d6533660d8-workload-certs\") pod \"4be26434-d0d6-43e2-a89f-39d6533660d8\" (UID: \"4be26434-d0d6-43e2-a89f-39d6533660d8\") " Nov 03 22:47:29.074907 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:29.074451 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4be26434-d0d6-43e2-a89f-39d6533660d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be26434-d0d6-43e2-a89f-39d6533660d8-istio-podinfo\") pod \"4be26434-d0d6-43e2-a89f-39d6533660d8\" (UID: \"4be26434-d0d6-43e2-a89f-39d6533660d8\") " Nov 03 22:47:29.076279 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:29.075663 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4be26434-d0d6-43e2-a89f-39d6533660d8-istiod-ca-cert" pod "4be26434-d0d6-43e2-a89f-39d6533660d8" (UID: "4be26434-d0d6-43e2-a89f-39d6533660d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:29.076279 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:29.076027 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be26434-d0d6-43e2-a89f-39d6533660d8-workload-socket" pod "4be26434-d0d6-43e2-a89f-39d6533660d8" (UID: "4be26434-d0d6-43e2-a89f-39d6533660d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:29.076771 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:29.076747 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be26434-d0d6-43e2-a89f-39d6533660d8-istio-data" pod "4be26434-d0d6-43e2-a89f-39d6533660d8" (UID: "4be26434-d0d6-43e2-a89f-39d6533660d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:29.077104 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:29.077085 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be26434-d0d6-43e2-a89f-39d6533660d8-credential-socket" pod "4be26434-d0d6-43e2-a89f-39d6533660d8" (UID: "4be26434-d0d6-43e2-a89f-39d6533660d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:29.077391 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:29.077372 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be26434-d0d6-43e2-a89f-39d6533660d8-workload-certs" pod "4be26434-d0d6-43e2-a89f-39d6533660d8" (UID: "4be26434-d0d6-43e2-a89f-39d6533660d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:29.078553 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:29.078524 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4be26434-d0d6-43e2-a89f-39d6533660d8-istio-podinfo" pod "4be26434-d0d6-43e2-a89f-39d6533660d8" (UID: "4be26434-d0d6-43e2-a89f-39d6533660d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:47:29.080500 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:29.080466 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4be26434-d0d6-43e2-a89f-39d6533660d8-istio-token" pod "4be26434-d0d6-43e2-a89f-39d6533660d8" (UID: "4be26434-d0d6-43e2-a89f-39d6533660d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:29.081033 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:29.081007 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be26434-d0d6-43e2-a89f-39d6533660d8-istio-envoy" pod "4be26434-d0d6-43e2-a89f-39d6533660d8" (UID: "4be26434-d0d6-43e2-a89f-39d6533660d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:29.082691 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:29.082658 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4be26434-d0d6-43e2-a89f-39d6533660d8-kube-api-access-668m5" pod "4be26434-d0d6-43e2-a89f-39d6533660d8" (UID: "4be26434-d0d6-43e2-a89f-39d6533660d8"). InnerVolumeSpecName "kube-api-access-668m5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:29.177128 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:29.177091 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4be26434-d0d6-43e2-a89f-39d6533660d8-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:47:29.177128 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:29.177128 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be26434-d0d6-43e2-a89f-39d6533660d8-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:47:29.177486 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:29.177144 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be26434-d0d6-43e2-a89f-39d6533660d8-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:47:29.177486 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:29.177158 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-668m5\" (UniqueName: \"kubernetes.io/projected/4be26434-d0d6-43e2-a89f-39d6533660d8-kube-api-access-668m5\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:47:29.177486 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:29.177202 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be26434-d0d6-43e2-a89f-39d6533660d8-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:47:29.177486 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:29.177246 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be26434-d0d6-43e2-a89f-39d6533660d8-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:47:29.177486 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:29.177282 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be26434-d0d6-43e2-a89f-39d6533660d8-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:47:29.177486 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:29.177299 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be26434-d0d6-43e2-a89f-39d6533660d8-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:47:29.177486 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:29.177313 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be26434-d0d6-43e2-a89f-39d6533660d8-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:47:29.277896 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:29.277840 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:29.446980 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:29.446639 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:29.771821 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:29.771732 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" Nov 03 22:47:29.771821 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:29.771763 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd" event={"ID":"4be26434-d0d6-43e2-a89f-39d6533660d8","Type":"ContainerDied","Data":"827adc0b738f1ae16019aadb448f3c106cd6f1e3dad62240a02ca3d286c43f70"} Nov 03 22:47:29.772386 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:29.771908 5196 scope.go:122] "RemoveContainer" containerID="ee56c17d9c5f4e64993c0113c5269e222fc2f171fcd5e49625c279abc2980143" Nov 03 22:47:29.989114 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:29.989084 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd"] Nov 03 22:47:30.040209 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:30.040133 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-fc129ccf-gw-openshift-default-5cd657dd45-4hrkd"] Nov 03 22:47:30.816541 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:30.816507 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4be26434-d0d6-43e2-a89f-39d6533660d8" path="/var/lib/kubelet/pods/4be26434-d0d6-43e2-a89f-39d6533660d8/volumes" Nov 03 22:47:32.291182 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:32.291136 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:32.465917 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:32.463939 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:34.508784 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:34.508753 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:39.519559 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:39.519525 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:44.288308 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:44.288270 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:47.984651 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:47.984615 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:48.539881 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:48.539664 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn"] Nov 03 22:47:48.545913 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:48.540702 5196 cpu_manager.go:414] "containerMap: removing container" podUID="4be26434-d0d6-43e2-a89f-39d6533660d8" containerName="istio-proxy" Nov 03 22:47:48.545913 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:48.540727 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4be26434-d0d6-43e2-a89f-39d6533660d8" containerName="istio-proxy" Nov 03 22:47:48.545913 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:48.544637 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="4be26434-d0d6-43e2-a89f-39d6533660d8" containerName="istio-proxy" Nov 03 22:47:48.562878 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:48.560149 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn" Nov 03 22:47:48.597890 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:48.596900 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58d33b25-27e9-4d87-bb8c-78033c13eff3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vsltn\" (UID: \"58d33b25-27e9-4d87-bb8c-78033c13eff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn" Nov 03 22:47:48.597890 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:48.596949 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/58d33b25-27e9-4d87-bb8c-78033c13eff3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vsltn\" (UID: \"58d33b25-27e9-4d87-bb8c-78033c13eff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn" Nov 03 22:47:48.597890 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:48.597058 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrqhb\" (UniqueName: \"kubernetes.io/projected/58d33b25-27e9-4d87-bb8c-78033c13eff3-kube-api-access-lrqhb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vsltn\" (UID: \"58d33b25-27e9-4d87-bb8c-78033c13eff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn" Nov 03 22:47:48.597890 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:48.597104 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/58d33b25-27e9-4d87-bb8c-78033c13eff3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vsltn\" (UID: \"58d33b25-27e9-4d87-bb8c-78033c13eff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn" Nov 03 22:47:48.597890 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:48.597265 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/58d33b25-27e9-4d87-bb8c-78033c13eff3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vsltn\" (UID: \"58d33b25-27e9-4d87-bb8c-78033c13eff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn" Nov 03 22:47:48.597890 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:48.597290 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/58d33b25-27e9-4d87-bb8c-78033c13eff3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vsltn\" (UID: \"58d33b25-27e9-4d87-bb8c-78033c13eff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn" Nov 03 22:47:48.597890 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:48.597324 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d33b25-27e9-4d87-bb8c-78033c13eff3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vsltn\" (UID: \"58d33b25-27e9-4d87-bb8c-78033c13eff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn" Nov 03 22:47:48.618220 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:48.618178 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 22:47:48.618404 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:48.618384 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-69szt\"" Nov 03 22:47:48.618452 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:48.618440 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 22:47:48.619171 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:48.619155 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 22:47:48.695409 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:48.695355 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn"] Nov 03 22:47:48.700012 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:48.698894 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58d33b25-27e9-4d87-bb8c-78033c13eff3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vsltn\" (UID: \"58d33b25-27e9-4d87-bb8c-78033c13eff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn" Nov 03 22:47:48.702167 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:48.702141 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/58d33b25-27e9-4d87-bb8c-78033c13eff3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vsltn\" (UID: \"58d33b25-27e9-4d87-bb8c-78033c13eff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn" Nov 03 22:47:48.702448 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:48.702428 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrqhb\" (UniqueName: \"kubernetes.io/projected/58d33b25-27e9-4d87-bb8c-78033c13eff3-kube-api-access-lrqhb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vsltn\" (UID: \"58d33b25-27e9-4d87-bb8c-78033c13eff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn" Nov 03 22:47:48.702601 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:48.702585 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/58d33b25-27e9-4d87-bb8c-78033c13eff3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vsltn\" (UID: \"58d33b25-27e9-4d87-bb8c-78033c13eff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn" Nov 03 22:47:48.702782 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:48.702765 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/58d33b25-27e9-4d87-bb8c-78033c13eff3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vsltn\" (UID: \"58d33b25-27e9-4d87-bb8c-78033c13eff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn" Nov 03 22:47:48.702934 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:48.702920 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/58d33b25-27e9-4d87-bb8c-78033c13eff3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vsltn\" (UID: \"58d33b25-27e9-4d87-bb8c-78033c13eff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn" Nov 03 22:47:48.703058 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:48.703045 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d33b25-27e9-4d87-bb8c-78033c13eff3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vsltn\" (UID: \"58d33b25-27e9-4d87-bb8c-78033c13eff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn" Nov 03 22:47:48.711452 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:48.711366 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:48.729925 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:48.721714 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/58d33b25-27e9-4d87-bb8c-78033c13eff3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vsltn\" (UID: \"58d33b25-27e9-4d87-bb8c-78033c13eff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn" Nov 03 22:47:48.782813 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:48.781837 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d33b25-27e9-4d87-bb8c-78033c13eff3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vsltn\" (UID: \"58d33b25-27e9-4d87-bb8c-78033c13eff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn" Nov 03 22:47:48.782813 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:48.782228 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/58d33b25-27e9-4d87-bb8c-78033c13eff3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vsltn\" (UID: \"58d33b25-27e9-4d87-bb8c-78033c13eff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn" Nov 03 22:47:48.801433 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:48.801362 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/58d33b25-27e9-4d87-bb8c-78033c13eff3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vsltn\" (UID: \"58d33b25-27e9-4d87-bb8c-78033c13eff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn" Nov 03 22:47:48.802544 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:48.802499 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/58d33b25-27e9-4d87-bb8c-78033c13eff3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vsltn\" (UID: \"58d33b25-27e9-4d87-bb8c-78033c13eff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn" Nov 03 22:47:48.853784 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:48.853712 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58d33b25-27e9-4d87-bb8c-78033c13eff3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vsltn\" (UID: \"58d33b25-27e9-4d87-bb8c-78033c13eff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn" Nov 03 22:47:48.855678 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:48.855647 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrqhb\" (UniqueName: \"kubernetes.io/projected/58d33b25-27e9-4d87-bb8c-78033c13eff3-kube-api-access-lrqhb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vsltn\" (UID: \"58d33b25-27e9-4d87-bb8c-78033c13eff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn" Nov 03 22:47:48.885781 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:48.885475 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn" Nov 03 22:47:49.047287 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:49.047254 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:49.201480 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:49.201441 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:49.477318 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:49.477283 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn"] Nov 03 22:47:49.512112 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:47:49.512075 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58d33b25_27e9_4d87_bb8c_78033c13eff3.slice/crio-d27f0a20fd95f04acb67aac1a8379f6432566557cd60c09cd44b163c7df287bb WatchSource:0}: Error finding container d27f0a20fd95f04acb67aac1a8379f6432566557cd60c09cd44b163c7df287bb: Status 404 returned error can't find the container with id d27f0a20fd95f04acb67aac1a8379f6432566557cd60c09cd44b163c7df287bb Nov 03 22:47:49.999418 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:49.999375 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn" event={"ID":"58d33b25-27e9-4d87-bb8c-78033c13eff3","Type":"ContainerStarted","Data":"d27f0a20fd95f04acb67aac1a8379f6432566557cd60c09cd44b163c7df287bb"} Nov 03 22:47:53.341052 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:53.341012 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:53.539577 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:53.539534 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:53.691056 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:53.690976 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:54.363877 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:54.363375 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:55.084486 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:55.084449 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:47:55.084642 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:55.084522 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:47:55.921222 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:47:55.921184 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:47:56.049200 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:56.049167 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn" event={"ID":"58d33b25-27e9-4d87-bb8c-78033c13eff3","Type":"ContainerStarted","Data":"30ef2c70c1bb1536d1f2979e06f34c1837bc539030811b4daf84af28b2057818"} Nov 03 22:47:56.049378 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:56.049232 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn" Nov 03 22:47:56.050940 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:56.050916 5196 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-vsltn container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:47:56.051069 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:56.050977 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn" podUID="58d33b25-27e9-4d87-bb8c-78033c13eff3" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:47:56.239941 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:56.236951 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn" podStartSLOduration=2.668899367 podStartE2EDuration="8.236930356s" totalImagesPullingTime="5.568030989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:47:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:47:56.183447115 +0000 UTC m=+1505.493006409" watchObservedRunningTime="2026-11-03 22:47:56.236930356 +0000 UTC m=+1505.546489649" Nov 03 22:47:57.057383 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:57.057355 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn" Nov 03 22:47:57.426758 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:57.426728 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:57.626130 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:57.626096 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:47:59.743373 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:47:59.743340 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:04.755624 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:04.755575 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:04.952994 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:04.952962 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:08.814639 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:08.814607 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:08.960342 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:08.960308 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:09.710410 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:09.710383 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:09.876420 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:09.876393 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:10.025653 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:10.025549 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:10.251415 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:10.251380 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:10.943218 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:10.943190 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:11.283887 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:11.283792 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:11.909289 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:11.909254 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:14.853482 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:14.853436 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:19.465751 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:19.465720 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:19.630279 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:19.630250 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:19.781407 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:19.781296 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:24.404248 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:24.404203 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:25.919921 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:48:25.919844 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:48:29.169984 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:29.169922 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:30.339711 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:30.339676 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:31.178262 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:31.178230 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:31.380917 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:31.378351 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:33.956267 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:33.956228 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:35.643551 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:35.643519 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:35.811135 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:35.811091 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:35.971529 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:35.971431 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:38.280484 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:38.280451 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1cb89500-56959c5b65-nx997"] Nov 03 22:48:38.280898 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:38.280689 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cb89500-56959c5b65-nx997" podUID="ecc35923-4d08-43be-91e7-6e905385fc13" containerName="maas-api" containerID="cri-o://c540476fb78413c110fe33d282e0a57d12d4fd6da39a6477f340568c402c998f" gracePeriod=30 Nov 03 22:48:38.316602 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:38.316575 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:38.467064 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:38.466891 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:38.683467 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:38.683441 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cb89500-56959c5b65-nx997" Nov 03 22:48:38.794520 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:38.794482 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecc35923-4d08-43be-91e7-6e905385fc13-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecc35923-4d08-43be-91e7-6e905385fc13-metrics-certs\") pod \"ecc35923-4d08-43be-91e7-6e905385fc13\" (UID: \"ecc35923-4d08-43be-91e7-6e905385fc13\") " Nov 03 22:48:38.799027 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:38.798990 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecc35923-4d08-43be-91e7-6e905385fc13-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecc35923-4d08-43be-91e7-6e905385fc13-openshift-service-ca\") pod \"ecc35923-4d08-43be-91e7-6e905385fc13\" (UID: \"ecc35923-4d08-43be-91e7-6e905385fc13\") " Nov 03 22:48:38.806714 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:38.806682 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecc35923-4d08-43be-91e7-6e905385fc13-kube-api-access-xghb4\" (UniqueName: \"kubernetes.io/projected/ecc35923-4d08-43be-91e7-6e905385fc13-kube-api-access-xghb4\") pod \"ecc35923-4d08-43be-91e7-6e905385fc13\" (UID: \"ecc35923-4d08-43be-91e7-6e905385fc13\") " Nov 03 22:48:38.807068 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:38.807047 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecc35923-4d08-43be-91e7-6e905385fc13-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecc35923-4d08-43be-91e7-6e905385fc13-maas-api-tls\") pod \"ecc35923-4d08-43be-91e7-6e905385fc13\" (UID: \"ecc35923-4d08-43be-91e7-6e905385fc13\") " Nov 03 22:48:38.816914 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:38.799691 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecc35923-4d08-43be-91e7-6e905385fc13-openshift-service-ca" pod "ecc35923-4d08-43be-91e7-6e905385fc13" (UID: "ecc35923-4d08-43be-91e7-6e905385fc13"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:48:38.849953 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:38.849572 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc35923-4d08-43be-91e7-6e905385fc13-kube-api-access-xghb4" pod "ecc35923-4d08-43be-91e7-6e905385fc13" (UID: "ecc35923-4d08-43be-91e7-6e905385fc13"). InnerVolumeSpecName "kube-api-access-xghb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:48:38.852407 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:38.852350 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecc35923-4d08-43be-91e7-6e905385fc13-metrics-certs" pod "ecc35923-4d08-43be-91e7-6e905385fc13" (UID: "ecc35923-4d08-43be-91e7-6e905385fc13"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:48:38.856284 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:38.856256 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecc35923-4d08-43be-91e7-6e905385fc13-maas-api-tls" pod "ecc35923-4d08-43be-91e7-6e905385fc13" (UID: "ecc35923-4d08-43be-91e7-6e905385fc13"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:48:38.896517 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:38.896248 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:38.909876 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:38.909833 5196 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecc35923-4d08-43be-91e7-6e905385fc13-openshift-service-ca\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:48:38.910086 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:38.910063 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xghb4\" (UniqueName: \"kubernetes.io/projected/ecc35923-4d08-43be-91e7-6e905385fc13-kube-api-access-xghb4\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:48:38.910086 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:38.910085 5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecc35923-4d08-43be-91e7-6e905385fc13-maas-api-tls\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:48:38.910221 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:38.910096 5196 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecc35923-4d08-43be-91e7-6e905385fc13-metrics-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:48:39.060657 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:39.060629 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:39.423052 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:39.423014 5196 generic.go:356] "Generic (PLEG): container finished" podID="ecc35923-4d08-43be-91e7-6e905385fc13" containerID="c540476fb78413c110fe33d282e0a57d12d4fd6da39a6477f340568c402c998f" exitCode=0 Nov 03 22:48:39.423551 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:39.423086 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cb89500-56959c5b65-nx997" Nov 03 22:48:39.423551 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:39.423197 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cb89500-56959c5b65-nx997" event={"ID":"ecc35923-4d08-43be-91e7-6e905385fc13","Type":"ContainerDied","Data":"c540476fb78413c110fe33d282e0a57d12d4fd6da39a6477f340568c402c998f"} Nov 03 22:48:39.423551 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:39.423239 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cb89500-56959c5b65-nx997" event={"ID":"ecc35923-4d08-43be-91e7-6e905385fc13","Type":"ContainerDied","Data":"9d87794c322baefab2610423c3651cc4153b79d2053b945874d90c9d29b0ab7c"} Nov 03 22:48:39.423551 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:39.423261 5196 scope.go:122] "RemoveContainer" containerID="c540476fb78413c110fe33d282e0a57d12d4fd6da39a6477f340568c402c998f" Nov 03 22:48:39.446489 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:39.446463 5196 scope.go:122] "RemoveContainer" containerID="c540476fb78413c110fe33d282e0a57d12d4fd6da39a6477f340568c402c998f" Nov 03 22:48:39.446967 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:48:39.446941 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c540476fb78413c110fe33d282e0a57d12d4fd6da39a6477f340568c402c998f\": container with ID starting with c540476fb78413c110fe33d282e0a57d12d4fd6da39a6477f340568c402c998f not found: ID does not exist" containerID="c540476fb78413c110fe33d282e0a57d12d4fd6da39a6477f340568c402c998f" Nov 03 22:48:39.447082 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:39.446983 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c540476fb78413c110fe33d282e0a57d12d4fd6da39a6477f340568c402c998f"} err="failed to get container status \"c540476fb78413c110fe33d282e0a57d12d4fd6da39a6477f340568c402c998f\": rpc error: code = NotFound desc = could not find container \"c540476fb78413c110fe33d282e0a57d12d4fd6da39a6477f340568c402c998f\": container with ID starting with c540476fb78413c110fe33d282e0a57d12d4fd6da39a6477f340568c402c998f not found: ID does not exist" Nov 03 22:48:39.634711 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:39.634682 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1cb89500-56959c5b65-nx997"] Nov 03 22:48:39.685270 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:39.685177 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1cb89500-56959c5b65-nx997"] Nov 03 22:48:39.874565 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:39.874535 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:40.069423 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:40.069336 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:40.325421 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:40.325350 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:40.526282 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:40.526255 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:40.817479 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:40.817444 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecc35923-4d08-43be-91e7-6e905385fc13" path="/var/lib/kubelet/pods/ecc35923-4d08-43be-91e7-6e905385fc13/volumes" Nov 03 22:48:42.091009 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:42.090970 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:43.007626 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:43.007595 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:43.156434 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:43.156403 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:43.473098 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:43.473063 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:44.947512 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:44.947482 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:45.400881 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:45.399977 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq"] Nov 03 22:48:45.401078 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:45.400919 5196 cpu_manager.go:414] "containerMap: removing container" podUID="ecc35923-4d08-43be-91e7-6e905385fc13" containerName="maas-api" Nov 03 22:48:45.401078 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:45.400936 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecc35923-4d08-43be-91e7-6e905385fc13" containerName="maas-api" Nov 03 22:48:45.401193 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:45.401078 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecc35923-4d08-43be-91e7-6e905385fc13" containerName="maas-api" Nov 03 22:48:45.405171 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:45.405148 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq" Nov 03 22:48:45.458713 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:45.458673 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0801535-901a-46f0-a882-e9634537f0fc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq\" (UID: \"c0801535-901a-46f0-a882-e9634537f0fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq" Nov 03 22:48:45.458899 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:45.458787 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsgwh\" (UniqueName: \"kubernetes.io/projected/c0801535-901a-46f0-a882-e9634537f0fc-kube-api-access-jsgwh\") pod \"maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq\" (UID: \"c0801535-901a-46f0-a882-e9634537f0fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq" Nov 03 22:48:45.459015 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:45.458996 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0801535-901a-46f0-a882-e9634537f0fc-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq\" (UID: \"c0801535-901a-46f0-a882-e9634537f0fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq" Nov 03 22:48:45.459092 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:45.459072 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0801535-901a-46f0-a882-e9634537f0fc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq\" (UID: \"c0801535-901a-46f0-a882-e9634537f0fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq" Nov 03 22:48:45.462680 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:45.462657 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq"] Nov 03 22:48:45.522189 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:45.522159 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-c93e70-serving-cert\"" Nov 03 22:48:45.562995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:45.562961 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0801535-901a-46f0-a882-e9634537f0fc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq\" (UID: \"c0801535-901a-46f0-a882-e9634537f0fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq" Nov 03 22:48:45.564115 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:45.564087 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsgwh\" (UniqueName: \"kubernetes.io/projected/c0801535-901a-46f0-a882-e9634537f0fc-kube-api-access-jsgwh\") pod \"maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq\" (UID: \"c0801535-901a-46f0-a882-e9634537f0fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq" Nov 03 22:48:45.564798 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:45.564774 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0801535-901a-46f0-a882-e9634537f0fc-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq\" (UID: \"c0801535-901a-46f0-a882-e9634537f0fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq" Nov 03 22:48:45.565479 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:45.565460 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0801535-901a-46f0-a882-e9634537f0fc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq\" (UID: \"c0801535-901a-46f0-a882-e9634537f0fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq" Nov 03 22:48:45.567060 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:45.563950 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0801535-901a-46f0-a882-e9634537f0fc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq\" (UID: \"c0801535-901a-46f0-a882-e9634537f0fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq" Nov 03 22:48:45.567060 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:48:45.565666 5196 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c93e70-serving-cert: secret "maas-api-e2e-shared-a-w2-c93e70-serving-cert" not found Nov 03 22:48:45.567060 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:48:45.566790 5196 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0801535-901a-46f0-a882-e9634537f0fc-maas-api-tls podName:c0801535-901a-46f0-a882-e9634537f0fc nodeName:}" failed. No retries permitted until 2026-11-03 22:48:46.066767815 +0000 UTC m=+1555.376327101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c0801535-901a-46f0-a882-e9634537f0fc-maas-api-tls") pod "maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq" (UID: "c0801535-901a-46f0-a882-e9634537f0fc") : secret "maas-api-e2e-shared-a-w2-c93e70-serving-cert" not found Nov 03 22:48:45.568434 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:45.568414 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0801535-901a-46f0-a882-e9634537f0fc-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq\" (UID: \"c0801535-901a-46f0-a882-e9634537f0fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq" Nov 03 22:48:45.614966 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:45.597392 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:45.626444 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:45.626416 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsgwh\" (UniqueName: \"kubernetes.io/projected/c0801535-901a-46f0-a882-e9634537f0fc-kube-api-access-jsgwh\") pod \"maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq\" (UID: \"c0801535-901a-46f0-a882-e9634537f0fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq" Nov 03 22:48:45.753773 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:45.753684 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:45.920882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:45.912501 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:46.077550 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:46.075724 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0801535-901a-46f0-a882-e9634537f0fc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq\" (UID: \"c0801535-901a-46f0-a882-e9634537f0fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq" Nov 03 22:48:46.079106 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:46.079081 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0801535-901a-46f0-a882-e9634537f0fc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq\" (UID: \"c0801535-901a-46f0-a882-e9634537f0fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq" Nov 03 22:48:46.332405 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:46.332317 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq" Nov 03 22:48:46.739940 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:46.739914 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq"] Nov 03 22:48:46.741147 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:48:46.741116 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0801535_901a_46f0_a882_e9634537f0fc.slice/crio-93236eb763cfe2bc9a4cfb43307574d45d143fe0e29598088e5c121180aa93fb WatchSource:0}: Error finding container 93236eb763cfe2bc9a4cfb43307574d45d143fe0e29598088e5c121180aa93fb: Status 404 returned error can't find the container with id 93236eb763cfe2bc9a4cfb43307574d45d143fe0e29598088e5c121180aa93fb Nov 03 22:48:47.489237 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:47.489194 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq" event={"ID":"c0801535-901a-46f0-a882-e9634537f0fc","Type":"ContainerStarted","Data":"93236eb763cfe2bc9a4cfb43307574d45d143fe0e29598088e5c121180aa93fb"} Nov 03 22:48:48.525737 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:48.525703 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq" event={"ID":"c0801535-901a-46f0-a882-e9634537f0fc","Type":"ContainerStarted","Data":"572d15d69c928ab800f55c1991f6d9ad1ab421ceb8dc8b3e742bb41a65dd52ce"} Nov 03 22:48:48.526270 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:48.525799 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq" Nov 03 22:48:48.648046 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:48.647999 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq" podStartSLOduration=2.810215383 podStartE2EDuration="3.647985165s" totalImagesPullingTime="837.769782ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:48:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:48:48.643956022 +0000 UTC m=+1557.953515334" watchObservedRunningTime="2026-11-03 22:48:48.647985165 +0000 UTC m=+1557.957544458" Nov 03 22:48:49.390163 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:49.390127 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:49.537322 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:49.537290 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:50.337526 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:50.337483 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:50.922281 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:50.922241 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:51.146882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:51.146718 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:51.437999 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:51.437947 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:53.692164 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:53.692133 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:54.568014 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:54.567986 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq" Nov 03 22:48:54.577438 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:54.577409 5196 generic.go:356] "Generic (PLEG): container finished" podID="ae839647-c999-4479-9174-3e4c389d24b8" containerID="72554492ecab9434f7b00cc6dfdc65372965387acd8c498e27dc31ff42e14c3a" exitCode=0 Nov 03 22:48:54.577595 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:54.577490 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" event={"ID":"ae839647-c999-4479-9174-3e4c389d24b8","Type":"ContainerDied","Data":"72554492ecab9434f7b00cc6dfdc65372965387acd8c498e27dc31ff42e14c3a"} Nov 03 22:48:54.578039 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:54.578021 5196 scope.go:122] "RemoveContainer" containerID="72554492ecab9434f7b00cc6dfdc65372965387acd8c498e27dc31ff42e14c3a" Nov 03 22:48:54.579902 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:54.579850 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:48:54.579975 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:54.579958 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:48:54.580027 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:54.580011 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:48:54.821262 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:54.821169 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:55.145882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:55.145290 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:55.586222 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:55.586142 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" event={"ID":"ae839647-c999-4479-9174-3e4c389d24b8","Type":"ContainerStarted","Data":"77725e5d2ca56619f7237934fb35a2235bbbeff574359b1c8c63d27aa56dd04c"} Nov 03 22:48:55.650923 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:55.650882 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:48:55.652231 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:55.652208 5196 patch_prober.go:29] interesting pod/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 22:48:55.652510 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:55.652253 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" podUID="ae839647-c999-4479-9174-3e4c389d24b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 22:48:55.947241 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:48:55.947206 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:48:56.129708 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:56.129678 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:56.651771 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:56.651739 5196 patch_prober.go:29] interesting pod/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 22:48:56.652088 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:56.652046 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" podUID="ae839647-c999-4479-9174-3e4c389d24b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 22:48:56.764228 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:56.764197 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:56.965290 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:56.965215 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:57.116272 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:57.116237 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:57.651797 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:57.651768 5196 patch_prober.go:29] interesting pod/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 22:48:57.652038 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:57.651820 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" podUID="ae839647-c999-4479-9174-3e4c389d24b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 22:48:58.651353 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:58.651317 5196 patch_prober.go:29] interesting pod/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 22:48:58.651851 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:58.651365 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" podUID="ae839647-c999-4479-9174-3e4c389d24b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 22:48:58.761974 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:58.761942 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:48:59.155179 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:59.155134 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:59.316285 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:59.316248 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:48:59.650693 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:59.650661 5196 patch_prober.go:29] interesting pod/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 22:48:59.650912 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:48:59.650709 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" podUID="ae839647-c999-4479-9174-3e4c389d24b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 22:49:00.039529 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:00.039438 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:00.217485 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:00.217409 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:00.386117 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:00.386078 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:00.547076 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:00.547044 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:00.664249 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:00.664161 5196 patch_prober.go:29] interesting pod/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 22:49:00.664496 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:00.664466 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" podUID="ae839647-c999-4479-9174-3e4c389d24b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 22:49:01.636287 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:01.636249 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:01.655889 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:01.655837 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:49:01.656795 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:01.656777 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:49:05.056448 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:05.056418 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:05.366667 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:05.366631 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:05.531150 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:05.525385 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:05.683064 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:05.682985 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:06.731679 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:06.731631 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:10.624775 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:10.624741 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:11.601225 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:11.601189 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:16.714845 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:16.714792 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:21.969439 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:21.969401 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:25.919205 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:49:25.919169 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:49:26.845101 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:26.845071 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:27.072256 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:27.072227 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:27.274208 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:27.274109 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:31.971055 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:31.971014 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ebffae22-bf688975f-smqn6"] Nov 03 22:49:31.974366 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:31.974301 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebffae22-bf688975f-smqn6" Nov 03 22:49:32.027695 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:32.027659 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ebffae22-serving-cert\"" Nov 03 22:49:32.032748 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:32.032716 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97debf33-6577-40bc-8819-838d638777cf-openshift-service-ca\") pod \"maas-api-e2e-mt-ebffae22-bf688975f-smqn6\" (UID: \"97debf33-6577-40bc-8819-838d638777cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebffae22-bf688975f-smqn6" Nov 03 22:49:32.032938 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:32.032917 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97debf33-6577-40bc-8819-838d638777cf-metrics-certs\") pod \"maas-api-e2e-mt-ebffae22-bf688975f-smqn6\" (UID: \"97debf33-6577-40bc-8819-838d638777cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebffae22-bf688975f-smqn6" Nov 03 22:49:32.033043 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:32.033030 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97debf33-6577-40bc-8819-838d638777cf-maas-api-tls\") pod \"maas-api-e2e-mt-ebffae22-bf688975f-smqn6\" (UID: \"97debf33-6577-40bc-8819-838d638777cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebffae22-bf688975f-smqn6" Nov 03 22:49:32.033102 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:32.033053 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxs4v\" (UniqueName: \"kubernetes.io/projected/97debf33-6577-40bc-8819-838d638777cf-kube-api-access-mxs4v\") pod \"maas-api-e2e-mt-ebffae22-bf688975f-smqn6\" (UID: \"97debf33-6577-40bc-8819-838d638777cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebffae22-bf688975f-smqn6" Nov 03 22:49:32.083029 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:32.082996 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ebffae22-bf688975f-smqn6"] Nov 03 22:49:32.134597 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:32.134559 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97debf33-6577-40bc-8819-838d638777cf-openshift-service-ca\") pod \"maas-api-e2e-mt-ebffae22-bf688975f-smqn6\" (UID: \"97debf33-6577-40bc-8819-838d638777cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebffae22-bf688975f-smqn6" Nov 03 22:49:32.134775 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:32.134756 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97debf33-6577-40bc-8819-838d638777cf-metrics-certs\") pod \"maas-api-e2e-mt-ebffae22-bf688975f-smqn6\" (UID: \"97debf33-6577-40bc-8819-838d638777cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebffae22-bf688975f-smqn6" Nov 03 22:49:32.134948 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:32.134927 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97debf33-6577-40bc-8819-838d638777cf-maas-api-tls\") pod \"maas-api-e2e-mt-ebffae22-bf688975f-smqn6\" (UID: \"97debf33-6577-40bc-8819-838d638777cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebffae22-bf688975f-smqn6" Nov 03 22:49:32.135053 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:32.134962 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxs4v\" (UniqueName: \"kubernetes.io/projected/97debf33-6577-40bc-8819-838d638777cf-kube-api-access-mxs4v\") pod \"maas-api-e2e-mt-ebffae22-bf688975f-smqn6\" (UID: \"97debf33-6577-40bc-8819-838d638777cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebffae22-bf688975f-smqn6" Nov 03 22:49:32.135426 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:49:32.135202 5196 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-ebffae22-serving-cert: secret "maas-api-e2e-mt-ebffae22-serving-cert" not found Nov 03 22:49:32.135426 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:49:32.135287 5196 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97debf33-6577-40bc-8819-838d638777cf-maas-api-tls podName:97debf33-6577-40bc-8819-838d638777cf nodeName:}" failed. No retries permitted until 2026-11-03 22:49:32.635264009 +0000 UTC m=+1601.944823286 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/97debf33-6577-40bc-8819-838d638777cf-maas-api-tls") pod "maas-api-e2e-mt-ebffae22-bf688975f-smqn6" (UID: "97debf33-6577-40bc-8819-838d638777cf") : secret "maas-api-e2e-mt-ebffae22-serving-cert" not found Nov 03 22:49:32.135772 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:32.135418 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97debf33-6577-40bc-8819-838d638777cf-openshift-service-ca\") pod \"maas-api-e2e-mt-ebffae22-bf688975f-smqn6\" (UID: \"97debf33-6577-40bc-8819-838d638777cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebffae22-bf688975f-smqn6" Nov 03 22:49:32.137592 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:32.137568 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97debf33-6577-40bc-8819-838d638777cf-metrics-certs\") pod \"maas-api-e2e-mt-ebffae22-bf688975f-smqn6\" (UID: \"97debf33-6577-40bc-8819-838d638777cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebffae22-bf688975f-smqn6" Nov 03 22:49:32.164373 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:32.164346 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:32.197015 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:32.196997 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxs4v\" (UniqueName: \"kubernetes.io/projected/97debf33-6577-40bc-8819-838d638777cf-kube-api-access-mxs4v\") pod \"maas-api-e2e-mt-ebffae22-bf688975f-smqn6\" (UID: \"97debf33-6577-40bc-8819-838d638777cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebffae22-bf688975f-smqn6" Nov 03 22:49:32.327969 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:32.327835 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:32.526490 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:32.526456 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:32.642899 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:32.642848 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97debf33-6577-40bc-8819-838d638777cf-maas-api-tls\") pod \"maas-api-e2e-mt-ebffae22-bf688975f-smqn6\" (UID: \"97debf33-6577-40bc-8819-838d638777cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebffae22-bf688975f-smqn6" Nov 03 22:49:32.645445 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:32.645419 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97debf33-6577-40bc-8819-838d638777cf-maas-api-tls\") pod \"maas-api-e2e-mt-ebffae22-bf688975f-smqn6\" (UID: \"97debf33-6577-40bc-8819-838d638777cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebffae22-bf688975f-smqn6" Nov 03 22:49:32.895176 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:32.895092 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebffae22-bf688975f-smqn6" Nov 03 22:49:33.275174 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:33.275145 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ebffae22-bf688975f-smqn6"] Nov 03 22:49:33.276253 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:49:33.276227 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97debf33_6577_40bc_8819_838d638777cf.slice/crio-3a99714e94788118a6d917ed31c07ababad6f7983ef7f49bd491348ea7cbf8f9 WatchSource:0}: Error finding container 3a99714e94788118a6d917ed31c07ababad6f7983ef7f49bd491348ea7cbf8f9: Status 404 returned error can't find the container with id 3a99714e94788118a6d917ed31c07ababad6f7983ef7f49bd491348ea7cbf8f9 Nov 03 22:49:33.278142 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:33.278124 5196 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:49:33.976636 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:33.976599 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebffae22-bf688975f-smqn6" event={"ID":"97debf33-6577-40bc-8819-838d638777cf","Type":"ContainerStarted","Data":"3a99714e94788118a6d917ed31c07ababad6f7983ef7f49bd491348ea7cbf8f9"} Nov 03 22:49:34.997882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:34.996342 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebffae22-bf688975f-smqn6" event={"ID":"97debf33-6577-40bc-8819-838d638777cf","Type":"ContainerStarted","Data":"d2f4a4a95a552f058f63615190b577bce24ff9a95328c166478a6e1f5a737bcd"} Nov 03 22:49:35.000394 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:34.998986 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebffae22-bf688975f-smqn6" Nov 03 22:49:35.121883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:35.120053 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebffae22-bf688975f-smqn6" podStartSLOduration=3.261884431 podStartE2EDuration="4.120032458s" totalImagesPullingTime="858.148027ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:49:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:49:35.118595404 +0000 UTC m=+1604.428154697" watchObservedRunningTime="2026-11-03 22:49:35.120032458 +0000 UTC m=+1604.429591752" Nov 03 22:49:37.112601 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:37.112568 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:38.095885 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:38.095823 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:38.244060 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:38.243999 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:41.009068 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:41.009040 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebffae22-bf688975f-smqn6" Nov 03 22:49:41.906167 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:41.906116 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:46.706684 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:46.706656 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:51.614959 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:51.614919 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:55.920784 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:49:55.920738 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:49:55.981184 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:55.981152 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:56.431473 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:56.431439 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:56.581732 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:56.581698 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:56.732271 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:56.732164 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:56.885701 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:56.885658 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:57.556684 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:57.556628 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:49:58.077325 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:49:58.077277 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:01.554193 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:01.554095 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:02.128781 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:02.128746 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:02.278774 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:02.278744 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:05.466174 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:05.466138 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:06.589539 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:06.589503 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl"] Nov 03 22:50:06.592944 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:06.592926 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl" Nov 03 22:50:06.651887 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:06.651809 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-eb3c33-serving-cert\"" Nov 03 22:50:06.758786 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:06.758725 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff5d39d8-8b0c-4dc2-ab16-7626474d04de-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl\" (UID: \"ff5d39d8-8b0c-4dc2-ab16-7626474d04de\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl" Nov 03 22:50:06.759164 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:06.759138 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff5d39d8-8b0c-4dc2-ab16-7626474d04de-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl\" (UID: \"ff5d39d8-8b0c-4dc2-ab16-7626474d04de\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl" Nov 03 22:50:06.759645 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:06.759623 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gfxr\" (UniqueName: \"kubernetes.io/projected/ff5d39d8-8b0c-4dc2-ab16-7626474d04de-kube-api-access-4gfxr\") pod \"maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl\" (UID: \"ff5d39d8-8b0c-4dc2-ab16-7626474d04de\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl" Nov 03 22:50:06.759767 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:06.759752 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff5d39d8-8b0c-4dc2-ab16-7626474d04de-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl\" (UID: \"ff5d39d8-8b0c-4dc2-ab16-7626474d04de\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl" Nov 03 22:50:06.784185 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:06.784147 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl"] Nov 03 22:50:06.834375 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:06.834350 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:06.861998 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:06.861907 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gfxr\" (UniqueName: \"kubernetes.io/projected/ff5d39d8-8b0c-4dc2-ab16-7626474d04de-kube-api-access-4gfxr\") pod \"maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl\" (UID: \"ff5d39d8-8b0c-4dc2-ab16-7626474d04de\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl" Nov 03 22:50:06.861998 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:06.861947 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff5d39d8-8b0c-4dc2-ab16-7626474d04de-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl\" (UID: \"ff5d39d8-8b0c-4dc2-ab16-7626474d04de\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl" Nov 03 22:50:06.862230 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:06.862059 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff5d39d8-8b0c-4dc2-ab16-7626474d04de-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl\" (UID: \"ff5d39d8-8b0c-4dc2-ab16-7626474d04de\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl" Nov 03 22:50:06.862230 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:06.862082 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff5d39d8-8b0c-4dc2-ab16-7626474d04de-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl\" (UID: \"ff5d39d8-8b0c-4dc2-ab16-7626474d04de\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl" Nov 03 22:50:06.862617 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:50:06.862573 5196 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eb3c33-serving-cert: secret "maas-api-e2e-shared-a-w3-eb3c33-serving-cert" not found Nov 03 22:50:06.862617 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:50:06.862638 5196 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff5d39d8-8b0c-4dc2-ab16-7626474d04de-maas-api-tls podName:ff5d39d8-8b0c-4dc2-ab16-7626474d04de nodeName:}" failed. No retries permitted until 2026-11-03 22:50:07.362621354 +0000 UTC m=+1636.672180628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ff5d39d8-8b0c-4dc2-ab16-7626474d04de-maas-api-tls") pod "maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl" (UID: "ff5d39d8-8b0c-4dc2-ab16-7626474d04de") : secret "maas-api-e2e-shared-a-w3-eb3c33-serving-cert" not found Nov 03 22:50:06.863329 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:06.863302 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff5d39d8-8b0c-4dc2-ab16-7626474d04de-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl\" (UID: \"ff5d39d8-8b0c-4dc2-ab16-7626474d04de\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl" Nov 03 22:50:06.866950 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:06.866925 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff5d39d8-8b0c-4dc2-ab16-7626474d04de-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl\" (UID: \"ff5d39d8-8b0c-4dc2-ab16-7626474d04de\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl" Nov 03 22:50:06.927130 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:06.927103 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gfxr\" (UniqueName: \"kubernetes.io/projected/ff5d39d8-8b0c-4dc2-ab16-7626474d04de-kube-api-access-4gfxr\") pod \"maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl\" (UID: \"ff5d39d8-8b0c-4dc2-ab16-7626474d04de\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl" Nov 03 22:50:07.026739 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:07.026704 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:07.219511 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:07.219427 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:07.370139 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:07.370107 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff5d39d8-8b0c-4dc2-ab16-7626474d04de-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl\" (UID: \"ff5d39d8-8b0c-4dc2-ab16-7626474d04de\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl" Nov 03 22:50:07.373377 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:07.373341 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff5d39d8-8b0c-4dc2-ab16-7626474d04de-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl\" (UID: \"ff5d39d8-8b0c-4dc2-ab16-7626474d04de\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl" Nov 03 22:50:07.508417 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:07.508306 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl" Nov 03 22:50:07.519957 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:07.517148 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:07.979785 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:07.979519 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl"] Nov 03 22:50:08.269829 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:08.269737 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl" event={"ID":"ff5d39d8-8b0c-4dc2-ab16-7626474d04de","Type":"ContainerStarted","Data":"1b5698880d69bbc9f4d2eb8d54889ecec5276d20a38809c15d0a9c341844d73f"} Nov 03 22:50:09.278822 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:09.278781 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl" event={"ID":"ff5d39d8-8b0c-4dc2-ab16-7626474d04de","Type":"ContainerStarted","Data":"dabc62bc0af004394bec0614005191fbeb7487e2d474ce0fa95aafa7c0c1f559"} Nov 03 22:50:09.279229 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:09.278872 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl" Nov 03 22:50:09.398888 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:09.398807 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl" podStartSLOduration=2.576227539 podStartE2EDuration="3.398792277s" totalImagesPullingTime="822.564738ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:50:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:50:09.397078635 +0000 UTC m=+1638.706637929" watchObservedRunningTime="2026-11-03 22:50:09.398792277 +0000 UTC m=+1638.708351567" Nov 03 22:50:11.009673 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:11.009630 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:11.724524 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:11.724491 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:12.763302 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:12.763274 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:12.916536 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:12.916496 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:15.292501 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:15.292470 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl" Nov 03 22:50:15.938276 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:15.938242 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:16.980484 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:16.980444 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s"] Nov 03 22:50:16.985668 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:16.985483 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" Nov 03 22:50:17.074541 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:17.074517 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ebffae22-openshift-default-dockercfg-hq996\"" Nov 03 22:50:17.077377 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:17.077346 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/441335de-a4e8-4d44-8728-d16ba47b88ab-istiod-ca-cert\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s\" (UID: \"441335de-a4e8-4d44-8728-d16ba47b88ab\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" Nov 03 22:50:17.078440 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:17.078419 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/441335de-a4e8-4d44-8728-d16ba47b88ab-istio-token\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s\" (UID: \"441335de-a4e8-4d44-8728-d16ba47b88ab\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" Nov 03 22:50:17.078608 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:17.078594 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/441335de-a4e8-4d44-8728-d16ba47b88ab-credential-socket\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s\" (UID: \"441335de-a4e8-4d44-8728-d16ba47b88ab\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" Nov 03 22:50:17.079110 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:17.079069 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/441335de-a4e8-4d44-8728-d16ba47b88ab-workload-certs\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s\" (UID: \"441335de-a4e8-4d44-8728-d16ba47b88ab\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" Nov 03 22:50:17.079336 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:17.079297 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c76b\" (UniqueName: \"kubernetes.io/projected/441335de-a4e8-4d44-8728-d16ba47b88ab-kube-api-access-2c76b\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s\" (UID: \"441335de-a4e8-4d44-8728-d16ba47b88ab\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" Nov 03 22:50:17.079614 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:17.079597 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/441335de-a4e8-4d44-8728-d16ba47b88ab-workload-socket\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s\" (UID: \"441335de-a4e8-4d44-8728-d16ba47b88ab\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" Nov 03 22:50:17.079988 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:17.079909 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/441335de-a4e8-4d44-8728-d16ba47b88ab-istio-envoy\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s\" (UID: \"441335de-a4e8-4d44-8728-d16ba47b88ab\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" Nov 03 22:50:17.080395 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:17.080299 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/441335de-a4e8-4d44-8728-d16ba47b88ab-istio-podinfo\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s\" (UID: \"441335de-a4e8-4d44-8728-d16ba47b88ab\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" Nov 03 22:50:17.080604 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:17.080590 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/441335de-a4e8-4d44-8728-d16ba47b88ab-istio-data\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s\" (UID: \"441335de-a4e8-4d44-8728-d16ba47b88ab\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" Nov 03 22:50:17.129424 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:17.129390 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s"] Nov 03 22:50:17.182369 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:17.182332 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/441335de-a4e8-4d44-8728-d16ba47b88ab-istio-token\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s\" (UID: \"441335de-a4e8-4d44-8728-d16ba47b88ab\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" Nov 03 22:50:17.182542 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:17.182392 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/441335de-a4e8-4d44-8728-d16ba47b88ab-credential-socket\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s\" (UID: \"441335de-a4e8-4d44-8728-d16ba47b88ab\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" Nov 03 22:50:17.182636 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:17.182567 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/441335de-a4e8-4d44-8728-d16ba47b88ab-workload-certs\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s\" (UID: \"441335de-a4e8-4d44-8728-d16ba47b88ab\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" Nov 03 22:50:17.182636 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:17.182602 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c76b\" (UniqueName: \"kubernetes.io/projected/441335de-a4e8-4d44-8728-d16ba47b88ab-kube-api-access-2c76b\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s\" (UID: \"441335de-a4e8-4d44-8728-d16ba47b88ab\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" Nov 03 22:50:17.182636 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:17.182631 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/441335de-a4e8-4d44-8728-d16ba47b88ab-workload-socket\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s\" (UID: \"441335de-a4e8-4d44-8728-d16ba47b88ab\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" Nov 03 22:50:17.182800 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:17.182677 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/441335de-a4e8-4d44-8728-d16ba47b88ab-istio-envoy\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s\" (UID: \"441335de-a4e8-4d44-8728-d16ba47b88ab\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" Nov 03 22:50:17.182800 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:17.182770 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/441335de-a4e8-4d44-8728-d16ba47b88ab-istio-podinfo\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s\" (UID: \"441335de-a4e8-4d44-8728-d16ba47b88ab\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" Nov 03 22:50:17.182949 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:17.182805 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/441335de-a4e8-4d44-8728-d16ba47b88ab-istio-data\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s\" (UID: \"441335de-a4e8-4d44-8728-d16ba47b88ab\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" Nov 03 22:50:17.182949 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:17.182895 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/441335de-a4e8-4d44-8728-d16ba47b88ab-credential-socket\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s\" (UID: \"441335de-a4e8-4d44-8728-d16ba47b88ab\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" Nov 03 22:50:17.182949 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:17.182920 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/441335de-a4e8-4d44-8728-d16ba47b88ab-istiod-ca-cert\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s\" (UID: \"441335de-a4e8-4d44-8728-d16ba47b88ab\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" Nov 03 22:50:17.183188 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:17.183164 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/441335de-a4e8-4d44-8728-d16ba47b88ab-workload-certs\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s\" (UID: \"441335de-a4e8-4d44-8728-d16ba47b88ab\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" Nov 03 22:50:17.183656 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:17.183628 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/441335de-a4e8-4d44-8728-d16ba47b88ab-workload-socket\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s\" (UID: \"441335de-a4e8-4d44-8728-d16ba47b88ab\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" Nov 03 22:50:17.183976 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:17.183669 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/441335de-a4e8-4d44-8728-d16ba47b88ab-istiod-ca-cert\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s\" (UID: \"441335de-a4e8-4d44-8728-d16ba47b88ab\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" Nov 03 22:50:17.183976 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:17.183729 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/441335de-a4e8-4d44-8728-d16ba47b88ab-istio-data\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s\" (UID: \"441335de-a4e8-4d44-8728-d16ba47b88ab\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" Nov 03 22:50:17.185795 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:17.185764 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/441335de-a4e8-4d44-8728-d16ba47b88ab-istio-envoy\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s\" (UID: \"441335de-a4e8-4d44-8728-d16ba47b88ab\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" Nov 03 22:50:17.185923 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:17.185775 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/441335de-a4e8-4d44-8728-d16ba47b88ab-istio-podinfo\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s\" (UID: \"441335de-a4e8-4d44-8728-d16ba47b88ab\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" Nov 03 22:50:17.244328 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:17.244265 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/441335de-a4e8-4d44-8728-d16ba47b88ab-istio-token\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s\" (UID: \"441335de-a4e8-4d44-8728-d16ba47b88ab\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" Nov 03 22:50:17.245362 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:17.245337 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c76b\" (UniqueName: \"kubernetes.io/projected/441335de-a4e8-4d44-8728-d16ba47b88ab-kube-api-access-2c76b\") pod \"e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s\" (UID: \"441335de-a4e8-4d44-8728-d16ba47b88ab\") " pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" Nov 03 22:50:17.304085 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:17.304060 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" Nov 03 22:50:17.630882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:17.629848 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:17.798890 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:17.794877 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s"] Nov 03 22:50:17.806321 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:50:17.806290 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod441335de_a4e8_4d44_8728_d16ba47b88ab.slice/crio-1e30247c395a91a7f257ad399bffba98b498beeebbad417c5762f69e84d38224 WatchSource:0}: Error finding container 1e30247c395a91a7f257ad399bffba98b498beeebbad417c5762f69e84d38224: Status 404 returned error can't find the container with id 1e30247c395a91a7f257ad399bffba98b498beeebbad417c5762f69e84d38224 Nov 03 22:50:17.809574 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:17.809536 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:50:17.809684 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:17.809611 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:50:17.809684 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:17.809653 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:50:17.853906 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:17.853875 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:18.005018 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:18.004983 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:18.369960 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:18.369915 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" event={"ID":"441335de-a4e8-4d44-8728-d16ba47b88ab","Type":"ContainerStarted","Data":"369bd997835415dec09a2505d632b236752b9e2ba47a4298047563ecb83c62c7"} Nov 03 22:50:18.370195 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:18.369978 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" event={"ID":"441335de-a4e8-4d44-8728-d16ba47b88ab","Type":"ContainerStarted","Data":"1e30247c395a91a7f257ad399bffba98b498beeebbad417c5762f69e84d38224"} Nov 03 22:50:18.506144 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:18.506078 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" podStartSLOduration=2.50605291 podStartE2EDuration="2.50605291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:50:18.497443815 +0000 UTC m=+1647.807003122" watchObservedRunningTime="2026-11-03 22:50:18.50605291 +0000 UTC m=+1647.815612204" Nov 03 22:50:18.772542 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:18.763017 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:19.314127 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:19.313027 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" Nov 03 22:50:19.316405 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:19.316378 5196 patch_prober.go:29] interesting pod/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 03 22:50:19.316511 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:19.316443 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" podUID="441335de-a4e8-4d44-8728-d16ba47b88ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 03 22:50:19.517337 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:19.517305 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:20.286937 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:20.286726 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:20.307656 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:20.305257 5196 patch_prober.go:29] interesting pod/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 03 22:50:20.307656 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:20.305309 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" podUID="441335de-a4e8-4d44-8728-d16ba47b88ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 03 22:50:20.738883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:20.738334 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:21.305319 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:21.305282 5196 patch_prober.go:29] interesting pod/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 03 22:50:21.305641 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:21.305612 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" podUID="441335de-a4e8-4d44-8728-d16ba47b88ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 03 22:50:22.310885 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:22.310329 5196 patch_prober.go:29] interesting pod/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 03 22:50:22.310885 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:22.310404 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" podUID="441335de-a4e8-4d44-8728-d16ba47b88ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 03 22:50:22.625348 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:22.625264 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:23.312872 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:23.312751 5196 patch_prober.go:29] interesting pod/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 03 22:50:23.324230 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:23.324190 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" podUID="441335de-a4e8-4d44-8728-d16ba47b88ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 03 22:50:23.635602 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:23.635559 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:23.947301 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:23.947216 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:24.311006 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:24.310927 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" Nov 03 22:50:24.311446 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:24.311429 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" Nov 03 22:50:24.312611 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:24.312590 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" Nov 03 22:50:24.433770 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:24.433739 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:24.699224 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:24.699179 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:24.851512 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:24.851475 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:25.002067 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:25.001973 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td"] Nov 03 22:50:25.002235 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:25.002194 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" podUID="ae839647-c999-4479-9174-3e4c389d24b8" containerName="istio-proxy" containerID="cri-o://77725e5d2ca56619f7237934fb35a2235bbbeff574359b1c8c63d27aa56dd04c" gracePeriod=30 Nov 03 22:50:25.035046 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:25.035007 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:25.396519 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:25.396482 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:25.569331 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:25.569297 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:25.739669 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:25.739568 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:25.891116 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:25.891080 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:25.920806 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:50:25.920772 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:50:26.921713 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:26.921670 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:27.183459 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:27.183337 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:29.897032 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:29.896998 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:30.064194 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.064147 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:30.244907 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.213295 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:30.427126 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.427093 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:50:30.576413 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.576323 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae839647-c999-4479-9174-3e4c389d24b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae839647-c999-4479-9174-3e4c389d24b8-workload-certs\") pod \"ae839647-c999-4479-9174-3e4c389d24b8\" (UID: \"ae839647-c999-4479-9174-3e4c389d24b8\") " Nov 03 22:50:30.576413 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.576401 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae839647-c999-4479-9174-3e4c389d24b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae839647-c999-4479-9174-3e4c389d24b8-istio-envoy\") pod \"ae839647-c999-4479-9174-3e4c389d24b8\" (UID: \"ae839647-c999-4479-9174-3e4c389d24b8\") " Nov 03 22:50:30.576657 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.576486 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae839647-c999-4479-9174-3e4c389d24b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae839647-c999-4479-9174-3e4c389d24b8-istiod-ca-cert\") pod \"ae839647-c999-4479-9174-3e4c389d24b8\" (UID: \"ae839647-c999-4479-9174-3e4c389d24b8\") " Nov 03 22:50:30.576657 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.576526 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae839647-c999-4479-9174-3e4c389d24b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae839647-c999-4479-9174-3e4c389d24b8-istio-podinfo\") pod \"ae839647-c999-4479-9174-3e4c389d24b8\" (UID: \"ae839647-c999-4479-9174-3e4c389d24b8\") " Nov 03 22:50:30.576657 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.576571 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae839647-c999-4479-9174-3e4c389d24b8-workload-certs" pod "ae839647-c999-4479-9174-3e4c389d24b8" (UID: "ae839647-c999-4479-9174-3e4c389d24b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:30.576657 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.576649 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae839647-c999-4479-9174-3e4c389d24b8-istio-token\" (UniqueName: \"kubernetes.io/projected/ae839647-c999-4479-9174-3e4c389d24b8-istio-token\") pod \"ae839647-c999-4479-9174-3e4c389d24b8\" (UID: \"ae839647-c999-4479-9174-3e4c389d24b8\") " Nov 03 22:50:30.576893 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.576685 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae839647-c999-4479-9174-3e4c389d24b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae839647-c999-4479-9174-3e4c389d24b8-credential-socket\") pod \"ae839647-c999-4479-9174-3e4c389d24b8\" (UID: \"ae839647-c999-4479-9174-3e4c389d24b8\") " Nov 03 22:50:30.576893 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.576701 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae839647-c999-4479-9174-3e4c389d24b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae839647-c999-4479-9174-3e4c389d24b8-istio-data\") pod \"ae839647-c999-4479-9174-3e4c389d24b8\" (UID: \"ae839647-c999-4479-9174-3e4c389d24b8\") " Nov 03 22:50:30.576893 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.576738 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae839647-c999-4479-9174-3e4c389d24b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae839647-c999-4479-9174-3e4c389d24b8-workload-socket\") pod \"ae839647-c999-4479-9174-3e4c389d24b8\" (UID: \"ae839647-c999-4479-9174-3e4c389d24b8\") " Nov 03 22:50:30.576893 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.576781 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae839647-c999-4479-9174-3e4c389d24b8-kube-api-access-kzrn9\" (UniqueName: \"kubernetes.io/projected/ae839647-c999-4479-9174-3e4c389d24b8-kube-api-access-kzrn9\") pod \"ae839647-c999-4479-9174-3e4c389d24b8\" (UID: \"ae839647-c999-4479-9174-3e4c389d24b8\") " Nov 03 22:50:30.577415 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.577382 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae839647-c999-4479-9174-3e4c389d24b8-istio-data" pod "ae839647-c999-4479-9174-3e4c389d24b8" (UID: "ae839647-c999-4479-9174-3e4c389d24b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:30.577415 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.577396 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae839647-c999-4479-9174-3e4c389d24b8-credential-socket" pod "ae839647-c999-4479-9174-3e4c389d24b8" (UID: "ae839647-c999-4479-9174-3e4c389d24b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:30.577628 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.577612 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae839647-c999-4479-9174-3e4c389d24b8-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:30.577709 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.577689 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae839647-c999-4479-9174-3e4c389d24b8-workload-socket" pod "ae839647-c999-4479-9174-3e4c389d24b8" (UID: "ae839647-c999-4479-9174-3e4c389d24b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:30.577804 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.577786 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae839647-c999-4479-9174-3e4c389d24b8-istiod-ca-cert" pod "ae839647-c999-4479-9174-3e4c389d24b8" (UID: "ae839647-c999-4479-9174-3e4c389d24b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:30.579833 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.579809 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae839647-c999-4479-9174-3e4c389d24b8-istio-podinfo" pod "ae839647-c999-4479-9174-3e4c389d24b8" (UID: "ae839647-c999-4479-9174-3e4c389d24b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:50:30.580343 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.580309 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae839647-c999-4479-9174-3e4c389d24b8-kube-api-access-kzrn9" pod "ae839647-c999-4479-9174-3e4c389d24b8" (UID: "ae839647-c999-4479-9174-3e4c389d24b8"). InnerVolumeSpecName "kube-api-access-kzrn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:30.581334 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.581309 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae839647-c999-4479-9174-3e4c389d24b8-istio-envoy" pod "ae839647-c999-4479-9174-3e4c389d24b8" (UID: "ae839647-c999-4479-9174-3e4c389d24b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:30.582536 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.582513 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae839647-c999-4479-9174-3e4c389d24b8-istio-token" pod "ae839647-c999-4479-9174-3e4c389d24b8" (UID: "ae839647-c999-4479-9174-3e4c389d24b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:30.678625 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.678591 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae839647-c999-4479-9174-3e4c389d24b8-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:30.678625 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.678622 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae839647-c999-4479-9174-3e4c389d24b8-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:30.678625 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.678631 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae839647-c999-4479-9174-3e4c389d24b8-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:30.679045 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.678639 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae839647-c999-4479-9174-3e4c389d24b8-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:30.679045 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.678650 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzrn9\" (UniqueName: \"kubernetes.io/projected/ae839647-c999-4479-9174-3e4c389d24b8-kube-api-access-kzrn9\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:30.679045 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.678669 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae839647-c999-4479-9174-3e4c389d24b8-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:30.679045 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.678677 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae839647-c999-4479-9174-3e4c389d24b8-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:30.679045 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.678686 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae839647-c999-4479-9174-3e4c389d24b8-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:50:30.725481 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.720252 5196 generic.go:356] "Generic (PLEG): container finished" podID="ae839647-c999-4479-9174-3e4c389d24b8" containerID="77725e5d2ca56619f7237934fb35a2235bbbeff574359b1c8c63d27aa56dd04c" exitCode=0 Nov 03 22:50:30.725481 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.720460 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" event={"ID":"ae839647-c999-4479-9174-3e4c389d24b8","Type":"ContainerDied","Data":"77725e5d2ca56619f7237934fb35a2235bbbeff574359b1c8c63d27aa56dd04c"} Nov 03 22:50:30.725481 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.720509 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" event={"ID":"ae839647-c999-4479-9174-3e4c389d24b8","Type":"ContainerDied","Data":"d646e0fbf8c57ce5564e89d5d481f322bd6d9d31a9d7c0eb9f1eb2a0b798e2ff"} Nov 03 22:50:30.725481 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.720542 5196 scope.go:122] "RemoveContainer" containerID="77725e5d2ca56619f7237934fb35a2235bbbeff574359b1c8c63d27aa56dd04c" Nov 03 22:50:30.725481 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.720789 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td" Nov 03 22:50:30.742580 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.742556 5196 scope.go:122] "RemoveContainer" containerID="72554492ecab9434f7b00cc6dfdc65372965387acd8c498e27dc31ff42e14c3a" Nov 03 22:50:30.754460 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.754426 5196 scope.go:122] "RemoveContainer" containerID="77725e5d2ca56619f7237934fb35a2235bbbeff574359b1c8c63d27aa56dd04c" Nov 03 22:50:30.754841 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:50:30.754805 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77725e5d2ca56619f7237934fb35a2235bbbeff574359b1c8c63d27aa56dd04c\": container with ID starting with 77725e5d2ca56619f7237934fb35a2235bbbeff574359b1c8c63d27aa56dd04c not found: ID does not exist" containerID="77725e5d2ca56619f7237934fb35a2235bbbeff574359b1c8c63d27aa56dd04c" Nov 03 22:50:30.754968 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.754898 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77725e5d2ca56619f7237934fb35a2235bbbeff574359b1c8c63d27aa56dd04c"} err="failed to get container status \"77725e5d2ca56619f7237934fb35a2235bbbeff574359b1c8c63d27aa56dd04c\": rpc error: code = NotFound desc = could not find container \"77725e5d2ca56619f7237934fb35a2235bbbeff574359b1c8c63d27aa56dd04c\": container with ID starting with 77725e5d2ca56619f7237934fb35a2235bbbeff574359b1c8c63d27aa56dd04c not found: ID does not exist" Nov 03 22:50:30.754968 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.754925 5196 scope.go:122] "RemoveContainer" containerID="72554492ecab9434f7b00cc6dfdc65372965387acd8c498e27dc31ff42e14c3a" Nov 03 22:50:30.755328 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:50:30.755301 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72554492ecab9434f7b00cc6dfdc65372965387acd8c498e27dc31ff42e14c3a\": container with ID starting with 72554492ecab9434f7b00cc6dfdc65372965387acd8c498e27dc31ff42e14c3a not found: ID does not exist" containerID="72554492ecab9434f7b00cc6dfdc65372965387acd8c498e27dc31ff42e14c3a" Nov 03 22:50:30.755429 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.755349 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72554492ecab9434f7b00cc6dfdc65372965387acd8c498e27dc31ff42e14c3a"} err="failed to get container status \"72554492ecab9434f7b00cc6dfdc65372965387acd8c498e27dc31ff42e14c3a\": rpc error: code = NotFound desc = could not find container \"72554492ecab9434f7b00cc6dfdc65372965387acd8c498e27dc31ff42e14c3a\": container with ID starting with 72554492ecab9434f7b00cc6dfdc65372965387acd8c498e27dc31ff42e14c3a not found: ID does not exist" Nov 03 22:50:30.898928 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.898898 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td"] Nov 03 22:50:30.904718 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:30.904689 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-9d7aa682-openshift-default-74bdd985b5-sv4td"] Nov 03 22:50:31.062218 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:31.062177 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:31.231517 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:31.230201 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:31.385574 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:31.385541 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:32.816704 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:32.816668 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae839647-c999-4479-9174-3e4c389d24b8" path="/var/lib/kubelet/pods/ae839647-c999-4479-9174-3e4c389d24b8/volumes" Nov 03 22:50:34.994073 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:34.994033 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:35.147241 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:35.147215 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:35.351933 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:35.351251 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:40.431871 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:40.431821 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:43.302086 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:43.302046 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:44.160646 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:44.160613 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:44.338437 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:44.338404 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:44.912035 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:44.911995 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:45.419570 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:45.419531 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:45.628675 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:45.628638 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:50.709638 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:50.709604 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:51.579324 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:51.579242 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:52.242350 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:52.242316 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:55.506223 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:50:55.506188 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:50:55.920157 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:50:55.920121 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:51:00.228706 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:00.228665 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:04.967681 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:04.967625 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:05.144967 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:05.144934 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:05.298097 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:05.298024 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:05.810086 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:05.810053 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:07.868694 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:07.868662 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g"] Nov 03 22:51:07.869229 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:07.868884 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" podUID="cacf0851-d190-4ef3-8df0-e181b4444430" containerName="istio-proxy" containerID="cri-o://edb6ad9aabcee8b78073613b96b985deef1a5a72c4d2e4b1672a97edb6406383" gracePeriod=30 Nov 03 22:51:07.922414 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:07.922380 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq"] Nov 03 22:51:07.922684 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:07.922639 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" podUID="5ee2347e-d2c8-4319-a5f8-f926e83ae1c6" containerName="istio-proxy" containerID="cri-o://fcb540b2641ca06fc9dbc1019e9965a288bbf186c22bbf5e1be3b12d40a55646" gracePeriod=30 Nov 03 22:51:07.927196 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:07.927170 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj"] Nov 03 22:51:07.927416 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:07.927389 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" podUID="b0aafab2-abe8-4dd8-81ea-71278482df2b" containerName="istio-proxy" containerID="cri-o://2103d3f8cb26b75106b0ad40d3b841a5c8c60ba6adf5bd50917c1b9551ed594b" gracePeriod=30 Nov 03 22:51:07.970492 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:07.970460 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8"] Nov 03 22:51:07.970707 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:07.970670 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" podUID="c9bf76ea-2e4e-46ee-9c23-3f084e9f8475" containerName="istio-proxy" containerID="cri-o://9a957a86b24bfae22736f54372a25239a3d03fcb45f7ce9cf94a7eced1aeb886" gracePeriod=30 Nov 03 22:51:08.549570 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:08.549537 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv"] Nov 03 22:51:08.549799 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:08.549774 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" podUID="bdc73375-64fa-42d8-8ba8-23a936fd1f06" containerName="istio-proxy" containerID="cri-o://6c5a5d87185c8163c7d8735001be10125282042d09f99a6850bd6d1e40e77b97" gracePeriod=30 Nov 03 22:51:08.553371 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:08.553343 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h"] Nov 03 22:51:08.553580 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:08.553552 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" podUID="cea0ff59-b9f4-49e6-995e-0c568ca8dd9c" containerName="istio-proxy" containerID="cri-o://e3afd01a447b93d65eacbf6d57ae5355d38e6eab04b6cedc9423f275527b2770" gracePeriod=30 Nov 03 22:51:08.557020 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:08.556998 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk"] Nov 03 22:51:08.557197 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:08.557178 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" podUID="03244edf-87f5-4237-be87-8a523c4d96b3" containerName="istio-proxy" containerID="cri-o://48aff0ee2038482766ac246ed8191f082ac6300f2908702843d62a0ed2e9274d" gracePeriod=30 Nov 03 22:51:08.560901 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:08.560876 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm"] Nov 03 22:51:08.561068 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:08.561049 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" podUID="3a1c6229-58c7-412d-b67d-9a31e460a59c" containerName="istio-proxy" containerID="cri-o://ca502e0ebe562a6264e655b1f9b99bdef227a9ffd7b50f3623e30c3a9a273f13" gracePeriod=30 Nov 03 22:51:08.879834 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:08.879804 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:10.199049 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:10.199010 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:11.359285 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:11.359252 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:11.883394 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:11.883362 5196 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:51:11.883586 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:11.883433 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" podUID="03244edf-87f5-4237-be87-8a523c4d96b3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:51:11.973905 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:11.973851 5196 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:51:11.974058 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:11.973920 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" podUID="b0aafab2-abe8-4dd8-81ea-71278482df2b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:51:12.557490 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:12.557457 5196 patch_prober.go:29] interesting pod/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:51:12.558138 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:12.558107 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" podUID="c9bf76ea-2e4e-46ee-9c23-3f084e9f8475" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:51:13.115085 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.114506 5196 generic.go:356] "Generic (PLEG): container finished" podID="5ee2347e-d2c8-4319-a5f8-f926e83ae1c6" containerID="fcb540b2641ca06fc9dbc1019e9965a288bbf186c22bbf5e1be3b12d40a55646" exitCode=0 Nov 03 22:51:13.115085 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.114623 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" event={"ID":"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6","Type":"ContainerDied","Data":"fcb540b2641ca06fc9dbc1019e9965a288bbf186c22bbf5e1be3b12d40a55646"} Nov 03 22:51:13.117288 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.117259 5196 generic.go:356] "Generic (PLEG): container finished" podID="cacf0851-d190-4ef3-8df0-e181b4444430" containerID="edb6ad9aabcee8b78073613b96b985deef1a5a72c4d2e4b1672a97edb6406383" exitCode=0 Nov 03 22:51:13.117458 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.117395 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" event={"ID":"cacf0851-d190-4ef3-8df0-e181b4444430","Type":"ContainerDied","Data":"edb6ad9aabcee8b78073613b96b985deef1a5a72c4d2e4b1672a97edb6406383"} Nov 03 22:51:13.120257 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.119629 5196 generic.go:356] "Generic (PLEG): container finished" podID="b0aafab2-abe8-4dd8-81ea-71278482df2b" containerID="2103d3f8cb26b75106b0ad40d3b841a5c8c60ba6adf5bd50917c1b9551ed594b" exitCode=0 Nov 03 22:51:13.120257 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.119695 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" event={"ID":"b0aafab2-abe8-4dd8-81ea-71278482df2b","Type":"ContainerDied","Data":"2103d3f8cb26b75106b0ad40d3b841a5c8c60ba6adf5bd50917c1b9551ed594b"} Nov 03 22:51:13.196170 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.194776 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" Nov 03 22:51:13.216143 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.216086 5196 patch_prober.go:29] interesting pod/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:51:13.216292 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.216172 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" podUID="cea0ff59-b9f4-49e6-995e-0c568ca8dd9c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:51:13.246058 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.246002 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-istio-podinfo\") pod \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\" (UID: \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\") " Nov 03 22:51:13.246234 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.246104 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-credential-socket\") pod \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\" (UID: \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\") " Nov 03 22:51:13.246519 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.246423 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-istio-token\" (UniqueName: \"kubernetes.io/projected/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-istio-token\") pod \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\" (UID: \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\") " Nov 03 22:51:13.246519 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.246479 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-istio-data\") pod \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\" (UID: \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\") " Nov 03 22:51:13.246692 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.246540 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-workload-socket\") pod \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\" (UID: \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\") " Nov 03 22:51:13.246692 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.246642 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-kube-api-access-gmkb6\" (UniqueName: \"kubernetes.io/projected/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-kube-api-access-gmkb6\") pod \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\" (UID: \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\") " Nov 03 22:51:13.246692 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.246669 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-workload-certs\") pod \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\" (UID: \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\") " Nov 03 22:51:13.246877 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.246725 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-istiod-ca-cert\") pod \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\" (UID: \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\") " Nov 03 22:51:13.246877 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.246812 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-istio-envoy\") pod \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\" (UID: \"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6\") " Nov 03 22:51:13.247002 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.246849 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-istio-data" pod "5ee2347e-d2c8-4319-a5f8-f926e83ae1c6" (UID: "5ee2347e-d2c8-4319-a5f8-f926e83ae1c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:13.247066 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.246976 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-workload-socket" pod "5ee2347e-d2c8-4319-a5f8-f926e83ae1c6" (UID: "5ee2347e-d2c8-4319-a5f8-f926e83ae1c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:13.247127 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.247097 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-workload-certs" pod "5ee2347e-d2c8-4319-a5f8-f926e83ae1c6" (UID: "5ee2347e-d2c8-4319-a5f8-f926e83ae1c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:13.247456 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.247429 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-istiod-ca-cert" pod "5ee2347e-d2c8-4319-a5f8-f926e83ae1c6" (UID: "5ee2347e-d2c8-4319-a5f8-f926e83ae1c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:13.247536 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.247525 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:13.247595 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.247542 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:13.247595 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.247558 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:13.247595 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.247571 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:13.247730 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.247647 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-credential-socket" pod "5ee2347e-d2c8-4319-a5f8-f926e83ae1c6" (UID: "5ee2347e-d2c8-4319-a5f8-f926e83ae1c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:13.253398 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.253356 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-istio-podinfo" pod "5ee2347e-d2c8-4319-a5f8-f926e83ae1c6" (UID: "5ee2347e-d2c8-4319-a5f8-f926e83ae1c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:51:13.253398 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.253373 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-istio-envoy" pod "5ee2347e-d2c8-4319-a5f8-f926e83ae1c6" (UID: "5ee2347e-d2c8-4319-a5f8-f926e83ae1c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:13.253526 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.253417 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-kube-api-access-gmkb6" pod "5ee2347e-d2c8-4319-a5f8-f926e83ae1c6" (UID: "5ee2347e-d2c8-4319-a5f8-f926e83ae1c6"). InnerVolumeSpecName "kube-api-access-gmkb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:13.254294 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.254266 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-istio-token" pod "5ee2347e-d2c8-4319-a5f8-f926e83ae1c6" (UID: "5ee2347e-d2c8-4319-a5f8-f926e83ae1c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:13.277098 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.277032 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" Nov 03 22:51:13.332385 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.332355 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" Nov 03 22:51:13.348934 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.348902 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cacf0851-d190-4ef3-8df0-e181b4444430-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cacf0851-d190-4ef3-8df0-e181b4444430-istio-envoy\") pod \"cacf0851-d190-4ef3-8df0-e181b4444430\" (UID: \"cacf0851-d190-4ef3-8df0-e181b4444430\") " Nov 03 22:51:13.349101 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.348989 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cacf0851-d190-4ef3-8df0-e181b4444430-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacf0851-d190-4ef3-8df0-e181b4444430-credential-socket\") pod \"cacf0851-d190-4ef3-8df0-e181b4444430\" (UID: \"cacf0851-d190-4ef3-8df0-e181b4444430\") " Nov 03 22:51:13.349101 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.349029 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cacf0851-d190-4ef3-8df0-e181b4444430-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cacf0851-d190-4ef3-8df0-e181b4444430-istio-data\") pod \"cacf0851-d190-4ef3-8df0-e181b4444430\" (UID: \"cacf0851-d190-4ef3-8df0-e181b4444430\") " Nov 03 22:51:13.349295 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.349182 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cacf0851-d190-4ef3-8df0-e181b4444430-kube-api-access-jbs2j\" (UniqueName: \"kubernetes.io/projected/cacf0851-d190-4ef3-8df0-e181b4444430-kube-api-access-jbs2j\") pod \"cacf0851-d190-4ef3-8df0-e181b4444430\" (UID: \"cacf0851-d190-4ef3-8df0-e181b4444430\") " Nov 03 22:51:13.349295 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.349224 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cacf0851-d190-4ef3-8df0-e181b4444430-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cacf0851-d190-4ef3-8df0-e181b4444430-workload-certs\") pod \"cacf0851-d190-4ef3-8df0-e181b4444430\" (UID: \"cacf0851-d190-4ef3-8df0-e181b4444430\") " Nov 03 22:51:13.349295 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.349276 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cacf0851-d190-4ef3-8df0-e181b4444430-credential-socket" pod "cacf0851-d190-4ef3-8df0-e181b4444430" (UID: "cacf0851-d190-4ef3-8df0-e181b4444430"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:13.349452 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.349292 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cacf0851-d190-4ef3-8df0-e181b4444430-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacf0851-d190-4ef3-8df0-e181b4444430-workload-socket\") pod \"cacf0851-d190-4ef3-8df0-e181b4444430\" (UID: \"cacf0851-d190-4ef3-8df0-e181b4444430\") " Nov 03 22:51:13.349452 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.349402 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cacf0851-d190-4ef3-8df0-e181b4444430-istio-token\" (UniqueName: \"kubernetes.io/projected/cacf0851-d190-4ef3-8df0-e181b4444430-istio-token\") pod \"cacf0851-d190-4ef3-8df0-e181b4444430\" (UID: \"cacf0851-d190-4ef3-8df0-e181b4444430\") " Nov 03 22:51:13.349452 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.349458 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cacf0851-d190-4ef3-8df0-e181b4444430-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cacf0851-d190-4ef3-8df0-e181b4444430-istio-podinfo\") pod \"cacf0851-d190-4ef3-8df0-e181b4444430\" (UID: \"cacf0851-d190-4ef3-8df0-e181b4444430\") " Nov 03 22:51:13.349621 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.349490 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cacf0851-d190-4ef3-8df0-e181b4444430-workload-certs" pod "cacf0851-d190-4ef3-8df0-e181b4444430" (UID: "cacf0851-d190-4ef3-8df0-e181b4444430"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:13.349621 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.349505 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cacf0851-d190-4ef3-8df0-e181b4444430-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cacf0851-d190-4ef3-8df0-e181b4444430-istiod-ca-cert\") pod \"cacf0851-d190-4ef3-8df0-e181b4444430\" (UID: \"cacf0851-d190-4ef3-8df0-e181b4444430\") " Nov 03 22:51:13.349621 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.349529 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cacf0851-d190-4ef3-8df0-e181b4444430-workload-socket" pod "cacf0851-d190-4ef3-8df0-e181b4444430" (UID: "cacf0851-d190-4ef3-8df0-e181b4444430"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:13.349783 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.349765 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cacf0851-d190-4ef3-8df0-e181b4444430-istio-data" pod "cacf0851-d190-4ef3-8df0-e181b4444430" (UID: "cacf0851-d190-4ef3-8df0-e181b4444430"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:13.350125 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.350103 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cacf0851-d190-4ef3-8df0-e181b4444430-istiod-ca-cert" pod "cacf0851-d190-4ef3-8df0-e181b4444430" (UID: "cacf0851-d190-4ef3-8df0-e181b4444430"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:13.350416 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.350395 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:13.350504 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.350419 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:13.350504 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.350431 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacf0851-d190-4ef3-8df0-e181b4444430-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:13.350504 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.350441 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cacf0851-d190-4ef3-8df0-e181b4444430-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:13.350504 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.350450 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmkb6\" (UniqueName: \"kubernetes.io/projected/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-kube-api-access-gmkb6\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:13.350504 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.350462 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:13.350504 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.350477 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cacf0851-d190-4ef3-8df0-e181b4444430-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:13.350504 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.350490 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cacf0851-d190-4ef3-8df0-e181b4444430-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:13.350504 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.350499 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:13.350504 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.350507 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cacf0851-d190-4ef3-8df0-e181b4444430-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:13.352134 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.352110 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cacf0851-d190-4ef3-8df0-e181b4444430-istio-podinfo" pod "cacf0851-d190-4ef3-8df0-e181b4444430" (UID: "cacf0851-d190-4ef3-8df0-e181b4444430"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:51:13.352362 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.352340 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cacf0851-d190-4ef3-8df0-e181b4444430-istio-envoy" pod "cacf0851-d190-4ef3-8df0-e181b4444430" (UID: "cacf0851-d190-4ef3-8df0-e181b4444430"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:13.352819 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.352800 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cacf0851-d190-4ef3-8df0-e181b4444430-istio-token" pod "cacf0851-d190-4ef3-8df0-e181b4444430" (UID: "cacf0851-d190-4ef3-8df0-e181b4444430"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:13.352954 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.352825 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cacf0851-d190-4ef3-8df0-e181b4444430-kube-api-access-jbs2j" pod "cacf0851-d190-4ef3-8df0-e181b4444430" (UID: "cacf0851-d190-4ef3-8df0-e181b4444430"). InnerVolumeSpecName "kube-api-access-jbs2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:13.420063 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.419412 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" Nov 03 22:51:13.451812 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.451784 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0aafab2-abe8-4dd8-81ea-71278482df2b-kube-api-access-rpx22\" (UniqueName: \"kubernetes.io/projected/b0aafab2-abe8-4dd8-81ea-71278482df2b-kube-api-access-rpx22\") pod \"b0aafab2-abe8-4dd8-81ea-71278482df2b\" (UID: \"b0aafab2-abe8-4dd8-81ea-71278482df2b\") " Nov 03 22:51:13.452009 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.451840 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0aafab2-abe8-4dd8-81ea-71278482df2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0aafab2-abe8-4dd8-81ea-71278482df2b-credential-socket\") pod \"b0aafab2-abe8-4dd8-81ea-71278482df2b\" (UID: \"b0aafab2-abe8-4dd8-81ea-71278482df2b\") " Nov 03 22:51:13.452009 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.451919 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0aafab2-abe8-4dd8-81ea-71278482df2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0aafab2-abe8-4dd8-81ea-71278482df2b-workload-certs\") pod \"b0aafab2-abe8-4dd8-81ea-71278482df2b\" (UID: \"b0aafab2-abe8-4dd8-81ea-71278482df2b\") " Nov 03 22:51:13.452129 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.452030 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0aafab2-abe8-4dd8-81ea-71278482df2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0aafab2-abe8-4dd8-81ea-71278482df2b-istio-envoy\") pod \"b0aafab2-abe8-4dd8-81ea-71278482df2b\" (UID: \"b0aafab2-abe8-4dd8-81ea-71278482df2b\") " Nov 03 22:51:13.452129 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.452073 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0aafab2-abe8-4dd8-81ea-71278482df2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0aafab2-abe8-4dd8-81ea-71278482df2b-workload-socket\") pod \"b0aafab2-abe8-4dd8-81ea-71278482df2b\" (UID: \"b0aafab2-abe8-4dd8-81ea-71278482df2b\") " Nov 03 22:51:13.452129 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.452100 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0aafab2-abe8-4dd8-81ea-71278482df2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0aafab2-abe8-4dd8-81ea-71278482df2b-istio-podinfo\") pod \"b0aafab2-abe8-4dd8-81ea-71278482df2b\" (UID: \"b0aafab2-abe8-4dd8-81ea-71278482df2b\") " Nov 03 22:51:13.452283 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.452178 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0aafab2-abe8-4dd8-81ea-71278482df2b-credential-socket" pod "b0aafab2-abe8-4dd8-81ea-71278482df2b" (UID: "b0aafab2-abe8-4dd8-81ea-71278482df2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:13.452283 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.452215 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0aafab2-abe8-4dd8-81ea-71278482df2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0aafab2-abe8-4dd8-81ea-71278482df2b-istiod-ca-cert\") pod \"b0aafab2-abe8-4dd8-81ea-71278482df2b\" (UID: \"b0aafab2-abe8-4dd8-81ea-71278482df2b\") " Nov 03 22:51:13.452283 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.452269 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0aafab2-abe8-4dd8-81ea-71278482df2b-istio-token\" (UniqueName: \"kubernetes.io/projected/b0aafab2-abe8-4dd8-81ea-71278482df2b-istio-token\") pod \"b0aafab2-abe8-4dd8-81ea-71278482df2b\" (UID: \"b0aafab2-abe8-4dd8-81ea-71278482df2b\") " Nov 03 22:51:13.452445 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.452296 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0aafab2-abe8-4dd8-81ea-71278482df2b-workload-certs" pod "b0aafab2-abe8-4dd8-81ea-71278482df2b" (UID: "b0aafab2-abe8-4dd8-81ea-71278482df2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:13.452445 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.452361 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0aafab2-abe8-4dd8-81ea-71278482df2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0aafab2-abe8-4dd8-81ea-71278482df2b-istio-data\") pod \"b0aafab2-abe8-4dd8-81ea-71278482df2b\" (UID: \"b0aafab2-abe8-4dd8-81ea-71278482df2b\") " Nov 03 22:51:13.452445 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.452394 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0aafab2-abe8-4dd8-81ea-71278482df2b-workload-socket" pod "b0aafab2-abe8-4dd8-81ea-71278482df2b" (UID: "b0aafab2-abe8-4dd8-81ea-71278482df2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:13.452851 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.452673 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0aafab2-abe8-4dd8-81ea-71278482df2b-istio-data" pod "b0aafab2-abe8-4dd8-81ea-71278482df2b" (UID: "b0aafab2-abe8-4dd8-81ea-71278482df2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:13.452851 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.452620 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0aafab2-abe8-4dd8-81ea-71278482df2b-istiod-ca-cert" pod "b0aafab2-abe8-4dd8-81ea-71278482df2b" (UID: "b0aafab2-abe8-4dd8-81ea-71278482df2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:13.453070 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.453041 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0aafab2-abe8-4dd8-81ea-71278482df2b-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:13.453070 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.453060 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0aafab2-abe8-4dd8-81ea-71278482df2b-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:13.453070 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.453069 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cacf0851-d190-4ef3-8df0-e181b4444430-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:13.453256 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.453078 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0aafab2-abe8-4dd8-81ea-71278482df2b-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:13.453256 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.453086 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0aafab2-abe8-4dd8-81ea-71278482df2b-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:13.453256 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.453094 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbs2j\" (UniqueName: \"kubernetes.io/projected/cacf0851-d190-4ef3-8df0-e181b4444430-kube-api-access-jbs2j\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:13.453256 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.453109 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0aafab2-abe8-4dd8-81ea-71278482df2b-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:13.453256 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.453122 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cacf0851-d190-4ef3-8df0-e181b4444430-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:13.453256 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.453136 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cacf0851-d190-4ef3-8df0-e181b4444430-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:13.454491 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.454469 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0aafab2-abe8-4dd8-81ea-71278482df2b-istio-podinfo" pod "b0aafab2-abe8-4dd8-81ea-71278482df2b" (UID: "b0aafab2-abe8-4dd8-81ea-71278482df2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:51:13.454491 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.454476 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0aafab2-abe8-4dd8-81ea-71278482df2b-kube-api-access-rpx22" pod "b0aafab2-abe8-4dd8-81ea-71278482df2b" (UID: "b0aafab2-abe8-4dd8-81ea-71278482df2b"). InnerVolumeSpecName "kube-api-access-rpx22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:13.454884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.454851 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0aafab2-abe8-4dd8-81ea-71278482df2b-istio-envoy" pod "b0aafab2-abe8-4dd8-81ea-71278482df2b" (UID: "b0aafab2-abe8-4dd8-81ea-71278482df2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:13.454977 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.454964 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0aafab2-abe8-4dd8-81ea-71278482df2b-istio-token" pod "b0aafab2-abe8-4dd8-81ea-71278482df2b" (UID: "b0aafab2-abe8-4dd8-81ea-71278482df2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:13.554165 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.554136 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-workload-socket\") pod \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\" (UID: \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\") " Nov 03 22:51:13.554345 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.554186 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-kube-api-access-s2cdm\" (UniqueName: \"kubernetes.io/projected/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-kube-api-access-s2cdm\") pod \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\" (UID: \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\") " Nov 03 22:51:13.554345 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.554271 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-istiod-ca-cert\") pod \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\" (UID: \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\") " Nov 03 22:51:13.554483 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.554348 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-istio-podinfo\") pod \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\" (UID: \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\") " Nov 03 22:51:13.554483 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.554371 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-workload-socket" pod "c9bf76ea-2e4e-46ee-9c23-3f084e9f8475" (UID: "c9bf76ea-2e4e-46ee-9c23-3f084e9f8475"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:13.554483 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.554421 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-workload-certs\") pod \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\" (UID: \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\") " Nov 03 22:51:13.554483 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.554463 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-credential-socket\") pod \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\" (UID: \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\") " Nov 03 22:51:13.556286 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.554512 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-istio-token\" (UniqueName: \"kubernetes.io/projected/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-istio-token\") pod \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\" (UID: \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\") " Nov 03 22:51:13.556286 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.554878 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-istiod-ca-cert" pod "c9bf76ea-2e4e-46ee-9c23-3f084e9f8475" (UID: "c9bf76ea-2e4e-46ee-9c23-3f084e9f8475"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:13.556286 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.554900 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-istio-envoy\") pod \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\" (UID: \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\") " Nov 03 22:51:13.556286 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.555004 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-istio-data\") pod \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\" (UID: \"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475\") " Nov 03 22:51:13.556286 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.555059 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-workload-certs" pod "c9bf76ea-2e4e-46ee-9c23-3f084e9f8475" (UID: "c9bf76ea-2e4e-46ee-9c23-3f084e9f8475"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:13.556286 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.555598 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-credential-socket" pod "c9bf76ea-2e4e-46ee-9c23-3f084e9f8475" (UID: "c9bf76ea-2e4e-46ee-9c23-3f084e9f8475"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:13.556286 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.555736 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-istio-data" pod "c9bf76ea-2e4e-46ee-9c23-3f084e9f8475" (UID: "c9bf76ea-2e4e-46ee-9c23-3f084e9f8475"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:13.556286 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.555786 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0aafab2-abe8-4dd8-81ea-71278482df2b-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:13.556286 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.555808 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:13.556286 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.555823 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpx22\" (UniqueName: \"kubernetes.io/projected/b0aafab2-abe8-4dd8-81ea-71278482df2b-kube-api-access-rpx22\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:13.556286 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.555838 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:13.556286 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.555851 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0aafab2-abe8-4dd8-81ea-71278482df2b-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:13.556286 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.555885 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0aafab2-abe8-4dd8-81ea-71278482df2b-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:13.556286 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.555898 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:13.556286 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.555912 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:13.557130 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.556827 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-istio-podinfo" pod "c9bf76ea-2e4e-46ee-9c23-3f084e9f8475" (UID: "c9bf76ea-2e4e-46ee-9c23-3f084e9f8475"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:51:13.557356 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.557327 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-kube-api-access-s2cdm" pod "c9bf76ea-2e4e-46ee-9c23-3f084e9f8475" (UID: "c9bf76ea-2e4e-46ee-9c23-3f084e9f8475"). InnerVolumeSpecName "kube-api-access-s2cdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:13.557691 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.557671 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-istio-envoy" pod "c9bf76ea-2e4e-46ee-9c23-3f084e9f8475" (UID: "c9bf76ea-2e4e-46ee-9c23-3f084e9f8475"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:13.558139 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.557832 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-istio-token" pod "c9bf76ea-2e4e-46ee-9c23-3f084e9f8475" (UID: "c9bf76ea-2e4e-46ee-9c23-3f084e9f8475"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:13.657890 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.657789 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:13.657890 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.657826 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:13.657890 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.657841 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:13.657890 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.657874 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2cdm\" (UniqueName: \"kubernetes.io/projected/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-kube-api-access-s2cdm\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:13.657890 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.657890 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:13.810020 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.809986 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:13.867736 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.867707 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" Nov 03 22:51:13.978059 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.963589 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdc73375-64fa-42d8-8ba8-23a936fd1f06-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdc73375-64fa-42d8-8ba8-23a936fd1f06-istio-data\") pod \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\" (UID: \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\") " Nov 03 22:51:13.978059 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.963651 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bdc73375-64fa-42d8-8ba8-23a936fd1f06-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdc73375-64fa-42d8-8ba8-23a936fd1f06-istio-podinfo\") pod \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\" (UID: \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\") " Nov 03 22:51:13.978059 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.963710 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdc73375-64fa-42d8-8ba8-23a936fd1f06-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdc73375-64fa-42d8-8ba8-23a936fd1f06-workload-socket\") pod \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\" (UID: \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\") " Nov 03 22:51:13.978059 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.963754 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdc73375-64fa-42d8-8ba8-23a936fd1f06-kube-api-access-xfxgj\" (UniqueName: \"kubernetes.io/projected/bdc73375-64fa-42d8-8ba8-23a936fd1f06-kube-api-access-xfxgj\") pod \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\" (UID: \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\") " Nov 03 22:51:13.978059 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.963816 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdc73375-64fa-42d8-8ba8-23a936fd1f06-istio-token\" (UniqueName: \"kubernetes.io/projected/bdc73375-64fa-42d8-8ba8-23a936fd1f06-istio-token\") pod \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\" (UID: \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\") " Nov 03 22:51:13.978059 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.963907 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdc73375-64fa-42d8-8ba8-23a936fd1f06-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdc73375-64fa-42d8-8ba8-23a936fd1f06-workload-certs\") pod \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\" (UID: \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\") " Nov 03 22:51:13.978059 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.964126 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdc73375-64fa-42d8-8ba8-23a936fd1f06-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdc73375-64fa-42d8-8ba8-23a936fd1f06-istiod-ca-cert\") pod \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\" (UID: \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\") " Nov 03 22:51:13.978059 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.964177 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdc73375-64fa-42d8-8ba8-23a936fd1f06-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdc73375-64fa-42d8-8ba8-23a936fd1f06-istio-envoy\") pod \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\" (UID: \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\") " Nov 03 22:51:13.978059 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.964267 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdc73375-64fa-42d8-8ba8-23a936fd1f06-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdc73375-64fa-42d8-8ba8-23a936fd1f06-credential-socket\") pod \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\" (UID: \"bdc73375-64fa-42d8-8ba8-23a936fd1f06\") " Nov 03 22:51:13.978059 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.965176 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdc73375-64fa-42d8-8ba8-23a936fd1f06-credential-socket" pod "bdc73375-64fa-42d8-8ba8-23a936fd1f06" (UID: "bdc73375-64fa-42d8-8ba8-23a936fd1f06"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:13.978059 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.965376 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdc73375-64fa-42d8-8ba8-23a936fd1f06-istio-data" pod "bdc73375-64fa-42d8-8ba8-23a936fd1f06" (UID: "bdc73375-64fa-42d8-8ba8-23a936fd1f06"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:13.978059 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.971546 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdc73375-64fa-42d8-8ba8-23a936fd1f06-workload-certs" pod "bdc73375-64fa-42d8-8ba8-23a936fd1f06" (UID: "bdc73375-64fa-42d8-8ba8-23a936fd1f06"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:13.978059 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.972029 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdc73375-64fa-42d8-8ba8-23a936fd1f06-istiod-ca-cert" pod "bdc73375-64fa-42d8-8ba8-23a936fd1f06" (UID: "bdc73375-64fa-42d8-8ba8-23a936fd1f06"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:13.978059 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.972296 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdc73375-64fa-42d8-8ba8-23a936fd1f06-workload-socket" pod "bdc73375-64fa-42d8-8ba8-23a936fd1f06" (UID: "bdc73375-64fa-42d8-8ba8-23a936fd1f06"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:13.987009 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.986964 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bdc73375-64fa-42d8-8ba8-23a936fd1f06-istio-podinfo" pod "bdc73375-64fa-42d8-8ba8-23a936fd1f06" (UID: "bdc73375-64fa-42d8-8ba8-23a936fd1f06"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:51:13.995336 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.992087 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdc73375-64fa-42d8-8ba8-23a936fd1f06-kube-api-access-xfxgj" pod "bdc73375-64fa-42d8-8ba8-23a936fd1f06" (UID: "bdc73375-64fa-42d8-8ba8-23a936fd1f06"). InnerVolumeSpecName "kube-api-access-xfxgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:13.995336 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.995045 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdc73375-64fa-42d8-8ba8-23a936fd1f06-istio-token" pod "bdc73375-64fa-42d8-8ba8-23a936fd1f06" (UID: "bdc73375-64fa-42d8-8ba8-23a936fd1f06"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:13.995336 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:13.995139 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdc73375-64fa-42d8-8ba8-23a936fd1f06-istio-envoy" pod "bdc73375-64fa-42d8-8ba8-23a936fd1f06" (UID: "bdc73375-64fa-42d8-8ba8-23a936fd1f06"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:14.024955 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.024555 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" Nov 03 22:51:14.076420 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.076392 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdc73375-64fa-42d8-8ba8-23a936fd1f06-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:14.076420 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.076421 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdc73375-64fa-42d8-8ba8-23a936fd1f06-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:14.076629 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.076438 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdc73375-64fa-42d8-8ba8-23a936fd1f06-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:14.076629 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.076451 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdc73375-64fa-42d8-8ba8-23a936fd1f06-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:14.076629 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.076464 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdc73375-64fa-42d8-8ba8-23a936fd1f06-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:14.076629 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.076480 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdc73375-64fa-42d8-8ba8-23a936fd1f06-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:14.076629 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.076493 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdc73375-64fa-42d8-8ba8-23a936fd1f06-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:14.076629 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.076507 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdc73375-64fa-42d8-8ba8-23a936fd1f06-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:14.076629 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.076521 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfxgj\" (UniqueName: \"kubernetes.io/projected/bdc73375-64fa-42d8-8ba8-23a936fd1f06-kube-api-access-xfxgj\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:14.090508 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.090479 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" Nov 03 22:51:14.094672 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.094642 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" Nov 03 22:51:14.134924 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.134887 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" event={"ID":"b0aafab2-abe8-4dd8-81ea-71278482df2b","Type":"ContainerDied","Data":"101c2806e8da948f7e83a9e4eca04fe0b31f172ffb739db820d609ee0b3f96bb"} Nov 03 22:51:14.135083 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.134953 5196 scope.go:122] "RemoveContainer" containerID="2103d3f8cb26b75106b0ad40d3b841a5c8c60ba6adf5bd50917c1b9551ed594b" Nov 03 22:51:14.135183 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.135130 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj" Nov 03 22:51:14.138088 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.138062 5196 generic.go:356] "Generic (PLEG): container finished" podID="03244edf-87f5-4237-be87-8a523c4d96b3" containerID="48aff0ee2038482766ac246ed8191f082ac6300f2908702843d62a0ed2e9274d" exitCode=0 Nov 03 22:51:14.138206 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.138153 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" event={"ID":"03244edf-87f5-4237-be87-8a523c4d96b3","Type":"ContainerDied","Data":"48aff0ee2038482766ac246ed8191f082ac6300f2908702843d62a0ed2e9274d"} Nov 03 22:51:14.138206 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.138201 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" event={"ID":"03244edf-87f5-4237-be87-8a523c4d96b3","Type":"ContainerDied","Data":"ed9329b89804d5084dfb863d140e2495666945bcf11b1a04ee4c1879440e4820"} Nov 03 22:51:14.138335 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.138284 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk" Nov 03 22:51:14.141449 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.141424 5196 generic.go:356] "Generic (PLEG): container finished" podID="bdc73375-64fa-42d8-8ba8-23a936fd1f06" containerID="6c5a5d87185c8163c7d8735001be10125282042d09f99a6850bd6d1e40e77b97" exitCode=0 Nov 03 22:51:14.141545 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.141468 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" Nov 03 22:51:14.141673 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.141615 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" event={"ID":"bdc73375-64fa-42d8-8ba8-23a936fd1f06","Type":"ContainerDied","Data":"6c5a5d87185c8163c7d8735001be10125282042d09f99a6850bd6d1e40e77b97"} Nov 03 22:51:14.141803 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.141785 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv" event={"ID":"bdc73375-64fa-42d8-8ba8-23a936fd1f06","Type":"ContainerDied","Data":"60ba4966aa1aa33d56d2c654df7f58c9d590a3d4e5f33ffe7a9edd08f6bed3df"} Nov 03 22:51:14.145539 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.145513 5196 generic.go:356] "Generic (PLEG): container finished" podID="cea0ff59-b9f4-49e6-995e-0c568ca8dd9c" containerID="e3afd01a447b93d65eacbf6d57ae5355d38e6eab04b6cedc9423f275527b2770" exitCode=0 Nov 03 22:51:14.145646 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.145562 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" Nov 03 22:51:14.145715 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.145662 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" event={"ID":"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c","Type":"ContainerDied","Data":"e3afd01a447b93d65eacbf6d57ae5355d38e6eab04b6cedc9423f275527b2770"} Nov 03 22:51:14.145715 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.145703 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h" event={"ID":"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c","Type":"ContainerDied","Data":"e5e1b4aa7a939f0a1ac0312f1c548fb58459ea3ea6c8a96862ca39b000436269"} Nov 03 22:51:14.147375 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.147355 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" event={"ID":"5ee2347e-d2c8-4319-a5f8-f926e83ae1c6","Type":"ContainerDied","Data":"3fb06711fab6c410a8e31c049e8aaf028eaca5026adc4d55d2eadff0f797391b"} Nov 03 22:51:14.147481 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.147441 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq" Nov 03 22:51:14.151625 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.150396 5196 scope.go:122] "RemoveContainer" containerID="48aff0ee2038482766ac246ed8191f082ac6300f2908702843d62a0ed2e9274d" Nov 03 22:51:14.154620 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.152802 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" event={"ID":"cacf0851-d190-4ef3-8df0-e181b4444430","Type":"ContainerDied","Data":"7a9cd663f8332eb7a691ecfeb05dc9dd52fe4e18e8833e251f433f03c33ce7d5"} Nov 03 22:51:14.154620 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.153130 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g" Nov 03 22:51:14.169339 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.169314 5196 generic.go:356] "Generic (PLEG): container finished" podID="c9bf76ea-2e4e-46ee-9c23-3f084e9f8475" containerID="9a957a86b24bfae22736f54372a25239a3d03fcb45f7ce9cf94a7eced1aeb886" exitCode=0 Nov 03 22:51:14.169563 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.169542 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" event={"ID":"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475","Type":"ContainerDied","Data":"9a957a86b24bfae22736f54372a25239a3d03fcb45f7ce9cf94a7eced1aeb886"} Nov 03 22:51:14.169714 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.169697 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" event={"ID":"c9bf76ea-2e4e-46ee-9c23-3f084e9f8475","Type":"ContainerDied","Data":"4d1ee345ed8de9c7c2418920c289078f4c491a2118eaa04473b1c1012232ca33"} Nov 03 22:51:14.169948 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.169929 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8" Nov 03 22:51:14.174956 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.174845 5196 generic.go:356] "Generic (PLEG): container finished" podID="3a1c6229-58c7-412d-b67d-9a31e460a59c" containerID="ca502e0ebe562a6264e655b1f9b99bdef227a9ffd7b50f3623e30c3a9a273f13" exitCode=0 Nov 03 22:51:14.174956 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.174919 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" event={"ID":"3a1c6229-58c7-412d-b67d-9a31e460a59c","Type":"ContainerDied","Data":"ca502e0ebe562a6264e655b1f9b99bdef227a9ffd7b50f3623e30c3a9a273f13"} Nov 03 22:51:14.175078 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.174978 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" event={"ID":"3a1c6229-58c7-412d-b67d-9a31e460a59c","Type":"ContainerDied","Data":"d2efb22d25e904c3dfc9cdcde5ff66e5ebba8bdf35fb429f623ad393d6a6c31e"} Nov 03 22:51:14.175078 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.175072 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm" Nov 03 22:51:14.181561 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.181525 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-workload-socket\") pod \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\" (UID: \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\") " Nov 03 22:51:14.181666 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.181579 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03244edf-87f5-4237-be87-8a523c4d96b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03244edf-87f5-4237-be87-8a523c4d96b3-istio-envoy\") pod \"03244edf-87f5-4237-be87-8a523c4d96b3\" (UID: \"03244edf-87f5-4237-be87-8a523c4d96b3\") " Nov 03 22:51:14.181666 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.181629 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-kube-api-access-5sprd\" (UniqueName: \"kubernetes.io/projected/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-kube-api-access-5sprd\") pod \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\" (UID: \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\") " Nov 03 22:51:14.181751 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.181677 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-credential-socket\") pod \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\" (UID: \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\") " Nov 03 22:51:14.181751 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.181711 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03244edf-87f5-4237-be87-8a523c4d96b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03244edf-87f5-4237-be87-8a523c4d96b3-istiod-ca-cert\") pod \"03244edf-87f5-4237-be87-8a523c4d96b3\" (UID: \"03244edf-87f5-4237-be87-8a523c4d96b3\") " Nov 03 22:51:14.181822 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.181747 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a1c6229-58c7-412d-b67d-9a31e460a59c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a1c6229-58c7-412d-b67d-9a31e460a59c-istio-envoy\") pod \"3a1c6229-58c7-412d-b67d-9a31e460a59c\" (UID: \"3a1c6229-58c7-412d-b67d-9a31e460a59c\") " Nov 03 22:51:14.181822 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.181806 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-istiod-ca-cert\") pod \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\" (UID: \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\") " Nov 03 22:51:14.182657 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.181843 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a1c6229-58c7-412d-b67d-9a31e460a59c-istio-token\" (UniqueName: \"kubernetes.io/projected/3a1c6229-58c7-412d-b67d-9a31e460a59c-istio-token\") pod \"3a1c6229-58c7-412d-b67d-9a31e460a59c\" (UID: \"3a1c6229-58c7-412d-b67d-9a31e460a59c\") " Nov 03 22:51:14.182657 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.181925 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03244edf-87f5-4237-be87-8a523c4d96b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03244edf-87f5-4237-be87-8a523c4d96b3-workload-socket\") pod \"03244edf-87f5-4237-be87-8a523c4d96b3\" (UID: \"03244edf-87f5-4237-be87-8a523c4d96b3\") " Nov 03 22:51:14.182657 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.181981 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a1c6229-58c7-412d-b67d-9a31e460a59c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a1c6229-58c7-412d-b67d-9a31e460a59c-istio-podinfo\") pod \"3a1c6229-58c7-412d-b67d-9a31e460a59c\" (UID: \"3a1c6229-58c7-412d-b67d-9a31e460a59c\") " Nov 03 22:51:14.182657 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.182019 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a1c6229-58c7-412d-b67d-9a31e460a59c-kube-api-access-nlv9m\" (UniqueName: \"kubernetes.io/projected/3a1c6229-58c7-412d-b67d-9a31e460a59c-kube-api-access-nlv9m\") pod \"3a1c6229-58c7-412d-b67d-9a31e460a59c\" (UID: \"3a1c6229-58c7-412d-b67d-9a31e460a59c\") " Nov 03 22:51:14.182657 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.182079 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-istio-podinfo\") pod \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\" (UID: \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\") " Nov 03 22:51:14.182657 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.182111 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03244edf-87f5-4237-be87-8a523c4d96b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03244edf-87f5-4237-be87-8a523c4d96b3-istio-podinfo\") pod \"03244edf-87f5-4237-be87-8a523c4d96b3\" (UID: \"03244edf-87f5-4237-be87-8a523c4d96b3\") " Nov 03 22:51:14.182657 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.182153 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03244edf-87f5-4237-be87-8a523c4d96b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03244edf-87f5-4237-be87-8a523c4d96b3-credential-socket\") pod \"03244edf-87f5-4237-be87-8a523c4d96b3\" (UID: \"03244edf-87f5-4237-be87-8a523c4d96b3\") " Nov 03 22:51:14.182657 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.182180 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03244edf-87f5-4237-be87-8a523c4d96b3-istio-token\" (UniqueName: \"kubernetes.io/projected/03244edf-87f5-4237-be87-8a523c4d96b3-istio-token\") pod \"03244edf-87f5-4237-be87-8a523c4d96b3\" (UID: \"03244edf-87f5-4237-be87-8a523c4d96b3\") " Nov 03 22:51:14.182657 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.182227 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a1c6229-58c7-412d-b67d-9a31e460a59c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a1c6229-58c7-412d-b67d-9a31e460a59c-istiod-ca-cert\") pod \"3a1c6229-58c7-412d-b67d-9a31e460a59c\" (UID: \"3a1c6229-58c7-412d-b67d-9a31e460a59c\") " Nov 03 22:51:14.182657 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.182265 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a1c6229-58c7-412d-b67d-9a31e460a59c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a1c6229-58c7-412d-b67d-9a31e460a59c-workload-certs\") pod \"3a1c6229-58c7-412d-b67d-9a31e460a59c\" (UID: \"3a1c6229-58c7-412d-b67d-9a31e460a59c\") " Nov 03 22:51:14.182657 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.182307 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03244edf-87f5-4237-be87-8a523c4d96b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03244edf-87f5-4237-be87-8a523c4d96b3-workload-certs\") pod \"03244edf-87f5-4237-be87-8a523c4d96b3\" (UID: \"03244edf-87f5-4237-be87-8a523c4d96b3\") " Nov 03 22:51:14.182657 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.182366 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a1c6229-58c7-412d-b67d-9a31e460a59c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a1c6229-58c7-412d-b67d-9a31e460a59c-istio-data\") pod \"3a1c6229-58c7-412d-b67d-9a31e460a59c\" (UID: \"3a1c6229-58c7-412d-b67d-9a31e460a59c\") " Nov 03 22:51:14.182657 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.182396 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a1c6229-58c7-412d-b67d-9a31e460a59c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1c6229-58c7-412d-b67d-9a31e460a59c-workload-socket\") pod \"3a1c6229-58c7-412d-b67d-9a31e460a59c\" (UID: \"3a1c6229-58c7-412d-b67d-9a31e460a59c\") " Nov 03 22:51:14.182657 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.182444 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-workload-certs\") pod \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\" (UID: \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\") " Nov 03 22:51:14.182657 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.182479 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-istio-envoy\") pod \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\" (UID: \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\") " Nov 03 22:51:14.182657 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.182577 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-istio-token\" (UniqueName: \"kubernetes.io/projected/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-istio-token\") pod \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\" (UID: \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\") " Nov 03 22:51:14.182657 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.182616 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03244edf-87f5-4237-be87-8a523c4d96b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03244edf-87f5-4237-be87-8a523c4d96b3-istio-data\") pod \"03244edf-87f5-4237-be87-8a523c4d96b3\" (UID: \"03244edf-87f5-4237-be87-8a523c4d96b3\") " Nov 03 22:51:14.183199 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.182740 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-istio-data\") pod \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\" (UID: \"cea0ff59-b9f4-49e6-995e-0c568ca8dd9c\") " Nov 03 22:51:14.183199 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.182772 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a1c6229-58c7-412d-b67d-9a31e460a59c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1c6229-58c7-412d-b67d-9a31e460a59c-credential-socket\") pod \"3a1c6229-58c7-412d-b67d-9a31e460a59c\" (UID: \"3a1c6229-58c7-412d-b67d-9a31e460a59c\") " Nov 03 22:51:14.183199 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.182818 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03244edf-87f5-4237-be87-8a523c4d96b3-kube-api-access-fd76q\" (UniqueName: \"kubernetes.io/projected/03244edf-87f5-4237-be87-8a523c4d96b3-kube-api-access-fd76q\") pod \"03244edf-87f5-4237-be87-8a523c4d96b3\" (UID: \"03244edf-87f5-4237-be87-8a523c4d96b3\") " Nov 03 22:51:14.184982 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.184728 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03244edf-87f5-4237-be87-8a523c4d96b3-workload-socket" pod "03244edf-87f5-4237-be87-8a523c4d96b3" (UID: "03244edf-87f5-4237-be87-8a523c4d96b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:14.185514 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.185229 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-istiod-ca-cert" pod "cea0ff59-b9f4-49e6-995e-0c568ca8dd9c" (UID: "cea0ff59-b9f4-49e6-995e-0c568ca8dd9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:14.187810 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.187787 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a1c6229-58c7-412d-b67d-9a31e460a59c-workload-socket" pod "3a1c6229-58c7-412d-b67d-9a31e460a59c" (UID: "3a1c6229-58c7-412d-b67d-9a31e460a59c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:14.191979 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.191951 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03244edf-87f5-4237-be87-8a523c4d96b3-credential-socket" pod "03244edf-87f5-4237-be87-8a523c4d96b3" (UID: "03244edf-87f5-4237-be87-8a523c4d96b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:14.193213 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.193114 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-workload-certs" pod "cea0ff59-b9f4-49e6-995e-0c568ca8dd9c" (UID: "cea0ff59-b9f4-49e6-995e-0c568ca8dd9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:14.193551 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.193458 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a1c6229-58c7-412d-b67d-9a31e460a59c-credential-socket" pod "3a1c6229-58c7-412d-b67d-9a31e460a59c" (UID: "3a1c6229-58c7-412d-b67d-9a31e460a59c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:14.193746 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.193723 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03244edf-87f5-4237-be87-8a523c4d96b3-workload-certs" pod "03244edf-87f5-4237-be87-8a523c4d96b3" (UID: "03244edf-87f5-4237-be87-8a523c4d96b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:14.193962 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.193885 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a1c6229-58c7-412d-b67d-9a31e460a59c-workload-certs" pod "3a1c6229-58c7-412d-b67d-9a31e460a59c" (UID: "3a1c6229-58c7-412d-b67d-9a31e460a59c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:14.194054 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.194017 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03244edf-87f5-4237-be87-8a523c4d96b3-istio-data" pod "03244edf-87f5-4237-be87-8a523c4d96b3" (UID: "03244edf-87f5-4237-be87-8a523c4d96b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:14.194174 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.194131 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a1c6229-58c7-412d-b67d-9a31e460a59c-istio-data" pod "3a1c6229-58c7-412d-b67d-9a31e460a59c" (UID: "3a1c6229-58c7-412d-b67d-9a31e460a59c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:14.194394 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.194373 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a1c6229-58c7-412d-b67d-9a31e460a59c-istiod-ca-cert" pod "3a1c6229-58c7-412d-b67d-9a31e460a59c" (UID: "3a1c6229-58c7-412d-b67d-9a31e460a59c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:14.194658 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.194639 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03244edf-87f5-4237-be87-8a523c4d96b3-istiod-ca-cert" pod "03244edf-87f5-4237-be87-8a523c4d96b3" (UID: "03244edf-87f5-4237-be87-8a523c4d96b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:14.195286 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.195168 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-workload-socket" pod "cea0ff59-b9f4-49e6-995e-0c568ca8dd9c" (UID: "cea0ff59-b9f4-49e6-995e-0c568ca8dd9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:14.195286 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.195243 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-istio-data" pod "cea0ff59-b9f4-49e6-995e-0c568ca8dd9c" (UID: "cea0ff59-b9f4-49e6-995e-0c568ca8dd9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:14.195509 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.195422 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-credential-socket" pod "cea0ff59-b9f4-49e6-995e-0c568ca8dd9c" (UID: "cea0ff59-b9f4-49e6-995e-0c568ca8dd9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:14.201367 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.199186 5196 scope.go:122] "RemoveContainer" containerID="48aff0ee2038482766ac246ed8191f082ac6300f2908702843d62a0ed2e9274d" Nov 03 22:51:14.201367 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.199346 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03244edf-87f5-4237-be87-8a523c4d96b3-istio-podinfo" pod "03244edf-87f5-4237-be87-8a523c4d96b3" (UID: "03244edf-87f5-4237-be87-8a523c4d96b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:51:14.201367 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:51:14.200146 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48aff0ee2038482766ac246ed8191f082ac6300f2908702843d62a0ed2e9274d\": container with ID starting with 48aff0ee2038482766ac246ed8191f082ac6300f2908702843d62a0ed2e9274d not found: ID does not exist" containerID="48aff0ee2038482766ac246ed8191f082ac6300f2908702843d62a0ed2e9274d" Nov 03 22:51:14.203512 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.200545 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48aff0ee2038482766ac246ed8191f082ac6300f2908702843d62a0ed2e9274d"} err="failed to get container status \"48aff0ee2038482766ac246ed8191f082ac6300f2908702843d62a0ed2e9274d\": rpc error: code = NotFound desc = could not find container \"48aff0ee2038482766ac246ed8191f082ac6300f2908702843d62a0ed2e9274d\": container with ID starting with 48aff0ee2038482766ac246ed8191f082ac6300f2908702843d62a0ed2e9274d not found: ID does not exist" Nov 03 22:51:14.203512 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.202113 5196 scope.go:122] "RemoveContainer" containerID="6c5a5d87185c8163c7d8735001be10125282042d09f99a6850bd6d1e40e77b97" Nov 03 22:51:14.208634 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.207749 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-istio-token" pod "cea0ff59-b9f4-49e6-995e-0c568ca8dd9c" (UID: "cea0ff59-b9f4-49e6-995e-0c568ca8dd9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:14.208634 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.207798 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03244edf-87f5-4237-be87-8a523c4d96b3-kube-api-access-fd76q" pod "03244edf-87f5-4237-be87-8a523c4d96b3" (UID: "03244edf-87f5-4237-be87-8a523c4d96b3"). InnerVolumeSpecName "kube-api-access-fd76q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:14.208634 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.207819 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03244edf-87f5-4237-be87-8a523c4d96b3-istio-token" pod "03244edf-87f5-4237-be87-8a523c4d96b3" (UID: "03244edf-87f5-4237-be87-8a523c4d96b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:14.208634 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.207838 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a1c6229-58c7-412d-b67d-9a31e460a59c-istio-token" pod "3a1c6229-58c7-412d-b67d-9a31e460a59c" (UID: "3a1c6229-58c7-412d-b67d-9a31e460a59c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:14.208634 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.207878 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a1c6229-58c7-412d-b67d-9a31e460a59c-kube-api-access-nlv9m" pod "3a1c6229-58c7-412d-b67d-9a31e460a59c" (UID: "3a1c6229-58c7-412d-b67d-9a31e460a59c"). InnerVolumeSpecName "kube-api-access-nlv9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:14.208634 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.208049 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03244edf-87f5-4237-be87-8a523c4d96b3-istio-envoy" pod "03244edf-87f5-4237-be87-8a523c4d96b3" (UID: "03244edf-87f5-4237-be87-8a523c4d96b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:14.219350 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.219321 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-kube-api-access-5sprd" pod "cea0ff59-b9f4-49e6-995e-0c568ca8dd9c" (UID: "cea0ff59-b9f4-49e6-995e-0c568ca8dd9c"). InnerVolumeSpecName "kube-api-access-5sprd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:14.219449 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.219421 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a1c6229-58c7-412d-b67d-9a31e460a59c-istio-podinfo" pod "3a1c6229-58c7-412d-b67d-9a31e460a59c" (UID: "3a1c6229-58c7-412d-b67d-9a31e460a59c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:51:14.224496 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.224473 5196 scope.go:122] "RemoveContainer" containerID="6c5a5d87185c8163c7d8735001be10125282042d09f99a6850bd6d1e40e77b97" Nov 03 22:51:14.224895 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:51:14.224838 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c5a5d87185c8163c7d8735001be10125282042d09f99a6850bd6d1e40e77b97\": container with ID starting with 6c5a5d87185c8163c7d8735001be10125282042d09f99a6850bd6d1e40e77b97 not found: ID does not exist" containerID="6c5a5d87185c8163c7d8735001be10125282042d09f99a6850bd6d1e40e77b97" Nov 03 22:51:14.224985 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.224965 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c5a5d87185c8163c7d8735001be10125282042d09f99a6850bd6d1e40e77b97"} err="failed to get container status \"6c5a5d87185c8163c7d8735001be10125282042d09f99a6850bd6d1e40e77b97\": rpc error: code = NotFound desc = could not find container \"6c5a5d87185c8163c7d8735001be10125282042d09f99a6850bd6d1e40e77b97\": container with ID starting with 6c5a5d87185c8163c7d8735001be10125282042d09f99a6850bd6d1e40e77b97 not found: ID does not exist" Nov 03 22:51:14.225048 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.224990 5196 scope.go:122] "RemoveContainer" containerID="e3afd01a447b93d65eacbf6d57ae5355d38e6eab04b6cedc9423f275527b2770" Nov 03 22:51:14.229471 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.229424 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-istio-podinfo" pod "cea0ff59-b9f4-49e6-995e-0c568ca8dd9c" (UID: "cea0ff59-b9f4-49e6-995e-0c568ca8dd9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:51:14.248066 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.248026 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-istio-envoy" pod "cea0ff59-b9f4-49e6-995e-0c568ca8dd9c" (UID: "cea0ff59-b9f4-49e6-995e-0c568ca8dd9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:14.266011 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.265984 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a1c6229-58c7-412d-b67d-9a31e460a59c-istio-envoy" pod "3a1c6229-58c7-412d-b67d-9a31e460a59c" (UID: "3a1c6229-58c7-412d-b67d-9a31e460a59c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:14.270664 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.270611 5196 scope.go:122] "RemoveContainer" containerID="e3afd01a447b93d65eacbf6d57ae5355d38e6eab04b6cedc9423f275527b2770" Nov 03 22:51:14.271231 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:51:14.271089 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3afd01a447b93d65eacbf6d57ae5355d38e6eab04b6cedc9423f275527b2770\": container with ID starting with e3afd01a447b93d65eacbf6d57ae5355d38e6eab04b6cedc9423f275527b2770 not found: ID does not exist" containerID="e3afd01a447b93d65eacbf6d57ae5355d38e6eab04b6cedc9423f275527b2770" Nov 03 22:51:14.271231 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.271127 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3afd01a447b93d65eacbf6d57ae5355d38e6eab04b6cedc9423f275527b2770"} err="failed to get container status \"e3afd01a447b93d65eacbf6d57ae5355d38e6eab04b6cedc9423f275527b2770\": rpc error: code = NotFound desc = could not find container \"e3afd01a447b93d65eacbf6d57ae5355d38e6eab04b6cedc9423f275527b2770\": container with ID starting with e3afd01a447b93d65eacbf6d57ae5355d38e6eab04b6cedc9423f275527b2770 not found: ID does not exist" Nov 03 22:51:14.271231 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.271147 5196 scope.go:122] "RemoveContainer" containerID="fcb540b2641ca06fc9dbc1019e9965a288bbf186c22bbf5e1be3b12d40a55646" Nov 03 22:51:14.281243 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.281222 5196 scope.go:122] "RemoveContainer" containerID="edb6ad9aabcee8b78073613b96b985deef1a5a72c4d2e4b1672a97edb6406383" Nov 03 22:51:14.285069 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.285044 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03244edf-87f5-4237-be87-8a523c4d96b3-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:14.285171 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.285075 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:14.285171 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.285090 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1c6229-58c7-412d-b67d-9a31e460a59c-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:14.285171 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.285104 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd76q\" (UniqueName: \"kubernetes.io/projected/03244edf-87f5-4237-be87-8a523c4d96b3-kube-api-access-fd76q\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:14.285171 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.285120 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:14.285171 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.285133 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03244edf-87f5-4237-be87-8a523c4d96b3-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:14.285171 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.285147 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sprd\" (UniqueName: \"kubernetes.io/projected/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-kube-api-access-5sprd\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:14.285171 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.285160 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:14.285171 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.285173 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03244edf-87f5-4237-be87-8a523c4d96b3-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:14.285533 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.285187 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a1c6229-58c7-412d-b67d-9a31e460a59c-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:14.285533 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.285200 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:14.285533 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.285214 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a1c6229-58c7-412d-b67d-9a31e460a59c-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:14.285533 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.285227 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03244edf-87f5-4237-be87-8a523c4d96b3-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:14.285533 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.285243 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a1c6229-58c7-412d-b67d-9a31e460a59c-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:14.285533 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.285256 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlv9m\" (UniqueName: \"kubernetes.io/projected/3a1c6229-58c7-412d-b67d-9a31e460a59c-kube-api-access-nlv9m\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:14.285533 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.285270 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:14.285533 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.285283 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03244edf-87f5-4237-be87-8a523c4d96b3-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:14.285533 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.285296 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03244edf-87f5-4237-be87-8a523c4d96b3-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:14.285533 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.285309 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03244edf-87f5-4237-be87-8a523c4d96b3-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:14.285533 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.285322 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a1c6229-58c7-412d-b67d-9a31e460a59c-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:14.285533 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.285337 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a1c6229-58c7-412d-b67d-9a31e460a59c-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:14.285533 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.285352 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03244edf-87f5-4237-be87-8a523c4d96b3-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:14.285533 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.285365 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a1c6229-58c7-412d-b67d-9a31e460a59c-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:14.285533 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.285378 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1c6229-58c7-412d-b67d-9a31e460a59c-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:14.285533 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.285392 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:14.285533 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.285407 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:14.285533 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.285420 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:51:14.298212 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.298185 5196 scope.go:122] "RemoveContainer" containerID="9a957a86b24bfae22736f54372a25239a3d03fcb45f7ce9cf94a7eced1aeb886" Nov 03 22:51:14.312491 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.312468 5196 scope.go:122] "RemoveContainer" containerID="9a957a86b24bfae22736f54372a25239a3d03fcb45f7ce9cf94a7eced1aeb886" Nov 03 22:51:14.312887 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:51:14.312843 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a957a86b24bfae22736f54372a25239a3d03fcb45f7ce9cf94a7eced1aeb886\": container with ID starting with 9a957a86b24bfae22736f54372a25239a3d03fcb45f7ce9cf94a7eced1aeb886 not found: ID does not exist" containerID="9a957a86b24bfae22736f54372a25239a3d03fcb45f7ce9cf94a7eced1aeb886" Nov 03 22:51:14.312991 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.312898 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a957a86b24bfae22736f54372a25239a3d03fcb45f7ce9cf94a7eced1aeb886"} err="failed to get container status \"9a957a86b24bfae22736f54372a25239a3d03fcb45f7ce9cf94a7eced1aeb886\": rpc error: code = NotFound desc = could not find container \"9a957a86b24bfae22736f54372a25239a3d03fcb45f7ce9cf94a7eced1aeb886\": container with ID starting with 9a957a86b24bfae22736f54372a25239a3d03fcb45f7ce9cf94a7eced1aeb886 not found: ID does not exist" Nov 03 22:51:14.312991 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.312933 5196 scope.go:122] "RemoveContainer" containerID="ca502e0ebe562a6264e655b1f9b99bdef227a9ffd7b50f3623e30c3a9a273f13" Nov 03 22:51:14.328679 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.328656 5196 scope.go:122] "RemoveContainer" containerID="ca502e0ebe562a6264e655b1f9b99bdef227a9ffd7b50f3623e30c3a9a273f13" Nov 03 22:51:14.329042 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:51:14.329014 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca502e0ebe562a6264e655b1f9b99bdef227a9ffd7b50f3623e30c3a9a273f13\": container with ID starting with ca502e0ebe562a6264e655b1f9b99bdef227a9ffd7b50f3623e30c3a9a273f13 not found: ID does not exist" containerID="ca502e0ebe562a6264e655b1f9b99bdef227a9ffd7b50f3623e30c3a9a273f13" Nov 03 22:51:14.329126 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.329050 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca502e0ebe562a6264e655b1f9b99bdef227a9ffd7b50f3623e30c3a9a273f13"} err="failed to get container status \"ca502e0ebe562a6264e655b1f9b99bdef227a9ffd7b50f3623e30c3a9a273f13\": rpc error: code = NotFound desc = could not find container \"ca502e0ebe562a6264e655b1f9b99bdef227a9ffd7b50f3623e30c3a9a273f13\": container with ID starting with ca502e0ebe562a6264e655b1f9b99bdef227a9ffd7b50f3623e30c3a9a273f13 not found: ID does not exist" Nov 03 22:51:14.736794 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.736692 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj"] Nov 03 22:51:14.740738 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.740710 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-nc2kj"] Nov 03 22:51:14.817356 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.817325 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0aafab2-abe8-4dd8-81ea-71278482df2b" path="/var/lib/kubelet/pods/b0aafab2-abe8-4dd8-81ea-71278482df2b/volumes" Nov 03 22:51:14.851660 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.851631 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8"] Nov 03 22:51:14.864908 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.864880 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-54fm8"] Nov 03 22:51:14.978606 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.978559 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm"] Nov 03 22:51:14.991980 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:14.991880 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-6jvmm"] Nov 03 22:51:15.101045 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:15.101005 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv"] Nov 03 22:51:15.106080 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:15.106047 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-g6qdv"] Nov 03 22:51:15.224151 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:15.224115 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g"] Nov 03 22:51:15.230949 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:15.230924 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-28w9g"] Nov 03 22:51:15.341874 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:15.341830 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h"] Nov 03 22:51:15.345345 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:15.345319 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-9p25h"] Nov 03 22:51:15.456344 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:15.456309 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq"] Nov 03 22:51:15.460275 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:15.460249 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7540da-openshift-default-bb989bcf9-2bdcq"] Nov 03 22:51:15.573199 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:15.573160 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk"] Nov 03 22:51:15.579087 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:15.579052 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-cbd26c-openshift-default-586cfdcc75-qk9xk"] Nov 03 22:51:15.651306 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:15.651217 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:16.816321 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:16.816287 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03244edf-87f5-4237-be87-8a523c4d96b3" path="/var/lib/kubelet/pods/03244edf-87f5-4237-be87-8a523c4d96b3/volumes" Nov 03 22:51:16.817043 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:16.817022 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a1c6229-58c7-412d-b67d-9a31e460a59c" path="/var/lib/kubelet/pods/3a1c6229-58c7-412d-b67d-9a31e460a59c/volumes" Nov 03 22:51:16.817540 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:16.817519 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ee2347e-d2c8-4319-a5f8-f926e83ae1c6" path="/var/lib/kubelet/pods/5ee2347e-d2c8-4319-a5f8-f926e83ae1c6/volumes" Nov 03 22:51:16.818282 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:16.818258 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdc73375-64fa-42d8-8ba8-23a936fd1f06" path="/var/lib/kubelet/pods/bdc73375-64fa-42d8-8ba8-23a936fd1f06/volumes" Nov 03 22:51:16.818749 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:16.818729 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9bf76ea-2e4e-46ee-9c23-3f084e9f8475" path="/var/lib/kubelet/pods/c9bf76ea-2e4e-46ee-9c23-3f084e9f8475/volumes" Nov 03 22:51:16.819239 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:16.819220 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cacf0851-d190-4ef3-8df0-e181b4444430" path="/var/lib/kubelet/pods/cacf0851-d190-4ef3-8df0-e181b4444430/volumes" Nov 03 22:51:16.819985 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:16.819966 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cea0ff59-b9f4-49e6-995e-0c568ca8dd9c" path="/var/lib/kubelet/pods/cea0ff59-b9f4-49e6-995e-0c568ca8dd9c/volumes" Nov 03 22:51:20.690918 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:20.690884 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:22.331606 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:22.331561 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:22.777461 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:22.777428 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:23.905134 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:23.905101 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:24.122199 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:24.122168 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:25.711486 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:25.711455 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:25.919796 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:51:25.919758 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:51:27.997486 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:27.997452 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:28.153748 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:28.153708 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:28.361685 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:28.361634 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:30.904763 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:30.904669 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:31.054844 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:31.054800 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:31.217672 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:31.217539 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:31.377588 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:31.377550 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:31.530725 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:31.530637 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:36.116991 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:36.116956 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:40.865956 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:40.865919 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:45.345230 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.345138 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn"] Nov 03 22:51:45.345972 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.345949 5196 cpu_manager.go:414] "containerMap: removing container" podUID="bdc73375-64fa-42d8-8ba8-23a936fd1f06" containerName="istio-proxy" Nov 03 22:51:45.345972 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.345972 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdc73375-64fa-42d8-8ba8-23a936fd1f06" containerName="istio-proxy" Nov 03 22:51:45.346153 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.346004 5196 cpu_manager.go:414] "containerMap: removing container" podUID="ae839647-c999-4479-9174-3e4c389d24b8" containerName="istio-proxy" Nov 03 22:51:45.346153 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.346012 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae839647-c999-4479-9174-3e4c389d24b8" containerName="istio-proxy" Nov 03 22:51:45.346153 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.346024 5196 cpu_manager.go:414] "containerMap: removing container" podUID="cacf0851-d190-4ef3-8df0-e181b4444430" containerName="istio-proxy" Nov 03 22:51:45.346153 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.346031 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cacf0851-d190-4ef3-8df0-e181b4444430" containerName="istio-proxy" Nov 03 22:51:45.346153 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.346047 5196 cpu_manager.go:414] "containerMap: removing container" podUID="03244edf-87f5-4237-be87-8a523c4d96b3" containerName="istio-proxy" Nov 03 22:51:45.346153 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.346055 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03244edf-87f5-4237-be87-8a523c4d96b3" containerName="istio-proxy" Nov 03 22:51:45.346153 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.346068 5196 cpu_manager.go:414] "containerMap: removing container" podUID="ae839647-c999-4479-9174-3e4c389d24b8" containerName="istio-proxy" Nov 03 22:51:45.346153 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.346075 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae839647-c999-4479-9174-3e4c389d24b8" containerName="istio-proxy" Nov 03 22:51:45.346153 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.346093 5196 cpu_manager.go:414] "containerMap: removing container" podUID="c9bf76ea-2e4e-46ee-9c23-3f084e9f8475" containerName="istio-proxy" Nov 03 22:51:45.346153 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.346100 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9bf76ea-2e4e-46ee-9c23-3f084e9f8475" containerName="istio-proxy" Nov 03 22:51:45.346153 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.346116 5196 cpu_manager.go:414] "containerMap: removing container" podUID="5ee2347e-d2c8-4319-a5f8-f926e83ae1c6" containerName="istio-proxy" Nov 03 22:51:45.346153 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.346123 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ee2347e-d2c8-4319-a5f8-f926e83ae1c6" containerName="istio-proxy" Nov 03 22:51:45.346153 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.346137 5196 cpu_manager.go:414] "containerMap: removing container" podUID="cea0ff59-b9f4-49e6-995e-0c568ca8dd9c" containerName="istio-proxy" Nov 03 22:51:45.346153 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.346144 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cea0ff59-b9f4-49e6-995e-0c568ca8dd9c" containerName="istio-proxy" Nov 03 22:51:45.346153 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.346157 5196 cpu_manager.go:414] "containerMap: removing container" podUID="b0aafab2-abe8-4dd8-81ea-71278482df2b" containerName="istio-proxy" Nov 03 22:51:45.346928 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.346164 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0aafab2-abe8-4dd8-81ea-71278482df2b" containerName="istio-proxy" Nov 03 22:51:45.346928 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.346187 5196 cpu_manager.go:414] "containerMap: removing container" podUID="3a1c6229-58c7-412d-b67d-9a31e460a59c" containerName="istio-proxy" Nov 03 22:51:45.346928 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.346195 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a1c6229-58c7-412d-b67d-9a31e460a59c" containerName="istio-proxy" Nov 03 22:51:45.346928 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.346321 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0aafab2-abe8-4dd8-81ea-71278482df2b" containerName="istio-proxy" Nov 03 22:51:45.346928 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.346335 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="cacf0851-d190-4ef3-8df0-e181b4444430" containerName="istio-proxy" Nov 03 22:51:45.346928 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.346346 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a1c6229-58c7-412d-b67d-9a31e460a59c" containerName="istio-proxy" Nov 03 22:51:45.346928 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.346358 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="03244edf-87f5-4237-be87-8a523c4d96b3" containerName="istio-proxy" Nov 03 22:51:45.346928 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.346368 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae839647-c999-4479-9174-3e4c389d24b8" containerName="istio-proxy" Nov 03 22:51:45.346928 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.346378 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae839647-c999-4479-9174-3e4c389d24b8" containerName="istio-proxy" Nov 03 22:51:45.346928 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.346387 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9bf76ea-2e4e-46ee-9c23-3f084e9f8475" containerName="istio-proxy" Nov 03 22:51:45.346928 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.346396 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdc73375-64fa-42d8-8ba8-23a936fd1f06" containerName="istio-proxy" Nov 03 22:51:45.346928 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.346404 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ee2347e-d2c8-4319-a5f8-f926e83ae1c6" containerName="istio-proxy" Nov 03 22:51:45.346928 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.346416 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="cea0ff59-b9f4-49e6-995e-0c568ca8dd9c" containerName="istio-proxy" Nov 03 22:51:45.350222 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.350200 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn" Nov 03 22:51:45.403695 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.403675 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-de5715-serving-cert\"" Nov 03 22:51:45.442250 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.442215 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80c67970-a697-4de5-bc1e-d925bc67d7f5-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn\" (UID: \"80c67970-a697-4de5-bc1e-d925bc67d7f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn" Nov 03 22:51:45.442467 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.442445 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmmxl\" (UniqueName: \"kubernetes.io/projected/80c67970-a697-4de5-bc1e-d925bc67d7f5-kube-api-access-dmmxl\") pod \"maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn\" (UID: \"80c67970-a697-4de5-bc1e-d925bc67d7f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn" Nov 03 22:51:45.442606 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.442586 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80c67970-a697-4de5-bc1e-d925bc67d7f5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn\" (UID: \"80c67970-a697-4de5-bc1e-d925bc67d7f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn" Nov 03 22:51:45.442663 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.442648 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80c67970-a697-4de5-bc1e-d925bc67d7f5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn\" (UID: \"80c67970-a697-4de5-bc1e-d925bc67d7f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn" Nov 03 22:51:45.457564 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.457537 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn"] Nov 03 22:51:45.540282 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.540253 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:45.543676 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.543653 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmmxl\" (UniqueName: \"kubernetes.io/projected/80c67970-a697-4de5-bc1e-d925bc67d7f5-kube-api-access-dmmxl\") pod \"maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn\" (UID: \"80c67970-a697-4de5-bc1e-d925bc67d7f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn" Nov 03 22:51:45.543791 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.543722 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80c67970-a697-4de5-bc1e-d925bc67d7f5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn\" (UID: \"80c67970-a697-4de5-bc1e-d925bc67d7f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn" Nov 03 22:51:45.543791 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.543761 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80c67970-a697-4de5-bc1e-d925bc67d7f5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn\" (UID: \"80c67970-a697-4de5-bc1e-d925bc67d7f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn" Nov 03 22:51:45.543920 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.543821 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80c67970-a697-4de5-bc1e-d925bc67d7f5-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn\" (UID: \"80c67970-a697-4de5-bc1e-d925bc67d7f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn" Nov 03 22:51:45.543980 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:51:45.543956 5196 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-de5715-serving-cert: secret "maas-api-e2e-shared-a-w4-de5715-serving-cert" not found Nov 03 22:51:45.544043 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:51:45.544030 5196 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80c67970-a697-4de5-bc1e-d925bc67d7f5-maas-api-tls podName:80c67970-a697-4de5-bc1e-d925bc67d7f5 nodeName:}" failed. No retries permitted until 2026-11-03 22:51:46.04400651 +0000 UTC m=+1735.353565781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/80c67970-a697-4de5-bc1e-d925bc67d7f5-maas-api-tls") pod "maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn" (UID: "80c67970-a697-4de5-bc1e-d925bc67d7f5") : secret "maas-api-e2e-shared-a-w4-de5715-serving-cert" not found Nov 03 22:51:45.544418 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.544398 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80c67970-a697-4de5-bc1e-d925bc67d7f5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn\" (UID: \"80c67970-a697-4de5-bc1e-d925bc67d7f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn" Nov 03 22:51:45.546420 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.546397 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80c67970-a697-4de5-bc1e-d925bc67d7f5-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn\" (UID: \"80c67970-a697-4de5-bc1e-d925bc67d7f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn" Nov 03 22:51:45.626158 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.626082 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmmxl\" (UniqueName: \"kubernetes.io/projected/80c67970-a697-4de5-bc1e-d925bc67d7f5-kube-api-access-dmmxl\") pod \"maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn\" (UID: \"80c67970-a697-4de5-bc1e-d925bc67d7f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn" Nov 03 22:51:45.696960 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.696912 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:45.842766 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:45.842735 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:46.051518 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:46.051421 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80c67970-a697-4de5-bc1e-d925bc67d7f5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn\" (UID: \"80c67970-a697-4de5-bc1e-d925bc67d7f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn" Nov 03 22:51:46.054080 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:46.054053 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80c67970-a697-4de5-bc1e-d925bc67d7f5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn\" (UID: \"80c67970-a697-4de5-bc1e-d925bc67d7f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn" Nov 03 22:51:46.263567 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:46.263534 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn" Nov 03 22:51:46.644367 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:46.644343 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn"] Nov 03 22:51:46.645094 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:51:46.645069 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80c67970_a697_4de5_bc1e_d925bc67d7f5.slice/crio-04080e76f12e52b594433c93b2799e2fe247e62aaffb349ccc47662ae403c546 WatchSource:0}: Error finding container 04080e76f12e52b594433c93b2799e2fe247e62aaffb349ccc47662ae403c546: Status 404 returned error can't find the container with id 04080e76f12e52b594433c93b2799e2fe247e62aaffb349ccc47662ae403c546 Nov 03 22:51:47.423655 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:47.423609 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn" event={"ID":"80c67970-a697-4de5-bc1e-d925bc67d7f5","Type":"ContainerStarted","Data":"04080e76f12e52b594433c93b2799e2fe247e62aaffb349ccc47662ae403c546"} Nov 03 22:51:48.435229 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:48.435183 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn" event={"ID":"80c67970-a697-4de5-bc1e-d925bc67d7f5","Type":"ContainerStarted","Data":"b74a00e279182b1693a5f5720705d2e86d6154381590565c3dbcbcf308e8e3e5"} Nov 03 22:51:48.435672 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:48.435261 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn" Nov 03 22:51:48.556954 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:48.556895 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn" podStartSLOduration=2.7220363130000003 podStartE2EDuration="3.556845578s" totalImagesPullingTime="834.809265ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:51:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:51:48.554296254 +0000 UTC m=+1737.863855559" watchObservedRunningTime="2026-11-03 22:51:48.556845578 +0000 UTC m=+1737.866404888" Nov 03 22:51:50.628068 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:50.628031 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:54.468753 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:54.468721 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn" Nov 03 22:51:55.357686 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:51:55.357628 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:51:55.918535 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:51:55.918495 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:52:00.109297 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:00.109262 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:04.805167 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:04.805134 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:06.230850 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:06.230806 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:07.505275 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:07.505242 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:08.021852 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:08.021816 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:08.193031 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:08.193001 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:08.348374 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:08.348345 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:09.560158 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:09.560124 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:11.700447 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:11.700412 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:14.335219 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:14.335187 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:15.891769 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:15.891726 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:16.568652 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:16.568617 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:16.852180 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:16.852154 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:19.444836 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:19.444800 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:20.840291 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:20.840255 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:21.021103 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:21.021067 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:21.321759 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:21.321688 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:24.731542 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:24.731511 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:24.874847 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:24.874809 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:25.017366 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:25.017274 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:25.918422 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:52:25.918380 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:52:28.676525 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:28.676490 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:28.836068 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:28.836032 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:29.985531 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:29.985499 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:30.360194 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:30.360169 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:32.205363 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:32.205327 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:32.701274 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:32.701229 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk"] Nov 03 22:52:32.705839 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:32.705803 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:52:32.771619 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:32.771589 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-26444f-openshift-default-dockercfg-kdf2n\"" Nov 03 22:52:32.797832 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:32.797790 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:32.828137 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:32.828102 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk"] Nov 03 22:52:32.909937 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:32.909829 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0474bb77-105a-4009-9f7e-2e19a4068fc9-istio-podinfo\") pod \"e2e-api-a-26444f-openshift-default-695897496f-mcdxk\" (UID: \"0474bb77-105a-4009-9f7e-2e19a4068fc9\") " pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:52:32.911060 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:32.911030 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0474bb77-105a-4009-9f7e-2e19a4068fc9-istio-data\") pod \"e2e-api-a-26444f-openshift-default-695897496f-mcdxk\" (UID: \"0474bb77-105a-4009-9f7e-2e19a4068fc9\") " pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:52:32.911270 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:32.911249 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0474bb77-105a-4009-9f7e-2e19a4068fc9-istiod-ca-cert\") pod \"e2e-api-a-26444f-openshift-default-695897496f-mcdxk\" (UID: \"0474bb77-105a-4009-9f7e-2e19a4068fc9\") " pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:52:32.911431 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:32.911412 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0474bb77-105a-4009-9f7e-2e19a4068fc9-workload-certs\") pod \"e2e-api-a-26444f-openshift-default-695897496f-mcdxk\" (UID: \"0474bb77-105a-4009-9f7e-2e19a4068fc9\") " pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:52:32.911557 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:32.911541 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0474bb77-105a-4009-9f7e-2e19a4068fc9-istio-token\") pod \"e2e-api-a-26444f-openshift-default-695897496f-mcdxk\" (UID: \"0474bb77-105a-4009-9f7e-2e19a4068fc9\") " pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:52:32.911717 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:32.911700 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0474bb77-105a-4009-9f7e-2e19a4068fc9-workload-socket\") pod \"e2e-api-a-26444f-openshift-default-695897496f-mcdxk\" (UID: \"0474bb77-105a-4009-9f7e-2e19a4068fc9\") " pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:52:32.911879 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:32.911844 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0474bb77-105a-4009-9f7e-2e19a4068fc9-istio-envoy\") pod \"e2e-api-a-26444f-openshift-default-695897496f-mcdxk\" (UID: \"0474bb77-105a-4009-9f7e-2e19a4068fc9\") " pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:52:32.912026 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:32.912008 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9cvn\" (UniqueName: \"kubernetes.io/projected/0474bb77-105a-4009-9f7e-2e19a4068fc9-kube-api-access-c9cvn\") pod \"e2e-api-a-26444f-openshift-default-695897496f-mcdxk\" (UID: \"0474bb77-105a-4009-9f7e-2e19a4068fc9\") " pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:52:32.912185 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:32.912168 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0474bb77-105a-4009-9f7e-2e19a4068fc9-credential-socket\") pod \"e2e-api-a-26444f-openshift-default-695897496f-mcdxk\" (UID: \"0474bb77-105a-4009-9f7e-2e19a4068fc9\") " pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:52:33.020014 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:33.019924 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0474bb77-105a-4009-9f7e-2e19a4068fc9-istio-podinfo\") pod \"e2e-api-a-26444f-openshift-default-695897496f-mcdxk\" (UID: \"0474bb77-105a-4009-9f7e-2e19a4068fc9\") " pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:52:33.021625 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:33.021595 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0474bb77-105a-4009-9f7e-2e19a4068fc9-istio-data\") pod \"e2e-api-a-26444f-openshift-default-695897496f-mcdxk\" (UID: \"0474bb77-105a-4009-9f7e-2e19a4068fc9\") " pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:52:33.021845 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:33.021826 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0474bb77-105a-4009-9f7e-2e19a4068fc9-istiod-ca-cert\") pod \"e2e-api-a-26444f-openshift-default-695897496f-mcdxk\" (UID: \"0474bb77-105a-4009-9f7e-2e19a4068fc9\") " pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:52:33.022015 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:33.021999 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0474bb77-105a-4009-9f7e-2e19a4068fc9-workload-certs\") pod \"e2e-api-a-26444f-openshift-default-695897496f-mcdxk\" (UID: \"0474bb77-105a-4009-9f7e-2e19a4068fc9\") " pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:52:33.036944 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:33.022500 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0474bb77-105a-4009-9f7e-2e19a4068fc9-istio-data\") pod \"e2e-api-a-26444f-openshift-default-695897496f-mcdxk\" (UID: \"0474bb77-105a-4009-9f7e-2e19a4068fc9\") " pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:52:33.037195 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:33.023247 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0474bb77-105a-4009-9f7e-2e19a4068fc9-istiod-ca-cert\") pod \"e2e-api-a-26444f-openshift-default-695897496f-mcdxk\" (UID: \"0474bb77-105a-4009-9f7e-2e19a4068fc9\") " pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:52:33.037368 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:33.023563 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0474bb77-105a-4009-9f7e-2e19a4068fc9-workload-certs\") pod \"e2e-api-a-26444f-openshift-default-695897496f-mcdxk\" (UID: \"0474bb77-105a-4009-9f7e-2e19a4068fc9\") " pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:52:33.037503 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:33.025510 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:33.038110 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:33.037070 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0474bb77-105a-4009-9f7e-2e19a4068fc9-istio-token\") pod \"e2e-api-a-26444f-openshift-default-695897496f-mcdxk\" (UID: \"0474bb77-105a-4009-9f7e-2e19a4068fc9\") " pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:52:33.038110 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:33.037758 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0474bb77-105a-4009-9f7e-2e19a4068fc9-workload-socket\") pod \"e2e-api-a-26444f-openshift-default-695897496f-mcdxk\" (UID: \"0474bb77-105a-4009-9f7e-2e19a4068fc9\") " pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:52:33.038110 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:33.037815 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0474bb77-105a-4009-9f7e-2e19a4068fc9-istio-envoy\") pod \"e2e-api-a-26444f-openshift-default-695897496f-mcdxk\" (UID: \"0474bb77-105a-4009-9f7e-2e19a4068fc9\") " pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:52:33.038110 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:33.037872 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9cvn\" (UniqueName: \"kubernetes.io/projected/0474bb77-105a-4009-9f7e-2e19a4068fc9-kube-api-access-c9cvn\") pod \"e2e-api-a-26444f-openshift-default-695897496f-mcdxk\" (UID: \"0474bb77-105a-4009-9f7e-2e19a4068fc9\") " pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:52:33.038110 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:33.037945 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0474bb77-105a-4009-9f7e-2e19a4068fc9-credential-socket\") pod \"e2e-api-a-26444f-openshift-default-695897496f-mcdxk\" (UID: \"0474bb77-105a-4009-9f7e-2e19a4068fc9\") " pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:52:33.038430 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:33.038172 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0474bb77-105a-4009-9f7e-2e19a4068fc9-workload-socket\") pod \"e2e-api-a-26444f-openshift-default-695897496f-mcdxk\" (UID: \"0474bb77-105a-4009-9f7e-2e19a4068fc9\") " pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:52:33.038730 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:33.038691 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0474bb77-105a-4009-9f7e-2e19a4068fc9-credential-socket\") pod \"e2e-api-a-26444f-openshift-default-695897496f-mcdxk\" (UID: \"0474bb77-105a-4009-9f7e-2e19a4068fc9\") " pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:52:33.046800 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:33.043132 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0474bb77-105a-4009-9f7e-2e19a4068fc9-istio-podinfo\") pod \"e2e-api-a-26444f-openshift-default-695897496f-mcdxk\" (UID: \"0474bb77-105a-4009-9f7e-2e19a4068fc9\") " pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:52:33.078881 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:33.068821 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0474bb77-105a-4009-9f7e-2e19a4068fc9-istio-envoy\") pod \"e2e-api-a-26444f-openshift-default-695897496f-mcdxk\" (UID: \"0474bb77-105a-4009-9f7e-2e19a4068fc9\") " pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:52:33.106182 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:33.106097 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0474bb77-105a-4009-9f7e-2e19a4068fc9-istio-token\") pod \"e2e-api-a-26444f-openshift-default-695897496f-mcdxk\" (UID: \"0474bb77-105a-4009-9f7e-2e19a4068fc9\") " pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:52:33.106182 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:33.106171 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9cvn\" (UniqueName: \"kubernetes.io/projected/0474bb77-105a-4009-9f7e-2e19a4068fc9-kube-api-access-c9cvn\") pod \"e2e-api-a-26444f-openshift-default-695897496f-mcdxk\" (UID: \"0474bb77-105a-4009-9f7e-2e19a4068fc9\") " pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:52:33.321606 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:33.321514 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:52:33.778920 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:33.778890 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk"] Nov 03 22:52:33.779772 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:52:33.779738 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0474bb77_105a_4009_9f7e_2e19a4068fc9.slice/crio-6ed1c7a54f2b912f3fa94f4ce7e3a697aac329fd4bd24f0055503366d8313b95 WatchSource:0}: Error finding container 6ed1c7a54f2b912f3fa94f4ce7e3a697aac329fd4bd24f0055503366d8313b95: Status 404 returned error can't find the container with id 6ed1c7a54f2b912f3fa94f4ce7e3a697aac329fd4bd24f0055503366d8313b95 Nov 03 22:52:33.782763 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:33.782726 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:52:33.782870 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:33.782799 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:52:33.782870 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:33.782830 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:52:33.793649 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:33.793620 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" event={"ID":"0474bb77-105a-4009-9f7e-2e19a4068fc9","Type":"ContainerStarted","Data":"6ed1c7a54f2b912f3fa94f4ce7e3a697aac329fd4bd24f0055503366d8313b95"} Nov 03 22:52:34.819545 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:34.819507 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" event={"ID":"0474bb77-105a-4009-9f7e-2e19a4068fc9","Type":"ContainerStarted","Data":"fb80c20770872da17ebaa7493076c9373630cbd3d9aa70d087b5070180aafdb9"} Nov 03 22:52:34.984805 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:34.984750 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" podStartSLOduration=2.9847361980000002 podStartE2EDuration="2.984736198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:52:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:52:34.98303171 +0000 UTC m=+1784.292591003" watchObservedRunningTime="2026-11-03 22:52:34.984736198 +0000 UTC m=+1784.294295553" Nov 03 22:52:35.222521 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:35.222484 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:35.322077 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:35.322040 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:52:35.330044 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:35.330018 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:52:35.817288 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:35.817259 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:52:35.818157 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:35.818134 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:52:37.925282 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:37.925249 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:38.084268 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:38.084236 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9"] Nov 03 22:52:38.084458 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:38.084438 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" podUID="45021679-817b-417f-9f0c-dcbad61cc1d3" containerName="istio-proxy" containerID="cri-o://0c3d9a242e94fe9503229343f2383db85ab47b8d460c16e08eb5830b71b80051" gracePeriod=30 Nov 03 22:52:38.103169 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:38.103140 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:38.252657 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:38.252560 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:40.422149 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:40.422110 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:41.175978 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:41.175946 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:41.355612 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:41.355571 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:42.239777 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:42.239740 5196 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:52:42.240194 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:42.239812 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" podUID="45021679-817b-417f-9f0c-dcbad61cc1d3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:52:43.376282 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:43.376251 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" Nov 03 22:52:43.492075 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:43.492048 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45021679-817b-417f-9f0c-dcbad61cc1d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45021679-817b-417f-9f0c-dcbad61cc1d3-workload-socket\") pod \"45021679-817b-417f-9f0c-dcbad61cc1d3\" (UID: \"45021679-817b-417f-9f0c-dcbad61cc1d3\") " Nov 03 22:52:43.492231 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:43.492093 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45021679-817b-417f-9f0c-dcbad61cc1d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45021679-817b-417f-9f0c-dcbad61cc1d3-credential-socket\") pod \"45021679-817b-417f-9f0c-dcbad61cc1d3\" (UID: \"45021679-817b-417f-9f0c-dcbad61cc1d3\") " Nov 03 22:52:43.492231 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:43.492128 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45021679-817b-417f-9f0c-dcbad61cc1d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45021679-817b-417f-9f0c-dcbad61cc1d3-istio-envoy\") pod \"45021679-817b-417f-9f0c-dcbad61cc1d3\" (UID: \"45021679-817b-417f-9f0c-dcbad61cc1d3\") " Nov 03 22:52:43.492231 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:43.492164 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45021679-817b-417f-9f0c-dcbad61cc1d3-kube-api-access-8b6zt\" (UniqueName: \"kubernetes.io/projected/45021679-817b-417f-9f0c-dcbad61cc1d3-kube-api-access-8b6zt\") pod \"45021679-817b-417f-9f0c-dcbad61cc1d3\" (UID: \"45021679-817b-417f-9f0c-dcbad61cc1d3\") " Nov 03 22:52:43.492231 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:43.492220 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45021679-817b-417f-9f0c-dcbad61cc1d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45021679-817b-417f-9f0c-dcbad61cc1d3-istiod-ca-cert\") pod \"45021679-817b-417f-9f0c-dcbad61cc1d3\" (UID: \"45021679-817b-417f-9f0c-dcbad61cc1d3\") " Nov 03 22:52:43.492437 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:43.492278 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45021679-817b-417f-9f0c-dcbad61cc1d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45021679-817b-417f-9f0c-dcbad61cc1d3-istio-data\") pod \"45021679-817b-417f-9f0c-dcbad61cc1d3\" (UID: \"45021679-817b-417f-9f0c-dcbad61cc1d3\") " Nov 03 22:52:43.492437 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:43.492327 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45021679-817b-417f-9f0c-dcbad61cc1d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45021679-817b-417f-9f0c-dcbad61cc1d3-istio-podinfo\") pod \"45021679-817b-417f-9f0c-dcbad61cc1d3\" (UID: \"45021679-817b-417f-9f0c-dcbad61cc1d3\") " Nov 03 22:52:43.492437 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:43.492341 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45021679-817b-417f-9f0c-dcbad61cc1d3-workload-socket" pod "45021679-817b-417f-9f0c-dcbad61cc1d3" (UID: "45021679-817b-417f-9f0c-dcbad61cc1d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:43.492437 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:43.492423 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45021679-817b-417f-9f0c-dcbad61cc1d3-credential-socket" pod "45021679-817b-417f-9f0c-dcbad61cc1d3" (UID: "45021679-817b-417f-9f0c-dcbad61cc1d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:43.492686 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:43.492527 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45021679-817b-417f-9f0c-dcbad61cc1d3-istio-token\" (UniqueName: \"kubernetes.io/projected/45021679-817b-417f-9f0c-dcbad61cc1d3-istio-token\") pod \"45021679-817b-417f-9f0c-dcbad61cc1d3\" (UID: \"45021679-817b-417f-9f0c-dcbad61cc1d3\") " Nov 03 22:52:43.492686 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:43.492568 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45021679-817b-417f-9f0c-dcbad61cc1d3-istio-data" pod "45021679-817b-417f-9f0c-dcbad61cc1d3" (UID: "45021679-817b-417f-9f0c-dcbad61cc1d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:43.492686 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:43.492647 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45021679-817b-417f-9f0c-dcbad61cc1d3-istiod-ca-cert" pod "45021679-817b-417f-9f0c-dcbad61cc1d3" (UID: "45021679-817b-417f-9f0c-dcbad61cc1d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:43.492845 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:43.492822 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45021679-817b-417f-9f0c-dcbad61cc1d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45021679-817b-417f-9f0c-dcbad61cc1d3-workload-certs\") pod \"45021679-817b-417f-9f0c-dcbad61cc1d3\" (UID: \"45021679-817b-417f-9f0c-dcbad61cc1d3\") " Nov 03 22:52:43.493344 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:43.493317 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45021679-817b-417f-9f0c-dcbad61cc1d3-workload-certs" pod "45021679-817b-417f-9f0c-dcbad61cc1d3" (UID: "45021679-817b-417f-9f0c-dcbad61cc1d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:43.493560 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:43.493543 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45021679-817b-417f-9f0c-dcbad61cc1d3-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:52:43.493602 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:43.493568 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45021679-817b-417f-9f0c-dcbad61cc1d3-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:52:43.493602 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:43.493585 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45021679-817b-417f-9f0c-dcbad61cc1d3-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:52:43.493662 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:43.493602 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45021679-817b-417f-9f0c-dcbad61cc1d3-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:52:43.493662 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:43.493616 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45021679-817b-417f-9f0c-dcbad61cc1d3-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:52:43.494822 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:43.494798 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45021679-817b-417f-9f0c-dcbad61cc1d3-istio-token" pod "45021679-817b-417f-9f0c-dcbad61cc1d3" (UID: "45021679-817b-417f-9f0c-dcbad61cc1d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:43.494822 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:43.494793 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45021679-817b-417f-9f0c-dcbad61cc1d3-istio-podinfo" pod "45021679-817b-417f-9f0c-dcbad61cc1d3" (UID: "45021679-817b-417f-9f0c-dcbad61cc1d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:52:43.495103 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:43.495086 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45021679-817b-417f-9f0c-dcbad61cc1d3-istio-envoy" pod "45021679-817b-417f-9f0c-dcbad61cc1d3" (UID: "45021679-817b-417f-9f0c-dcbad61cc1d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:43.495186 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:43.495172 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45021679-817b-417f-9f0c-dcbad61cc1d3-kube-api-access-8b6zt" pod "45021679-817b-417f-9f0c-dcbad61cc1d3" (UID: "45021679-817b-417f-9f0c-dcbad61cc1d3"). InnerVolumeSpecName "kube-api-access-8b6zt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:43.594932 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:43.594895 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45021679-817b-417f-9f0c-dcbad61cc1d3-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:52:43.594932 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:43.594933 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45021679-817b-417f-9f0c-dcbad61cc1d3-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:52:43.595184 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:43.594949 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45021679-817b-417f-9f0c-dcbad61cc1d3-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:52:43.595184 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:43.594963 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b6zt\" (UniqueName: \"kubernetes.io/projected/45021679-817b-417f-9f0c-dcbad61cc1d3-kube-api-access-8b6zt\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:52:43.873626 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:43.873588 5196 generic.go:356] "Generic (PLEG): container finished" podID="45021679-817b-417f-9f0c-dcbad61cc1d3" containerID="0c3d9a242e94fe9503229343f2383db85ab47b8d460c16e08eb5830b71b80051" exitCode=0 Nov 03 22:52:43.873812 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:43.873638 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" Nov 03 22:52:43.873812 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:43.873654 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" event={"ID":"45021679-817b-417f-9f0c-dcbad61cc1d3","Type":"ContainerDied","Data":"0c3d9a242e94fe9503229343f2383db85ab47b8d460c16e08eb5830b71b80051"} Nov 03 22:52:43.873812 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:43.873712 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9" event={"ID":"45021679-817b-417f-9f0c-dcbad61cc1d3","Type":"ContainerDied","Data":"8032eef40add27d2ac4c8123da9e819bdedb22956ef35bf2ef265ff9e11c4e79"} Nov 03 22:52:43.873812 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:43.873735 5196 scope.go:122] "RemoveContainer" containerID="0c3d9a242e94fe9503229343f2383db85ab47b8d460c16e08eb5830b71b80051" Nov 03 22:52:43.886490 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:43.886452 5196 scope.go:122] "RemoveContainer" containerID="0c3d9a242e94fe9503229343f2383db85ab47b8d460c16e08eb5830b71b80051" Nov 03 22:52:43.886806 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:52:43.886774 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c3d9a242e94fe9503229343f2383db85ab47b8d460c16e08eb5830b71b80051\": container with ID starting with 0c3d9a242e94fe9503229343f2383db85ab47b8d460c16e08eb5830b71b80051 not found: ID does not exist" containerID="0c3d9a242e94fe9503229343f2383db85ab47b8d460c16e08eb5830b71b80051" Nov 03 22:52:43.886977 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:43.886816 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c3d9a242e94fe9503229343f2383db85ab47b8d460c16e08eb5830b71b80051"} err="failed to get container status \"0c3d9a242e94fe9503229343f2383db85ab47b8d460c16e08eb5830b71b80051\": rpc error: code = NotFound desc = could not find container \"0c3d9a242e94fe9503229343f2383db85ab47b8d460c16e08eb5830b71b80051\": container with ID starting with 0c3d9a242e94fe9503229343f2383db85ab47b8d460c16e08eb5830b71b80051 not found: ID does not exist" Nov 03 22:52:44.043120 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:44.043091 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9"] Nov 03 22:52:44.093494 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:44.093466 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-nlgd9"] Nov 03 22:52:44.816063 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:44.816032 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45021679-817b-417f-9f0c-dcbad61cc1d3" path="/var/lib/kubelet/pods/45021679-817b-417f-9f0c-dcbad61cc1d3/volumes" Nov 03 22:52:45.160566 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:45.160532 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k"] Nov 03 22:52:45.161139 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:45.161124 5196 cpu_manager.go:414] "containerMap: removing container" podUID="45021679-817b-417f-9f0c-dcbad61cc1d3" containerName="istio-proxy" Nov 03 22:52:45.161199 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:45.161141 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45021679-817b-417f-9f0c-dcbad61cc1d3" containerName="istio-proxy" Nov 03 22:52:45.161249 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:45.161240 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="45021679-817b-417f-9f0c-dcbad61cc1d3" containerName="istio-proxy" Nov 03 22:52:45.165830 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:45.165812 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k" Nov 03 22:52:45.220262 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:45.220228 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-433db8-serving-cert\"" Nov 03 22:52:45.274518 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:45.274485 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k"] Nov 03 22:52:45.318020 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:45.317985 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64ntn\" (UniqueName: \"kubernetes.io/projected/3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b-kube-api-access-64ntn\") pod \"maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k\" (UID: \"3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k" Nov 03 22:52:45.318182 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:45.318108 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k\" (UID: \"3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k" Nov 03 22:52:45.318182 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:45.318159 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k\" (UID: \"3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k" Nov 03 22:52:45.318377 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:45.318360 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k\" (UID: \"3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k" Nov 03 22:52:45.351728 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:45.351698 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:45.422536 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:45.422439 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k\" (UID: \"3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k" Nov 03 22:52:45.422536 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:45.422496 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64ntn\" (UniqueName: \"kubernetes.io/projected/3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b-kube-api-access-64ntn\") pod \"maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k\" (UID: \"3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k" Nov 03 22:52:45.422773 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:45.422589 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k\" (UID: \"3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k" Nov 03 22:52:45.422773 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:45.422629 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k\" (UID: \"3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k" Nov 03 22:52:45.423287 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:52:45.423256 5196 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-433db8-serving-cert: secret "maas-api-e2e-shared-b-w3-433db8-serving-cert" not found Nov 03 22:52:45.423390 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:52:45.423353 5196 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b-maas-api-tls podName:3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b nodeName:}" failed. No retries permitted until 2026-11-03 22:52:45.923330775 +0000 UTC m=+1795.232890048 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b-maas-api-tls") pod "maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k" (UID: "3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b") : secret "maas-api-e2e-shared-b-w3-433db8-serving-cert" not found Nov 03 22:52:45.423610 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:45.423583 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k\" (UID: \"3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k" Nov 03 22:52:45.426431 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:45.426410 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k\" (UID: \"3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k" Nov 03 22:52:45.486070 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:45.486036 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64ntn\" (UniqueName: \"kubernetes.io/projected/3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b-kube-api-access-64ntn\") pod \"maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k\" (UID: \"3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k" Nov 03 22:52:45.549661 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:45.549631 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:45.691272 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:45.691183 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:45.929438 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:45.929400 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k\" (UID: \"3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k" Nov 03 22:52:45.932066 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:45.932043 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k\" (UID: \"3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k" Nov 03 22:52:46.081670 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:46.081600 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k" Nov 03 22:52:46.462264 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:46.462233 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k"] Nov 03 22:52:46.463359 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:52:46.463334 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f495de2_3c7c_4d3b_9da6_bdb97ac16e8b.slice/crio-6669e7e8f6da3d9b687df28f3ddf35923c2b866b48f5e6caab105e963e84addb WatchSource:0}: Error finding container 6669e7e8f6da3d9b687df28f3ddf35923c2b866b48f5e6caab105e963e84addb: Status 404 returned error can't find the container with id 6669e7e8f6da3d9b687df28f3ddf35923c2b866b48f5e6caab105e963e84addb Nov 03 22:52:46.895528 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:46.895491 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k" event={"ID":"3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b","Type":"ContainerStarted","Data":"6669e7e8f6da3d9b687df28f3ddf35923c2b866b48f5e6caab105e963e84addb"} Nov 03 22:52:47.903791 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:47.903757 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k" event={"ID":"3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b","Type":"ContainerStarted","Data":"cb823ccc60b573dfae944eef3c494c36099bb1d19cb2f5464bf06ad700efff7c"} Nov 03 22:52:47.904210 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:47.903982 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k" Nov 03 22:52:48.024479 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:48.024424 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k" podStartSLOduration=2.145259003 podStartE2EDuration="3.024411533s" totalImagesPullingTime="879.15253ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:52:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:52:48.022874393 +0000 UTC m=+1797.332433688" watchObservedRunningTime="2026-11-03 22:52:48.024411533 +0000 UTC m=+1797.333970826" Nov 03 22:52:50.760604 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:50.760573 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:52.597774 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:52.597736 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5"] Nov 03 22:52:52.598235 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:52.598045 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" podUID="12d2e868-b385-44e6-8a56-e933b33f5b7a" containerName="istio-proxy" containerID="cri-o://33112d1a0e294842f71f78bb7cb01b1cb5422e598e6b6ce48458ef7e0bff1b5f" gracePeriod=30 Nov 03 22:52:53.915474 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:53.915446 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k" Nov 03 22:52:55.425478 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:55.425443 5196 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:52:55.425849 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:55.425498 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" podUID="12d2e868-b385-44e6-8a56-e933b33f5b7a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:52:55.849690 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:55.849652 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:55.923117 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:52:55.923081 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:52:57.367151 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:57.367118 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk"] Nov 03 22:52:57.367556 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:57.367350 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" podUID="aa3310c5-bd35-4bef-b838-1722c7143cf5" containerName="istio-proxy" containerID="cri-o://779c2ca89aa4a9f4f3fa65ee3dc70cf070ed23f5d9088c47c1d008705cf57287" gracePeriod=30 Nov 03 22:52:57.418141 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:57.418110 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98"] Nov 03 22:52:57.418369 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:57.418332 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" podUID="3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c" containerName="istio-proxy" containerID="cri-o://0e6e2cb774a6795ad59b759598e1e51148d8c5b389b3816409eff65ded329859" gracePeriod=30 Nov 03 22:52:57.421202 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:57.421157 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz"] Nov 03 22:52:57.421369 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:57.421349 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" podUID="ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2" containerName="istio-proxy" containerID="cri-o://2faa7a1c976f7a0dc8b989b7d7e728293c13b15c6013e4222315347d36d2468a" gracePeriod=30 Nov 03 22:52:57.469700 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:57.469669 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x"] Nov 03 22:52:57.469931 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:57.469907 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" podUID="85a61150-46ba-406c-b384-99565f653f15" containerName="istio-proxy" containerID="cri-o://5af2747d92233beb4c13a39aa86448cace1e72168fd51edc1673ff7de72c9005" gracePeriod=30 Nov 03 22:52:57.882226 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:57.882159 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" Nov 03 22:52:58.013294 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.013261 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12d2e868-b385-44e6-8a56-e933b33f5b7a-istio-token\" (UniqueName: \"kubernetes.io/projected/12d2e868-b385-44e6-8a56-e933b33f5b7a-istio-token\") pod \"12d2e868-b385-44e6-8a56-e933b33f5b7a\" (UID: \"12d2e868-b385-44e6-8a56-e933b33f5b7a\") " Nov 03 22:52:58.013476 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.013318 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d2e868-b385-44e6-8a56-e933b33f5b7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d2e868-b385-44e6-8a56-e933b33f5b7a-workload-certs\") pod \"12d2e868-b385-44e6-8a56-e933b33f5b7a\" (UID: \"12d2e868-b385-44e6-8a56-e933b33f5b7a\") " Nov 03 22:52:58.013476 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.013357 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12d2e868-b385-44e6-8a56-e933b33f5b7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d2e868-b385-44e6-8a56-e933b33f5b7a-istiod-ca-cert\") pod \"12d2e868-b385-44e6-8a56-e933b33f5b7a\" (UID: \"12d2e868-b385-44e6-8a56-e933b33f5b7a\") " Nov 03 22:52:58.013476 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.013403 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12d2e868-b385-44e6-8a56-e933b33f5b7a-kube-api-access-z5dw6\" (UniqueName: \"kubernetes.io/projected/12d2e868-b385-44e6-8a56-e933b33f5b7a-kube-api-access-z5dw6\") pod \"12d2e868-b385-44e6-8a56-e933b33f5b7a\" (UID: \"12d2e868-b385-44e6-8a56-e933b33f5b7a\") " Nov 03 22:52:58.013644 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.013613 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12d2e868-b385-44e6-8a56-e933b33f5b7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d2e868-b385-44e6-8a56-e933b33f5b7a-istio-podinfo\") pod \"12d2e868-b385-44e6-8a56-e933b33f5b7a\" (UID: \"12d2e868-b385-44e6-8a56-e933b33f5b7a\") " Nov 03 22:52:58.013644 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.013628 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d2e868-b385-44e6-8a56-e933b33f5b7a-workload-certs" pod "12d2e868-b385-44e6-8a56-e933b33f5b7a" (UID: "12d2e868-b385-44e6-8a56-e933b33f5b7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:58.013751 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.013667 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d2e868-b385-44e6-8a56-e933b33f5b7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d2e868-b385-44e6-8a56-e933b33f5b7a-workload-socket\") pod \"12d2e868-b385-44e6-8a56-e933b33f5b7a\" (UID: \"12d2e868-b385-44e6-8a56-e933b33f5b7a\") " Nov 03 22:52:58.013751 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.013722 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d2e868-b385-44e6-8a56-e933b33f5b7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d2e868-b385-44e6-8a56-e933b33f5b7a-istio-data\") pod \"12d2e868-b385-44e6-8a56-e933b33f5b7a\" (UID: \"12d2e868-b385-44e6-8a56-e933b33f5b7a\") " Nov 03 22:52:58.013897 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.013761 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d2e868-b385-44e6-8a56-e933b33f5b7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d2e868-b385-44e6-8a56-e933b33f5b7a-istio-envoy\") pod \"12d2e868-b385-44e6-8a56-e933b33f5b7a\" (UID: \"12d2e868-b385-44e6-8a56-e933b33f5b7a\") " Nov 03 22:52:58.013897 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.013775 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12d2e868-b385-44e6-8a56-e933b33f5b7a-istiod-ca-cert" pod "12d2e868-b385-44e6-8a56-e933b33f5b7a" (UID: "12d2e868-b385-44e6-8a56-e933b33f5b7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:58.013897 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.013819 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d2e868-b385-44e6-8a56-e933b33f5b7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d2e868-b385-44e6-8a56-e933b33f5b7a-credential-socket\") pod \"12d2e868-b385-44e6-8a56-e933b33f5b7a\" (UID: \"12d2e868-b385-44e6-8a56-e933b33f5b7a\") " Nov 03 22:52:58.014061 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.013995 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d2e868-b385-44e6-8a56-e933b33f5b7a-workload-socket" pod "12d2e868-b385-44e6-8a56-e933b33f5b7a" (UID: "12d2e868-b385-44e6-8a56-e933b33f5b7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:58.014314 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.014282 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d2e868-b385-44e6-8a56-e933b33f5b7a-istio-data" pod "12d2e868-b385-44e6-8a56-e933b33f5b7a" (UID: "12d2e868-b385-44e6-8a56-e933b33f5b7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:58.014476 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.014390 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d2e868-b385-44e6-8a56-e933b33f5b7a-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:52:58.014476 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.014397 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d2e868-b385-44e6-8a56-e933b33f5b7a-credential-socket" pod "12d2e868-b385-44e6-8a56-e933b33f5b7a" (UID: "12d2e868-b385-44e6-8a56-e933b33f5b7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:58.014476 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.014406 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d2e868-b385-44e6-8a56-e933b33f5b7a-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:52:58.014476 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.014440 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d2e868-b385-44e6-8a56-e933b33f5b7a-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:52:58.015773 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.015750 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12d2e868-b385-44e6-8a56-e933b33f5b7a-istio-token" pod "12d2e868-b385-44e6-8a56-e933b33f5b7a" (UID: "12d2e868-b385-44e6-8a56-e933b33f5b7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:58.016034 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.016008 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12d2e868-b385-44e6-8a56-e933b33f5b7a-kube-api-access-z5dw6" pod "12d2e868-b385-44e6-8a56-e933b33f5b7a" (UID: "12d2e868-b385-44e6-8a56-e933b33f5b7a"). InnerVolumeSpecName "kube-api-access-z5dw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:58.016125 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.016106 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12d2e868-b385-44e6-8a56-e933b33f5b7a-istio-podinfo" pod "12d2e868-b385-44e6-8a56-e933b33f5b7a" (UID: "12d2e868-b385-44e6-8a56-e933b33f5b7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:52:58.016408 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.016391 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d2e868-b385-44e6-8a56-e933b33f5b7a-istio-envoy" pod "12d2e868-b385-44e6-8a56-e933b33f5b7a" (UID: "12d2e868-b385-44e6-8a56-e933b33f5b7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:58.026168 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.026138 5196 generic.go:356] "Generic (PLEG): container finished" podID="12d2e868-b385-44e6-8a56-e933b33f5b7a" containerID="33112d1a0e294842f71f78bb7cb01b1cb5422e598e6b6ce48458ef7e0bff1b5f" exitCode=0 Nov 03 22:52:58.026341 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.026324 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" Nov 03 22:52:58.026433 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.026378 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" event={"ID":"12d2e868-b385-44e6-8a56-e933b33f5b7a","Type":"ContainerDied","Data":"33112d1a0e294842f71f78bb7cb01b1cb5422e598e6b6ce48458ef7e0bff1b5f"} Nov 03 22:52:58.026433 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.026430 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5" event={"ID":"12d2e868-b385-44e6-8a56-e933b33f5b7a","Type":"ContainerDied","Data":"22d1d0266a630dcd1afc0b635d22ddc4367868e64fc33bcba766ee3656556d3a"} Nov 03 22:52:58.026600 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.026462 5196 scope.go:122] "RemoveContainer" containerID="33112d1a0e294842f71f78bb7cb01b1cb5422e598e6b6ce48458ef7e0bff1b5f" Nov 03 22:52:58.048623 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.048593 5196 scope.go:122] "RemoveContainer" containerID="33112d1a0e294842f71f78bb7cb01b1cb5422e598e6b6ce48458ef7e0bff1b5f" Nov 03 22:52:58.049499 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:52:58.049418 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33112d1a0e294842f71f78bb7cb01b1cb5422e598e6b6ce48458ef7e0bff1b5f\": container with ID starting with 33112d1a0e294842f71f78bb7cb01b1cb5422e598e6b6ce48458ef7e0bff1b5f not found: ID does not exist" containerID="33112d1a0e294842f71f78bb7cb01b1cb5422e598e6b6ce48458ef7e0bff1b5f" Nov 03 22:52:58.049499 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.049460 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33112d1a0e294842f71f78bb7cb01b1cb5422e598e6b6ce48458ef7e0bff1b5f"} err="failed to get container status \"33112d1a0e294842f71f78bb7cb01b1cb5422e598e6b6ce48458ef7e0bff1b5f\": rpc error: code = NotFound desc = could not find container \"33112d1a0e294842f71f78bb7cb01b1cb5422e598e6b6ce48458ef7e0bff1b5f\": container with ID starting with 33112d1a0e294842f71f78bb7cb01b1cb5422e598e6b6ce48458ef7e0bff1b5f not found: ID does not exist" Nov 03 22:52:58.115694 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.115667 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d2e868-b385-44e6-8a56-e933b33f5b7a-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:52:58.115694 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.115693 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d2e868-b385-44e6-8a56-e933b33f5b7a-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:52:58.115905 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.115705 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d2e868-b385-44e6-8a56-e933b33f5b7a-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:52:58.115905 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.115715 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d2e868-b385-44e6-8a56-e933b33f5b7a-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:52:58.115905 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.115723 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d2e868-b385-44e6-8a56-e933b33f5b7a-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:52:58.115905 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.115732 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5dw6\" (UniqueName: \"kubernetes.io/projected/12d2e868-b385-44e6-8a56-e933b33f5b7a-kube-api-access-z5dw6\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:52:58.197023 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.196928 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5"] Nov 03 22:52:58.200510 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.200483 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-wb2g5"] Nov 03 22:52:58.324307 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.324271 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:58.652336 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.652299 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vj28x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:52:58.652707 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.652352 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" podUID="85a61150-46ba-406c-b384-99565f653f15" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:52:58.774512 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.774481 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:58.816261 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:58.816231 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12d2e868-b385-44e6-8a56-e933b33f5b7a" path="/var/lib/kubelet/pods/12d2e868-b385-44e6-8a56-e933b33f5b7a/volumes" Nov 03 22:52:59.540120 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:59.540030 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj"] Nov 03 22:52:59.540740 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:59.540715 5196 cpu_manager.go:414] "containerMap: removing container" podUID="12d2e868-b385-44e6-8a56-e933b33f5b7a" containerName="istio-proxy" Nov 03 22:52:59.540740 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:59.540737 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12d2e868-b385-44e6-8a56-e933b33f5b7a" containerName="istio-proxy" Nov 03 22:52:59.540974 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:59.540959 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="12d2e868-b385-44e6-8a56-e933b33f5b7a" containerName="istio-proxy" Nov 03 22:52:59.545836 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:59.545814 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:52:59.604479 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:59.604447 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-0406a8-openshift-default-dockercfg-kgq75\"" Nov 03 22:52:59.636202 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:59.636152 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-workload-socket\") pod \"e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj\" (UID: \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\") " pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:52:59.636431 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:59.636380 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-credential-socket\") pod \"e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj\" (UID: \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\") " pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:52:59.636534 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:59.636497 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-workload-certs\") pod \"e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj\" (UID: \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\") " pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:52:59.636598 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:59.636545 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-istiod-ca-cert\") pod \"e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj\" (UID: \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\") " pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:52:59.636598 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:59.636581 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-istio-envoy\") pod \"e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj\" (UID: \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\") " pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:52:59.636688 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:59.636615 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-istio-podinfo\") pod \"e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj\" (UID: \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\") " pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:52:59.636746 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:59.636721 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-istio-token\") pod \"e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj\" (UID: \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\") " pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:52:59.636819 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:59.636798 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4ml4\" (UniqueName: \"kubernetes.io/projected/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-kube-api-access-n4ml4\") pod \"e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj\" (UID: \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\") " pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:52:59.636890 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:59.636840 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-istio-data\") pod \"e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj\" (UID: \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\") " pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:52:59.640908 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:59.640886 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj"] Nov 03 22:52:59.669433 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:59.669400 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:52:59.740294 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:59.740245 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-istio-token\") pod \"e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj\" (UID: \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\") " pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:52:59.740514 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:59.740350 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4ml4\" (UniqueName: \"kubernetes.io/projected/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-kube-api-access-n4ml4\") pod \"e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj\" (UID: \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\") " pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:52:59.740514 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:59.740387 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-istio-data\") pod \"e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj\" (UID: \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\") " pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:52:59.740514 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:59.740429 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-workload-socket\") pod \"e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj\" (UID: \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\") " pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:52:59.740680 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:59.740583 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-credential-socket\") pod \"e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj\" (UID: \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\") " pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:52:59.740680 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:59.740667 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-workload-certs\") pod \"e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj\" (UID: \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\") " pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:52:59.740785 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:59.740696 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-istiod-ca-cert\") pod \"e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj\" (UID: \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\") " pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:52:59.740785 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:59.740731 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-istio-envoy\") pod \"e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj\" (UID: \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\") " pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:52:59.740785 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:59.740760 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-istio-podinfo\") pod \"e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj\" (UID: \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\") " pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:52:59.742585 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:59.741786 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-istio-data\") pod \"e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj\" (UID: \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\") " pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:52:59.742585 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:59.742148 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-credential-socket\") pod \"e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj\" (UID: \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\") " pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:52:59.742585 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:59.742522 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-workload-certs\") pod \"e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj\" (UID: \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\") " pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:52:59.776510 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:59.776474 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-istio-podinfo\") pod \"e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj\" (UID: \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\") " pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:52:59.776814 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:59.776782 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-workload-socket\") pod \"e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj\" (UID: \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\") " pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:52:59.777075 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:59.777052 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-istio-envoy\") pod \"e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj\" (UID: \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\") " pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:52:59.777171 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:59.751539 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-istiod-ca-cert\") pod \"e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj\" (UID: \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\") " pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:52:59.805999 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:59.805926 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4ml4\" (UniqueName: \"kubernetes.io/projected/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-kube-api-access-n4ml4\") pod \"e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj\" (UID: \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\") " pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:52:59.838691 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:59.838660 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-istio-token\") pod \"e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj\" (UID: \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\") " pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:52:59.864833 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:59.864802 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:52:59.997206 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:52:59.997119 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:00.305284 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:00.305260 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj"] Nov 03 22:53:00.307434 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:53:00.307406 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fde3ea1_b20e_4828_b8e6_5ffd537294f6.slice/crio-d554ffa146fd65630cefc6114f513f1129168bd4d116150149307b55d9b0f0e5 WatchSource:0}: Error finding container d554ffa146fd65630cefc6114f513f1129168bd4d116150149307b55d9b0f0e5: Status 404 returned error can't find the container with id d554ffa146fd65630cefc6114f513f1129168bd4d116150149307b55d9b0f0e5 Nov 03 22:53:00.310193 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:00.310160 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:53:00.310272 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:00.310231 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:53:00.310272 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:00.310260 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:53:01.049762 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:01.049716 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" event={"ID":"9fde3ea1-b20e-4828-b8e6-5ffd537294f6","Type":"ContainerStarted","Data":"a48ffae424ef174332893917bac4904aad4fbab46d0da2576e210e54c8c3352f"} Nov 03 22:53:01.050182 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:01.049785 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" event={"ID":"9fde3ea1-b20e-4828-b8e6-5ffd537294f6","Type":"ContainerStarted","Data":"d554ffa146fd65630cefc6114f513f1129168bd4d116150149307b55d9b0f0e5"} Nov 03 22:53:01.179341 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:01.179294 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" podStartSLOduration=2.179277582 podStartE2EDuration="2.179277582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:53:01.176611878 +0000 UTC m=+1810.486171170" watchObservedRunningTime="2026-11-03 22:53:01.179277582 +0000 UTC m=+1810.488836874" Nov 03 22:53:01.192343 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:01.192305 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:01.865026 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:01.864998 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:53:01.871063 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:01.871040 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:53:02.057442 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.057414 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:53:02.058416 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.058399 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:53:02.725577 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.725545 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:53:02.786468 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.786438 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" Nov 03 22:53:02.831178 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.831156 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" Nov 03 22:53:02.834343 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.834320 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" Nov 03 22:53:02.883797 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.883767 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa3310c5-bd35-4bef-b838-1722c7143cf5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa3310c5-bd35-4bef-b838-1722c7143cf5-istio-podinfo\") pod \"aa3310c5-bd35-4bef-b838-1722c7143cf5\" (UID: \"aa3310c5-bd35-4bef-b838-1722c7143cf5\") " Nov 03 22:53:02.883942 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.883805 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa3310c5-bd35-4bef-b838-1722c7143cf5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3310c5-bd35-4bef-b838-1722c7143cf5-credential-socket\") pod \"aa3310c5-bd35-4bef-b838-1722c7143cf5\" (UID: \"aa3310c5-bd35-4bef-b838-1722c7143cf5\") " Nov 03 22:53:02.883942 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.883828 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-istio-podinfo\") pod \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\" (UID: \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\") " Nov 03 22:53:02.883942 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.883894 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa3310c5-bd35-4bef-b838-1722c7143cf5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3310c5-bd35-4bef-b838-1722c7143cf5-workload-socket\") pod \"aa3310c5-bd35-4bef-b838-1722c7143cf5\" (UID: \"aa3310c5-bd35-4bef-b838-1722c7143cf5\") " Nov 03 22:53:02.883942 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.883936 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa3310c5-bd35-4bef-b838-1722c7143cf5-istio-token\" (UniqueName: \"kubernetes.io/projected/aa3310c5-bd35-4bef-b838-1722c7143cf5-istio-token\") pod \"aa3310c5-bd35-4bef-b838-1722c7143cf5\" (UID: \"aa3310c5-bd35-4bef-b838-1722c7143cf5\") " Nov 03 22:53:02.884583 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.884025 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-istio-token\" (UniqueName: \"kubernetes.io/projected/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-istio-token\") pod \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\" (UID: \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\") " Nov 03 22:53:02.884583 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.884066 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-workload-socket\") pod \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\" (UID: \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\") " Nov 03 22:53:02.884583 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.884095 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa3310c5-bd35-4bef-b838-1722c7143cf5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa3310c5-bd35-4bef-b838-1722c7143cf5-istio-data\") pod \"aa3310c5-bd35-4bef-b838-1722c7143cf5\" (UID: \"aa3310c5-bd35-4bef-b838-1722c7143cf5\") " Nov 03 22:53:02.884583 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.884180 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3310c5-bd35-4bef-b838-1722c7143cf5-credential-socket" pod "aa3310c5-bd35-4bef-b838-1722c7143cf5" (UID: "aa3310c5-bd35-4bef-b838-1722c7143cf5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:02.884583 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.884284 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3310c5-bd35-4bef-b838-1722c7143cf5-workload-socket" pod "aa3310c5-bd35-4bef-b838-1722c7143cf5" (UID: "aa3310c5-bd35-4bef-b838-1722c7143cf5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:02.884583 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.884358 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3310c5-bd35-4bef-b838-1722c7143cf5-istio-data" pod "aa3310c5-bd35-4bef-b838-1722c7143cf5" (UID: "aa3310c5-bd35-4bef-b838-1722c7143cf5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:02.884583 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.884365 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-workload-socket" pod "3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c" (UID: "3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:02.884583 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.884123 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-kube-api-access-jjv82\" (UniqueName: \"kubernetes.io/projected/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-kube-api-access-jjv82\") pod \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\" (UID: \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\") " Nov 03 22:53:02.884583 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.884527 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-istio-data\") pod \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\" (UID: \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\") " Nov 03 22:53:02.885255 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.884957 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-istio-data" pod "3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c" (UID: "3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:02.885255 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.885012 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-istiod-ca-cert\") pod \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\" (UID: \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\") " Nov 03 22:53:02.885255 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.885134 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-istio-envoy\") pod \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\" (UID: \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\") " Nov 03 22:53:02.885255 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.885166 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-istiod-ca-cert" pod "3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c" (UID: "3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:53:02.885255 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.885186 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa3310c5-bd35-4bef-b838-1722c7143cf5-kube-api-access-nbbw2\" (UniqueName: \"kubernetes.io/projected/aa3310c5-bd35-4bef-b838-1722c7143cf5-kube-api-access-nbbw2\") pod \"aa3310c5-bd35-4bef-b838-1722c7143cf5\" (UID: \"aa3310c5-bd35-4bef-b838-1722c7143cf5\") " Nov 03 22:53:02.885519 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.885296 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-workload-certs\") pod \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\" (UID: \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\") " Nov 03 22:53:02.885519 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.885348 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa3310c5-bd35-4bef-b838-1722c7143cf5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa3310c5-bd35-4bef-b838-1722c7143cf5-istio-envoy\") pod \"aa3310c5-bd35-4bef-b838-1722c7143cf5\" (UID: \"aa3310c5-bd35-4bef-b838-1722c7143cf5\") " Nov 03 22:53:02.885519 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.885405 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-credential-socket\") pod \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\" (UID: \"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c\") " Nov 03 22:53:02.885519 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.885441 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa3310c5-bd35-4bef-b838-1722c7143cf5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa3310c5-bd35-4bef-b838-1722c7143cf5-istiod-ca-cert\") pod \"aa3310c5-bd35-4bef-b838-1722c7143cf5\" (UID: \"aa3310c5-bd35-4bef-b838-1722c7143cf5\") " Nov 03 22:53:02.885519 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.885483 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa3310c5-bd35-4bef-b838-1722c7143cf5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa3310c5-bd35-4bef-b838-1722c7143cf5-workload-certs\") pod \"aa3310c5-bd35-4bef-b838-1722c7143cf5\" (UID: \"aa3310c5-bd35-4bef-b838-1722c7143cf5\") " Nov 03 22:53:02.886417 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.886353 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:02.886417 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.886377 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3310c5-bd35-4bef-b838-1722c7143cf5-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:02.886417 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.886392 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3310c5-bd35-4bef-b838-1722c7143cf5-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:02.886417 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.886407 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:02.886417 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.886420 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa3310c5-bd35-4bef-b838-1722c7143cf5-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:02.886745 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.886438 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:02.886745 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.886549 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa3310c5-bd35-4bef-b838-1722c7143cf5-istio-token" pod "aa3310c5-bd35-4bef-b838-1722c7143cf5" (UID: "aa3310c5-bd35-4bef-b838-1722c7143cf5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:02.886745 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.886630 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-istio-token" pod "3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c" (UID: "3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:02.886965 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.886788 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3310c5-bd35-4bef-b838-1722c7143cf5-workload-certs" pod "aa3310c5-bd35-4bef-b838-1722c7143cf5" (UID: "aa3310c5-bd35-4bef-b838-1722c7143cf5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:02.886965 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.886817 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa3310c5-bd35-4bef-b838-1722c7143cf5-istio-podinfo" pod "aa3310c5-bd35-4bef-b838-1722c7143cf5" (UID: "aa3310c5-bd35-4bef-b838-1722c7143cf5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:53:02.887074 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.887047 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-credential-socket" pod "3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c" (UID: "3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:02.887756 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.887520 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa3310c5-bd35-4bef-b838-1722c7143cf5-istiod-ca-cert" pod "aa3310c5-bd35-4bef-b838-1722c7143cf5" (UID: "aa3310c5-bd35-4bef-b838-1722c7143cf5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:53:02.887756 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.887639 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-workload-certs" pod "3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c" (UID: "3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:02.887915 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.887759 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-istio-podinfo" pod "3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c" (UID: "3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:53:02.887915 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.887770 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-istio-envoy" pod "3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c" (UID: "3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:02.887915 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.887807 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-kube-api-access-jjv82" pod "3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c" (UID: "3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c"). InnerVolumeSpecName "kube-api-access-jjv82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:02.888241 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.888215 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa3310c5-bd35-4bef-b838-1722c7143cf5-kube-api-access-nbbw2" pod "aa3310c5-bd35-4bef-b838-1722c7143cf5" (UID: "aa3310c5-bd35-4bef-b838-1722c7143cf5"). InnerVolumeSpecName "kube-api-access-nbbw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:02.888909 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.888894 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3310c5-bd35-4bef-b838-1722c7143cf5-istio-envoy" pod "aa3310c5-bd35-4bef-b838-1722c7143cf5" (UID: "aa3310c5-bd35-4bef-b838-1722c7143cf5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:02.987835 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.987763 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85a61150-46ba-406c-b384-99565f653f15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85a61150-46ba-406c-b384-99565f653f15-istio-data\") pod \"85a61150-46ba-406c-b384-99565f653f15\" (UID: \"85a61150-46ba-406c-b384-99565f653f15\") " Nov 03 22:53:02.987835 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.987825 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85a61150-46ba-406c-b384-99565f653f15-istio-token\" (UniqueName: \"kubernetes.io/projected/85a61150-46ba-406c-b384-99565f653f15-istio-token\") pod \"85a61150-46ba-406c-b384-99565f653f15\" (UID: \"85a61150-46ba-406c-b384-99565f653f15\") " Nov 03 22:53:02.988059 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.987885 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-istio-data\") pod \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\" (UID: \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\") " Nov 03 22:53:02.988059 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.987903 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-istio-envoy\") pod \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\" (UID: \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\") " Nov 03 22:53:02.988059 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.987939 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85a61150-46ba-406c-b384-99565f653f15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85a61150-46ba-406c-b384-99565f653f15-istiod-ca-cert\") pod \"85a61150-46ba-406c-b384-99565f653f15\" (UID: \"85a61150-46ba-406c-b384-99565f653f15\") " Nov 03 22:53:02.988059 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.987975 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85a61150-46ba-406c-b384-99565f653f15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85a61150-46ba-406c-b384-99565f653f15-workload-certs\") pod \"85a61150-46ba-406c-b384-99565f653f15\" (UID: \"85a61150-46ba-406c-b384-99565f653f15\") " Nov 03 22:53:02.988059 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.988018 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-credential-socket\") pod \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\" (UID: \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\") " Nov 03 22:53:02.988333 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.988060 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-workload-certs\") pod \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\" (UID: \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\") " Nov 03 22:53:02.988333 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.988094 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85a61150-46ba-406c-b384-99565f653f15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a61150-46ba-406c-b384-99565f653f15-workload-socket\") pod \"85a61150-46ba-406c-b384-99565f653f15\" (UID: \"85a61150-46ba-406c-b384-99565f653f15\") " Nov 03 22:53:02.988333 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.988108 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85a61150-46ba-406c-b384-99565f653f15-istio-data" pod "85a61150-46ba-406c-b384-99565f653f15" (UID: "85a61150-46ba-406c-b384-99565f653f15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:02.988333 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.988127 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-istio-token\" (UniqueName: \"kubernetes.io/projected/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-istio-token\") pod \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\" (UID: \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\") " Nov 03 22:53:02.988333 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.988169 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-workload-socket\") pod \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\" (UID: \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\") " Nov 03 22:53:02.988333 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.988190 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-istio-data" pod "ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2" (UID: "ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:02.988333 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.988227 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-istio-podinfo\") pod \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\" (UID: \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\") " Nov 03 22:53:02.988333 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.988296 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85a61150-46ba-406c-b384-99565f653f15-kube-api-access-n9qkm\" (UniqueName: \"kubernetes.io/projected/85a61150-46ba-406c-b384-99565f653f15-kube-api-access-n9qkm\") pod \"85a61150-46ba-406c-b384-99565f653f15\" (UID: \"85a61150-46ba-406c-b384-99565f653f15\") " Nov 03 22:53:02.988333 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.988332 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85a61150-46ba-406c-b384-99565f653f15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85a61150-46ba-406c-b384-99565f653f15-istio-podinfo\") pod \"85a61150-46ba-406c-b384-99565f653f15\" (UID: \"85a61150-46ba-406c-b384-99565f653f15\") " Nov 03 22:53:02.988995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.988356 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85a61150-46ba-406c-b384-99565f653f15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a61150-46ba-406c-b384-99565f653f15-credential-socket\") pod \"85a61150-46ba-406c-b384-99565f653f15\" (UID: \"85a61150-46ba-406c-b384-99565f653f15\") " Nov 03 22:53:02.988995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.988293 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-credential-socket" pod "ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2" (UID: "ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:02.988995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.988446 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-istiod-ca-cert\") pod \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\" (UID: \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\") " Nov 03 22:53:02.988995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.988388 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-workload-certs" pod "ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2" (UID: "ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:02.988995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.988392 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85a61150-46ba-406c-b384-99565f653f15-workload-socket" pod "85a61150-46ba-406c-b384-99565f653f15" (UID: "85a61150-46ba-406c-b384-99565f653f15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:02.988995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.988414 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85a61150-46ba-406c-b384-99565f653f15-istiod-ca-cert" pod "85a61150-46ba-406c-b384-99565f653f15" (UID: "85a61150-46ba-406c-b384-99565f653f15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:53:02.988995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.988496 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-workload-socket" pod "ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2" (UID: "ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:02.988995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.988533 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-kube-api-access-lf5x7\" (UniqueName: \"kubernetes.io/projected/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-kube-api-access-lf5x7\") pod \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\" (UID: \"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2\") " Nov 03 22:53:02.988995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.988565 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85a61150-46ba-406c-b384-99565f653f15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85a61150-46ba-406c-b384-99565f653f15-istio-envoy\") pod \"85a61150-46ba-406c-b384-99565f653f15\" (UID: \"85a61150-46ba-406c-b384-99565f653f15\") " Nov 03 22:53:02.989464 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.989398 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-istiod-ca-cert" pod "ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2" (UID: "ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:53:02.989464 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.989404 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:02.989464 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.989432 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:02.989464 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.989447 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbbw2\" (UniqueName: \"kubernetes.io/projected/aa3310c5-bd35-4bef-b838-1722c7143cf5-kube-api-access-nbbw2\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:02.989464 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.989464 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:02.989720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.989479 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa3310c5-bd35-4bef-b838-1722c7143cf5-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:02.989720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.989494 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:02.989720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.989510 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa3310c5-bd35-4bef-b838-1722c7143cf5-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:02.989720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.989525 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa3310c5-bd35-4bef-b838-1722c7143cf5-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:02.989720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.989539 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa3310c5-bd35-4bef-b838-1722c7143cf5-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:02.989720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.989553 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:02.989720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.989569 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa3310c5-bd35-4bef-b838-1722c7143cf5-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:02.989720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.989582 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85a61150-46ba-406c-b384-99565f653f15-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:02.989720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.989595 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:02.989720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.989615 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:02.989720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.989629 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85a61150-46ba-406c-b384-99565f653f15-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:02.989720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.989644 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjv82\" (UniqueName: \"kubernetes.io/projected/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c-kube-api-access-jjv82\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:02.989720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.989645 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85a61150-46ba-406c-b384-99565f653f15-credential-socket" pod "85a61150-46ba-406c-b384-99565f653f15" (UID: "85a61150-46ba-406c-b384-99565f653f15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:02.989720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.989658 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:02.989720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.989674 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:02.989720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.989688 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a61150-46ba-406c-b384-99565f653f15-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:02.991535 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.989883 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85a61150-46ba-406c-b384-99565f653f15-workload-certs" pod "85a61150-46ba-406c-b384-99565f653f15" (UID: "85a61150-46ba-406c-b384-99565f653f15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:02.991535 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.991249 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-istio-envoy" pod "ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2" (UID: "ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:02.991535 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.991343 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85a61150-46ba-406c-b384-99565f653f15-istio-token" pod "85a61150-46ba-406c-b384-99565f653f15" (UID: "85a61150-46ba-406c-b384-99565f653f15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:02.991535 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.991361 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85a61150-46ba-406c-b384-99565f653f15-kube-api-access-n9qkm" pod "85a61150-46ba-406c-b384-99565f653f15" (UID: "85a61150-46ba-406c-b384-99565f653f15"). InnerVolumeSpecName "kube-api-access-n9qkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:02.991944 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.991565 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-istio-token" pod "ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2" (UID: "ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:02.991944 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.991633 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85a61150-46ba-406c-b384-99565f653f15-istio-podinfo" pod "85a61150-46ba-406c-b384-99565f653f15" (UID: "85a61150-46ba-406c-b384-99565f653f15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:53:02.992085 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.992017 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85a61150-46ba-406c-b384-99565f653f15-istio-envoy" pod "85a61150-46ba-406c-b384-99565f653f15" (UID: "85a61150-46ba-406c-b384-99565f653f15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:02.992246 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.992229 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-istio-podinfo" pod "ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2" (UID: "ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:53:02.993186 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:02.993160 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-kube-api-access-lf5x7" pod "ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2" (UID: "ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2"). InnerVolumeSpecName "kube-api-access-lf5x7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:03.066521 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.066481 5196 generic.go:356] "Generic (PLEG): container finished" podID="ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2" containerID="2faa7a1c976f7a0dc8b989b7d7e728293c13b15c6013e4222315347d36d2468a" exitCode=0 Nov 03 22:53:03.067079 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.066533 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" Nov 03 22:53:03.067079 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.066573 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" event={"ID":"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2","Type":"ContainerDied","Data":"2faa7a1c976f7a0dc8b989b7d7e728293c13b15c6013e4222315347d36d2468a"} Nov 03 22:53:03.067079 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.066634 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz" event={"ID":"ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2","Type":"ContainerDied","Data":"ee8a32c08bbc7223bc0775cd74a7c6b17fef67a8b5cc0ef99893d9741446b281"} Nov 03 22:53:03.067079 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.066665 5196 scope.go:122] "RemoveContainer" containerID="2faa7a1c976f7a0dc8b989b7d7e728293c13b15c6013e4222315347d36d2468a" Nov 03 22:53:03.071061 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.071031 5196 generic.go:356] "Generic (PLEG): container finished" podID="3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c" containerID="0e6e2cb774a6795ad59b759598e1e51148d8c5b389b3816409eff65ded329859" exitCode=0 Nov 03 22:53:03.071202 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.071189 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" Nov 03 22:53:03.071600 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.071352 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" event={"ID":"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c","Type":"ContainerDied","Data":"0e6e2cb774a6795ad59b759598e1e51148d8c5b389b3816409eff65ded329859"} Nov 03 22:53:03.071600 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.071438 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98" event={"ID":"3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c","Type":"ContainerDied","Data":"9d47737941cdca3d6a97c2342be7f9a93b1245be4de3fb3b2e09e0aa8f0d0559"} Nov 03 22:53:03.079897 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.073670 5196 generic.go:356] "Generic (PLEG): container finished" podID="aa3310c5-bd35-4bef-b838-1722c7143cf5" containerID="779c2ca89aa4a9f4f3fa65ee3dc70cf070ed23f5d9088c47c1d008705cf57287" exitCode=0 Nov 03 22:53:03.079897 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.073796 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" event={"ID":"aa3310c5-bd35-4bef-b838-1722c7143cf5","Type":"ContainerDied","Data":"779c2ca89aa4a9f4f3fa65ee3dc70cf070ed23f5d9088c47c1d008705cf57287"} Nov 03 22:53:03.079897 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.073833 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" event={"ID":"aa3310c5-bd35-4bef-b838-1722c7143cf5","Type":"ContainerDied","Data":"a2eb689fb6747f6a72b074f6587e3a0ad284188879a841eaabe0358f13b38053"} Nov 03 22:53:03.079897 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.073970 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk" Nov 03 22:53:03.084962 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.084794 5196 scope.go:122] "RemoveContainer" containerID="2faa7a1c976f7a0dc8b989b7d7e728293c13b15c6013e4222315347d36d2468a" Nov 03 22:53:03.091393 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:53:03.090686 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2faa7a1c976f7a0dc8b989b7d7e728293c13b15c6013e4222315347d36d2468a\": container with ID starting with 2faa7a1c976f7a0dc8b989b7d7e728293c13b15c6013e4222315347d36d2468a not found: ID does not exist" containerID="2faa7a1c976f7a0dc8b989b7d7e728293c13b15c6013e4222315347d36d2468a" Nov 03 22:53:03.091393 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.090730 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2faa7a1c976f7a0dc8b989b7d7e728293c13b15c6013e4222315347d36d2468a"} err="failed to get container status \"2faa7a1c976f7a0dc8b989b7d7e728293c13b15c6013e4222315347d36d2468a\": rpc error: code = NotFound desc = could not find container \"2faa7a1c976f7a0dc8b989b7d7e728293c13b15c6013e4222315347d36d2468a\": container with ID starting with 2faa7a1c976f7a0dc8b989b7d7e728293c13b15c6013e4222315347d36d2468a not found: ID does not exist" Nov 03 22:53:03.091393 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.090758 5196 scope.go:122] "RemoveContainer" containerID="0e6e2cb774a6795ad59b759598e1e51148d8c5b389b3816409eff65ded329859" Nov 03 22:53:03.091729 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.091590 5196 generic.go:356] "Generic (PLEG): container finished" podID="85a61150-46ba-406c-b384-99565f653f15" containerID="5af2747d92233beb4c13a39aa86448cace1e72168fd51edc1673ff7de72c9005" exitCode=0 Nov 03 22:53:03.091729 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.091649 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85a61150-46ba-406c-b384-99565f653f15-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:03.091729 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.091672 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:03.091729 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.091689 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:03.091729 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.091712 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9qkm\" (UniqueName: \"kubernetes.io/projected/85a61150-46ba-406c-b384-99565f653f15-kube-api-access-n9qkm\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:03.091729 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.091726 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85a61150-46ba-406c-b384-99565f653f15-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:03.092048 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.091743 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a61150-46ba-406c-b384-99565f653f15-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:03.092048 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.091761 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:03.092048 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.091780 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf5x7\" (UniqueName: \"kubernetes.io/projected/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-kube-api-access-lf5x7\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:03.092048 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.091795 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85a61150-46ba-406c-b384-99565f653f15-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:03.092048 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.091811 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85a61150-46ba-406c-b384-99565f653f15-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:03.092048 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.091826 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:03.092048 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.091874 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" event={"ID":"85a61150-46ba-406c-b384-99565f653f15","Type":"ContainerDied","Data":"5af2747d92233beb4c13a39aa86448cace1e72168fd51edc1673ff7de72c9005"} Nov 03 22:53:03.092048 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.091889 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" Nov 03 22:53:03.092048 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.091955 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x" event={"ID":"85a61150-46ba-406c-b384-99565f653f15","Type":"ContainerDied","Data":"a4281f489c0c462c60335ea43c56b41caa0ae0174243b46a103a034ad7a96434"} Nov 03 22:53:03.104678 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.104662 5196 scope.go:122] "RemoveContainer" containerID="0e6e2cb774a6795ad59b759598e1e51148d8c5b389b3816409eff65ded329859" Nov 03 22:53:03.104940 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:53:03.104921 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e6e2cb774a6795ad59b759598e1e51148d8c5b389b3816409eff65ded329859\": container with ID starting with 0e6e2cb774a6795ad59b759598e1e51148d8c5b389b3816409eff65ded329859 not found: ID does not exist" containerID="0e6e2cb774a6795ad59b759598e1e51148d8c5b389b3816409eff65ded329859" Nov 03 22:53:03.104998 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.104948 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e6e2cb774a6795ad59b759598e1e51148d8c5b389b3816409eff65ded329859"} err="failed to get container status \"0e6e2cb774a6795ad59b759598e1e51148d8c5b389b3816409eff65ded329859\": rpc error: code = NotFound desc = could not find container \"0e6e2cb774a6795ad59b759598e1e51148d8c5b389b3816409eff65ded329859\": container with ID starting with 0e6e2cb774a6795ad59b759598e1e51148d8c5b389b3816409eff65ded329859 not found: ID does not exist" Nov 03 22:53:03.104998 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.104965 5196 scope.go:122] "RemoveContainer" containerID="779c2ca89aa4a9f4f3fa65ee3dc70cf070ed23f5d9088c47c1d008705cf57287" Nov 03 22:53:03.113933 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.113916 5196 scope.go:122] "RemoveContainer" containerID="94185fe4b0d8ffd8765aacdeece9a13a1b59a1924ca7c60218f604135935965c" Nov 03 22:53:03.123335 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.123320 5196 scope.go:122] "RemoveContainer" containerID="779c2ca89aa4a9f4f3fa65ee3dc70cf070ed23f5d9088c47c1d008705cf57287" Nov 03 22:53:03.123583 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:53:03.123564 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"779c2ca89aa4a9f4f3fa65ee3dc70cf070ed23f5d9088c47c1d008705cf57287\": container with ID starting with 779c2ca89aa4a9f4f3fa65ee3dc70cf070ed23f5d9088c47c1d008705cf57287 not found: ID does not exist" containerID="779c2ca89aa4a9f4f3fa65ee3dc70cf070ed23f5d9088c47c1d008705cf57287" Nov 03 22:53:03.123660 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.123595 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"779c2ca89aa4a9f4f3fa65ee3dc70cf070ed23f5d9088c47c1d008705cf57287"} err="failed to get container status \"779c2ca89aa4a9f4f3fa65ee3dc70cf070ed23f5d9088c47c1d008705cf57287\": rpc error: code = NotFound desc = could not find container \"779c2ca89aa4a9f4f3fa65ee3dc70cf070ed23f5d9088c47c1d008705cf57287\": container with ID starting with 779c2ca89aa4a9f4f3fa65ee3dc70cf070ed23f5d9088c47c1d008705cf57287 not found: ID does not exist" Nov 03 22:53:03.123660 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.123618 5196 scope.go:122] "RemoveContainer" containerID="94185fe4b0d8ffd8765aacdeece9a13a1b59a1924ca7c60218f604135935965c" Nov 03 22:53:03.123880 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:53:03.123848 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94185fe4b0d8ffd8765aacdeece9a13a1b59a1924ca7c60218f604135935965c\": container with ID starting with 94185fe4b0d8ffd8765aacdeece9a13a1b59a1924ca7c60218f604135935965c not found: ID does not exist" containerID="94185fe4b0d8ffd8765aacdeece9a13a1b59a1924ca7c60218f604135935965c" Nov 03 22:53:03.123925 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.123889 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94185fe4b0d8ffd8765aacdeece9a13a1b59a1924ca7c60218f604135935965c"} err="failed to get container status \"94185fe4b0d8ffd8765aacdeece9a13a1b59a1924ca7c60218f604135935965c\": rpc error: code = NotFound desc = could not find container \"94185fe4b0d8ffd8765aacdeece9a13a1b59a1924ca7c60218f604135935965c\": container with ID starting with 94185fe4b0d8ffd8765aacdeece9a13a1b59a1924ca7c60218f604135935965c not found: ID does not exist" Nov 03 22:53:03.123925 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.123906 5196 scope.go:122] "RemoveContainer" containerID="5af2747d92233beb4c13a39aa86448cace1e72168fd51edc1673ff7de72c9005" Nov 03 22:53:03.138770 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.138731 5196 scope.go:122] "RemoveContainer" containerID="5af2747d92233beb4c13a39aa86448cace1e72168fd51edc1673ff7de72c9005" Nov 03 22:53:03.139207 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:53:03.139179 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5af2747d92233beb4c13a39aa86448cace1e72168fd51edc1673ff7de72c9005\": container with ID starting with 5af2747d92233beb4c13a39aa86448cace1e72168fd51edc1673ff7de72c9005 not found: ID does not exist" containerID="5af2747d92233beb4c13a39aa86448cace1e72168fd51edc1673ff7de72c9005" Nov 03 22:53:03.139319 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.139220 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5af2747d92233beb4c13a39aa86448cace1e72168fd51edc1673ff7de72c9005"} err="failed to get container status \"5af2747d92233beb4c13a39aa86448cace1e72168fd51edc1673ff7de72c9005\": rpc error: code = NotFound desc = could not find container \"5af2747d92233beb4c13a39aa86448cace1e72168fd51edc1673ff7de72c9005\": container with ID starting with 5af2747d92233beb4c13a39aa86448cace1e72168fd51edc1673ff7de72c9005 not found: ID does not exist" Nov 03 22:53:03.388690 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.388657 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk"] Nov 03 22:53:03.392330 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.392305 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nqmgk"] Nov 03 22:53:03.498546 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.498492 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x"] Nov 03 22:53:03.502469 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.502445 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vj28x"] Nov 03 22:53:03.613620 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.613590 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz"] Nov 03 22:53:03.617204 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.617182 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6jrgz"] Nov 03 22:53:03.728673 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.728595 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98"] Nov 03 22:53:03.734877 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.734827 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxr98"] Nov 03 22:53:03.968803 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:03.968759 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:04.200067 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:04.200029 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:04.346436 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:04.346404 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:04.815984 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:04.815950 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c" path="/var/lib/kubelet/pods/3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c/volumes" Nov 03 22:53:04.816392 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:04.816377 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85a61150-46ba-406c-b384-99565f653f15" path="/var/lib/kubelet/pods/85a61150-46ba-406c-b384-99565f653f15/volumes" Nov 03 22:53:04.816805 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:04.816791 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa3310c5-bd35-4bef-b838-1722c7143cf5" path="/var/lib/kubelet/pods/aa3310c5-bd35-4bef-b838-1722c7143cf5/volumes" Nov 03 22:53:04.817526 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:04.817511 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2" path="/var/lib/kubelet/pods/ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2/volumes" Nov 03 22:53:06.568111 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:06.568076 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:06.714674 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:06.714640 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:07.197334 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:07.197301 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:07.366517 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:07.366467 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:10.085619 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:10.085571 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:10.234556 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:10.234523 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:11.009444 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:11.009411 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ebffae22-bf688975f-smqn6"] Nov 03 22:53:11.009687 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:11.009647 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebffae22-bf688975f-smqn6" podUID="97debf33-6577-40bc-8819-838d638777cf" containerName="maas-api" containerID="cri-o://d2f4a4a95a552f058f63615190b577bce24ff9a95328c166478a6e1f5a737bcd" gracePeriod=30 Nov 03 22:53:11.113298 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:11.110591 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:11.155187 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:11.155156 5196 generic.go:356] "Generic (PLEG): container finished" podID="97debf33-6577-40bc-8819-838d638777cf" containerID="d2f4a4a95a552f058f63615190b577bce24ff9a95328c166478a6e1f5a737bcd" exitCode=0 Nov 03 22:53:11.155399 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:11.155379 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebffae22-bf688975f-smqn6" event={"ID":"97debf33-6577-40bc-8819-838d638777cf","Type":"ContainerDied","Data":"d2f4a4a95a552f058f63615190b577bce24ff9a95328c166478a6e1f5a737bcd"} Nov 03 22:53:11.391177 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:11.391149 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebffae22-bf688975f-smqn6" Nov 03 22:53:11.508278 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:11.508246 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97debf33-6577-40bc-8819-838d638777cf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97debf33-6577-40bc-8819-838d638777cf-openshift-service-ca\") pod \"97debf33-6577-40bc-8819-838d638777cf\" (UID: \"97debf33-6577-40bc-8819-838d638777cf\") " Nov 03 22:53:11.508602 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:11.508553 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97debf33-6577-40bc-8819-838d638777cf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/97debf33-6577-40bc-8819-838d638777cf-metrics-certs\") pod \"97debf33-6577-40bc-8819-838d638777cf\" (UID: \"97debf33-6577-40bc-8819-838d638777cf\") " Nov 03 22:53:11.508674 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:11.508615 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97debf33-6577-40bc-8819-838d638777cf-kube-api-access-mxs4v\" (UniqueName: \"kubernetes.io/projected/97debf33-6577-40bc-8819-838d638777cf-kube-api-access-mxs4v\") pod \"97debf33-6577-40bc-8819-838d638777cf\" (UID: \"97debf33-6577-40bc-8819-838d638777cf\") " Nov 03 22:53:11.508674 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:11.508641 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97debf33-6577-40bc-8819-838d638777cf-openshift-service-ca" pod "97debf33-6577-40bc-8819-838d638777cf" (UID: "97debf33-6577-40bc-8819-838d638777cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:53:11.508674 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:11.508668 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97debf33-6577-40bc-8819-838d638777cf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97debf33-6577-40bc-8819-838d638777cf-maas-api-tls\") pod \"97debf33-6577-40bc-8819-838d638777cf\" (UID: \"97debf33-6577-40bc-8819-838d638777cf\") " Nov 03 22:53:11.515485 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:11.509464 5196 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97debf33-6577-40bc-8819-838d638777cf-openshift-service-ca\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:11.515485 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:11.511905 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97debf33-6577-40bc-8819-838d638777cf-kube-api-access-mxs4v" pod "97debf33-6577-40bc-8819-838d638777cf" (UID: "97debf33-6577-40bc-8819-838d638777cf"). InnerVolumeSpecName "kube-api-access-mxs4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:11.515485 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:11.512171 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97debf33-6577-40bc-8819-838d638777cf-maas-api-tls" pod "97debf33-6577-40bc-8819-838d638777cf" (UID: "97debf33-6577-40bc-8819-838d638777cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:53:11.515485 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:11.513033 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97debf33-6577-40bc-8819-838d638777cf-metrics-certs" pod "97debf33-6577-40bc-8819-838d638777cf" (UID: "97debf33-6577-40bc-8819-838d638777cf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:53:11.611330 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:11.611248 5196 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97debf33-6577-40bc-8819-838d638777cf-metrics-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:11.611330 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:11.611281 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxs4v\" (UniqueName: \"kubernetes.io/projected/97debf33-6577-40bc-8819-838d638777cf-kube-api-access-mxs4v\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:11.611330 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:11.611292 5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97debf33-6577-40bc-8819-838d638777cf-maas-api-tls\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:12.058049 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:12.057950 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:12.163661 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:12.163623 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebffae22-bf688975f-smqn6" Nov 03 22:53:12.164131 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:12.163629 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ebffae22-bf688975f-smqn6" event={"ID":"97debf33-6577-40bc-8819-838d638777cf","Type":"ContainerDied","Data":"3a99714e94788118a6d917ed31c07ababad6f7983ef7f49bd491348ea7cbf8f9"} Nov 03 22:53:12.164131 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:12.163750 5196 scope.go:122] "RemoveContainer" containerID="d2f4a4a95a552f058f63615190b577bce24ff9a95328c166478a6e1f5a737bcd" Nov 03 22:53:12.339393 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:12.339358 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ebffae22-bf688975f-smqn6"] Nov 03 22:53:12.390339 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:12.390307 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ebffae22-bf688975f-smqn6"] Nov 03 22:53:12.816743 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:12.816705 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97debf33-6577-40bc-8819-838d638777cf" path="/var/lib/kubelet/pods/97debf33-6577-40bc-8819-838d638777cf/volumes" Nov 03 22:53:12.978757 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:12.978719 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:13.124934 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:13.124903 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:13.422869 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:13.422771 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:14.057273 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:14.057239 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:14.857466 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:14.857434 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:15.199550 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.199464 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:15.728066 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.728034 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2"] Nov 03 22:53:15.728577 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.728561 5196 cpu_manager.go:414] "containerMap: removing container" podUID="aa3310c5-bd35-4bef-b838-1722c7143cf5" containerName="istio-proxy" Nov 03 22:53:15.728577 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.728576 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa3310c5-bd35-4bef-b838-1722c7143cf5" containerName="istio-proxy" Nov 03 22:53:15.728739 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.728588 5196 cpu_manager.go:414] "containerMap: removing container" podUID="3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c" containerName="istio-proxy" Nov 03 22:53:15.728739 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.728594 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c" containerName="istio-proxy" Nov 03 22:53:15.728739 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.728616 5196 cpu_manager.go:414] "containerMap: removing container" podUID="85a61150-46ba-406c-b384-99565f653f15" containerName="istio-proxy" Nov 03 22:53:15.728739 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.728621 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85a61150-46ba-406c-b384-99565f653f15" containerName="istio-proxy" Nov 03 22:53:15.728739 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.728629 5196 cpu_manager.go:414] "containerMap: removing container" podUID="aa3310c5-bd35-4bef-b838-1722c7143cf5" containerName="istio-proxy" Nov 03 22:53:15.728739 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.728634 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa3310c5-bd35-4bef-b838-1722c7143cf5" containerName="istio-proxy" Nov 03 22:53:15.728739 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.728646 5196 cpu_manager.go:414] "containerMap: removing container" podUID="ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2" containerName="istio-proxy" Nov 03 22:53:15.728739 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.728650 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2" containerName="istio-proxy" Nov 03 22:53:15.728739 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.728660 5196 cpu_manager.go:414] "containerMap: removing container" podUID="97debf33-6577-40bc-8819-838d638777cf" containerName="maas-api" Nov 03 22:53:15.728739 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.728665 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97debf33-6577-40bc-8819-838d638777cf" containerName="maas-api" Nov 03 22:53:15.728739 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.728733 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa3310c5-bd35-4bef-b838-1722c7143cf5" containerName="istio-proxy" Nov 03 22:53:15.728739 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.728743 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="97debf33-6577-40bc-8819-838d638777cf" containerName="maas-api" Nov 03 22:53:15.729347 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.728754 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="3386155e-a5bb-4de6-aaf3-0cb3e1dacc4c" containerName="istio-proxy" Nov 03 22:53:15.729347 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.728761 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec68f4cb-7b6d-4693-9fbb-9a10cf2604a2" containerName="istio-proxy" Nov 03 22:53:15.729347 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.728769 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa3310c5-bd35-4bef-b838-1722c7143cf5" containerName="istio-proxy" Nov 03 22:53:15.729347 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.728783 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="85a61150-46ba-406c-b384-99565f653f15" containerName="istio-proxy" Nov 03 22:53:15.733883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.733842 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:53:15.760897 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.759705 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a97695c-47ae-48f9-ad2e-f97625e48234-istio-data\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2\" (UID: \"9a97695c-47ae-48f9-ad2e-f97625e48234\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:53:15.760897 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.759836 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a97695c-47ae-48f9-ad2e-f97625e48234-workload-certs\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2\" (UID: \"9a97695c-47ae-48f9-ad2e-f97625e48234\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:53:15.760897 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.759903 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a97695c-47ae-48f9-ad2e-f97625e48234-istiod-ca-cert\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2\" (UID: \"9a97695c-47ae-48f9-ad2e-f97625e48234\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:53:15.760897 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.759939 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a97695c-47ae-48f9-ad2e-f97625e48234-istio-podinfo\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2\" (UID: \"9a97695c-47ae-48f9-ad2e-f97625e48234\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:53:15.760897 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.759992 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmxhl\" (UniqueName: \"kubernetes.io/projected/9a97695c-47ae-48f9-ad2e-f97625e48234-kube-api-access-cmxhl\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2\" (UID: \"9a97695c-47ae-48f9-ad2e-f97625e48234\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:53:15.760897 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.760049 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a97695c-47ae-48f9-ad2e-f97625e48234-workload-socket\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2\" (UID: \"9a97695c-47ae-48f9-ad2e-f97625e48234\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:53:15.760897 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.760110 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a97695c-47ae-48f9-ad2e-f97625e48234-istio-token\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2\" (UID: \"9a97695c-47ae-48f9-ad2e-f97625e48234\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:53:15.760897 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.760151 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a97695c-47ae-48f9-ad2e-f97625e48234-credential-socket\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2\" (UID: \"9a97695c-47ae-48f9-ad2e-f97625e48234\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:53:15.760897 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.760189 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a97695c-47ae-48f9-ad2e-f97625e48234-istio-envoy\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2\" (UID: \"9a97695c-47ae-48f9-ad2e-f97625e48234\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:53:15.798882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.792926 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-aacab9-openshift-default-dockercfg-p5sjp\"" Nov 03 22:53:15.840989 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.833485 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:15.869881 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.869646 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2"] Nov 03 22:53:15.896907 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.871914 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a97695c-47ae-48f9-ad2e-f97625e48234-istio-token\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2\" (UID: \"9a97695c-47ae-48f9-ad2e-f97625e48234\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:53:15.896907 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.871978 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a97695c-47ae-48f9-ad2e-f97625e48234-credential-socket\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2\" (UID: \"9a97695c-47ae-48f9-ad2e-f97625e48234\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:53:15.896907 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.872022 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a97695c-47ae-48f9-ad2e-f97625e48234-istio-envoy\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2\" (UID: \"9a97695c-47ae-48f9-ad2e-f97625e48234\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:53:15.896907 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.872090 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a97695c-47ae-48f9-ad2e-f97625e48234-istio-data\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2\" (UID: \"9a97695c-47ae-48f9-ad2e-f97625e48234\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:53:15.896907 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.872181 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a97695c-47ae-48f9-ad2e-f97625e48234-workload-certs\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2\" (UID: \"9a97695c-47ae-48f9-ad2e-f97625e48234\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:53:15.896907 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.872221 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a97695c-47ae-48f9-ad2e-f97625e48234-istiod-ca-cert\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2\" (UID: \"9a97695c-47ae-48f9-ad2e-f97625e48234\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:53:15.896907 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.872268 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a97695c-47ae-48f9-ad2e-f97625e48234-istio-podinfo\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2\" (UID: \"9a97695c-47ae-48f9-ad2e-f97625e48234\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:53:15.896907 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.872323 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmxhl\" (UniqueName: \"kubernetes.io/projected/9a97695c-47ae-48f9-ad2e-f97625e48234-kube-api-access-cmxhl\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2\" (UID: \"9a97695c-47ae-48f9-ad2e-f97625e48234\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:53:15.896907 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.872403 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a97695c-47ae-48f9-ad2e-f97625e48234-workload-socket\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2\" (UID: \"9a97695c-47ae-48f9-ad2e-f97625e48234\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:53:15.896907 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.873462 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a97695c-47ae-48f9-ad2e-f97625e48234-credential-socket\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2\" (UID: \"9a97695c-47ae-48f9-ad2e-f97625e48234\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:53:15.896907 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.876341 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a97695c-47ae-48f9-ad2e-f97625e48234-istio-envoy\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2\" (UID: \"9a97695c-47ae-48f9-ad2e-f97625e48234\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:53:15.896907 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.876554 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a97695c-47ae-48f9-ad2e-f97625e48234-istio-data\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2\" (UID: \"9a97695c-47ae-48f9-ad2e-f97625e48234\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:53:15.896907 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.876975 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a97695c-47ae-48f9-ad2e-f97625e48234-workload-certs\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2\" (UID: \"9a97695c-47ae-48f9-ad2e-f97625e48234\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:53:15.896907 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.877493 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a97695c-47ae-48f9-ad2e-f97625e48234-istiod-ca-cert\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2\" (UID: \"9a97695c-47ae-48f9-ad2e-f97625e48234\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:53:15.896907 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.885431 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a97695c-47ae-48f9-ad2e-f97625e48234-workload-socket\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2\" (UID: \"9a97695c-47ae-48f9-ad2e-f97625e48234\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:53:15.896907 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.886356 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a97695c-47ae-48f9-ad2e-f97625e48234-istio-podinfo\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2\" (UID: \"9a97695c-47ae-48f9-ad2e-f97625e48234\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:53:15.951662 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.951588 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a97695c-47ae-48f9-ad2e-f97625e48234-istio-token\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2\" (UID: \"9a97695c-47ae-48f9-ad2e-f97625e48234\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:53:15.955549 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:15.955524 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmxhl\" (UniqueName: \"kubernetes.io/projected/9a97695c-47ae-48f9-ad2e-f97625e48234-kube-api-access-cmxhl\") pod \"e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2\" (UID: \"9a97695c-47ae-48f9-ad2e-f97625e48234\") " pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:53:16.050427 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:16.050341 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:53:16.470533 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:16.470494 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:16.509879 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:16.505908 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2"] Nov 03 22:53:16.562917 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:16.561130 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:53:16.562917 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:16.561238 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:53:16.562917 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:16.561283 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:53:17.229884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:17.229539 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" event={"ID":"9a97695c-47ae-48f9-ad2e-f97625e48234","Type":"ContainerStarted","Data":"6b274616ced7135811db4386afb9aa23676ae1e48aa028c422227d91e16644f8"} Nov 03 22:53:17.229884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:17.229612 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" event={"ID":"9a97695c-47ae-48f9-ad2e-f97625e48234","Type":"ContainerStarted","Data":"4a7df835939df77635801bd9626b502a0846d49dc4efa83e27ec78c83fee5355"} Nov 03 22:53:17.408716 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:17.408658 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" podStartSLOduration=2.408641874 podStartE2EDuration="2.408641874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:53:17.406269451 +0000 UTC m=+1826.715828768" watchObservedRunningTime="2026-11-03 22:53:17.408641874 +0000 UTC m=+1826.718201168" Nov 03 22:53:18.051348 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:18.051310 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:53:18.057241 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:18.057218 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:53:18.112674 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:18.112644 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:18.236952 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:18.236924 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:53:18.238146 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:18.238116 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:53:20.169184 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:20.169145 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:20.341867 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:20.341825 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:20.485164 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:20.485062 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:21.401955 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:21.401911 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:21.556121 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:21.556090 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:21.746271 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:21.746185 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:21.757254 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:21.757223 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s"] Nov 03 22:53:21.757460 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:21.757439 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" podUID="441335de-a4e8-4d44-8728-d16ba47b88ab" containerName="istio-proxy" containerID="cri-o://369bd997835415dec09a2505d632b236752b9e2ba47a4298047563ecb83c62c7" gracePeriod=30 Nov 03 22:53:22.945615 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:22.945583 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:23.266325 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:23.266232 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:23.751485 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:23.751444 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2"] Nov 03 22:53:23.755562 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:23.755543 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:53:23.820915 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:23.820877 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75a0d61a-c8e8-4925-941b-d729b4b21114-istio-envoy\") pod \"e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2\" (UID: \"75a0d61a-c8e8-4925-941b-d729b4b21114\") " pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:53:23.821109 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:23.820945 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75a0d61a-c8e8-4925-941b-d729b4b21114-workload-certs\") pod \"e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2\" (UID: \"75a0d61a-c8e8-4925-941b-d729b4b21114\") " pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:53:23.821109 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:23.820980 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75a0d61a-c8e8-4925-941b-d729b4b21114-istio-podinfo\") pod \"e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2\" (UID: \"75a0d61a-c8e8-4925-941b-d729b4b21114\") " pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:53:23.821259 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:23.821237 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75a0d61a-c8e8-4925-941b-d729b4b21114-istiod-ca-cert\") pod \"e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2\" (UID: \"75a0d61a-c8e8-4925-941b-d729b4b21114\") " pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:53:23.821388 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:23.821328 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a0d61a-c8e8-4925-941b-d729b4b21114-workload-socket\") pod \"e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2\" (UID: \"75a0d61a-c8e8-4925-941b-d729b4b21114\") " pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:53:23.821388 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:23.821364 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv6b2\" (UniqueName: \"kubernetes.io/projected/75a0d61a-c8e8-4925-941b-d729b4b21114-kube-api-access-tv6b2\") pod \"e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2\" (UID: \"75a0d61a-c8e8-4925-941b-d729b4b21114\") " pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:53:23.821495 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:23.821461 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75a0d61a-c8e8-4925-941b-d729b4b21114-istio-data\") pod \"e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2\" (UID: \"75a0d61a-c8e8-4925-941b-d729b4b21114\") " pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:53:23.821550 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:23.821528 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a0d61a-c8e8-4925-941b-d729b4b21114-credential-socket\") pod \"e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2\" (UID: \"75a0d61a-c8e8-4925-941b-d729b4b21114\") " pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:53:23.821630 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:23.821610 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75a0d61a-c8e8-4925-941b-d729b4b21114-istio-token\") pod \"e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2\" (UID: \"75a0d61a-c8e8-4925-941b-d729b4b21114\") " pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:53:23.832919 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:23.832898 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a91d2d-openshift-default-dockercfg-j6fb6\"" Nov 03 22:53:23.846589 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:23.846562 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:23.865189 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:23.865167 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2"] Nov 03 22:53:23.923268 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:23.923230 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75a0d61a-c8e8-4925-941b-d729b4b21114-istiod-ca-cert\") pod \"e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2\" (UID: \"75a0d61a-c8e8-4925-941b-d729b4b21114\") " pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:53:23.923480 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:23.923319 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a0d61a-c8e8-4925-941b-d729b4b21114-workload-socket\") pod \"e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2\" (UID: \"75a0d61a-c8e8-4925-941b-d729b4b21114\") " pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:53:23.923480 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:23.923360 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv6b2\" (UniqueName: \"kubernetes.io/projected/75a0d61a-c8e8-4925-941b-d729b4b21114-kube-api-access-tv6b2\") pod \"e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2\" (UID: \"75a0d61a-c8e8-4925-941b-d729b4b21114\") " pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:53:23.923480 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:23.923432 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75a0d61a-c8e8-4925-941b-d729b4b21114-istio-data\") pod \"e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2\" (UID: \"75a0d61a-c8e8-4925-941b-d729b4b21114\") " pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:53:23.923480 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:23.923486 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a0d61a-c8e8-4925-941b-d729b4b21114-credential-socket\") pod \"e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2\" (UID: \"75a0d61a-c8e8-4925-941b-d729b4b21114\") " pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:53:23.923775 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:23.923535 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75a0d61a-c8e8-4925-941b-d729b4b21114-istio-token\") pod \"e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2\" (UID: \"75a0d61a-c8e8-4925-941b-d729b4b21114\") " pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:53:23.923775 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:23.923598 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75a0d61a-c8e8-4925-941b-d729b4b21114-istio-envoy\") pod \"e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2\" (UID: \"75a0d61a-c8e8-4925-941b-d729b4b21114\") " pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:53:23.923775 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:23.923636 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75a0d61a-c8e8-4925-941b-d729b4b21114-workload-certs\") pod \"e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2\" (UID: \"75a0d61a-c8e8-4925-941b-d729b4b21114\") " pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:53:23.923775 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:23.923676 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75a0d61a-c8e8-4925-941b-d729b4b21114-istio-podinfo\") pod \"e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2\" (UID: \"75a0d61a-c8e8-4925-941b-d729b4b21114\") " pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:53:23.924032 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:23.923930 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75a0d61a-c8e8-4925-941b-d729b4b21114-istiod-ca-cert\") pod \"e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2\" (UID: \"75a0d61a-c8e8-4925-941b-d729b4b21114\") " pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:53:23.924263 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:23.924218 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75a0d61a-c8e8-4925-941b-d729b4b21114-istio-data\") pod \"e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2\" (UID: \"75a0d61a-c8e8-4925-941b-d729b4b21114\") " pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:53:23.924263 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:23.924245 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75a0d61a-c8e8-4925-941b-d729b4b21114-workload-certs\") pod \"e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2\" (UID: \"75a0d61a-c8e8-4925-941b-d729b4b21114\") " pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:53:23.924427 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:23.924303 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a0d61a-c8e8-4925-941b-d729b4b21114-credential-socket\") pod \"e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2\" (UID: \"75a0d61a-c8e8-4925-941b-d729b4b21114\") " pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:53:23.925529 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:23.925505 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a0d61a-c8e8-4925-941b-d729b4b21114-workload-socket\") pod \"e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2\" (UID: \"75a0d61a-c8e8-4925-941b-d729b4b21114\") " pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:53:23.928098 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:23.928066 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75a0d61a-c8e8-4925-941b-d729b4b21114-istio-podinfo\") pod \"e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2\" (UID: \"75a0d61a-c8e8-4925-941b-d729b4b21114\") " pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:53:23.929187 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:23.929168 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75a0d61a-c8e8-4925-941b-d729b4b21114-istio-envoy\") pod \"e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2\" (UID: \"75a0d61a-c8e8-4925-941b-d729b4b21114\") " pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:53:23.986230 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:23.986188 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75a0d61a-c8e8-4925-941b-d729b4b21114-istio-token\") pod \"e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2\" (UID: \"75a0d61a-c8e8-4925-941b-d729b4b21114\") " pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:53:23.986753 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:23.986305 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv6b2\" (UniqueName: \"kubernetes.io/projected/75a0d61a-c8e8-4925-941b-d729b4b21114-kube-api-access-tv6b2\") pod \"e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2\" (UID: \"75a0d61a-c8e8-4925-941b-d729b4b21114\") " pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:53:24.070206 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:24.070124 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:53:24.301272 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:24.301230 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:24.314441 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:24.314407 5196 patch_prober.go:29] interesting pod/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:53:24.314659 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:24.314463 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" podUID="441335de-a4e8-4d44-8728-d16ba47b88ab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:53:24.507364 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:24.507337 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2"] Nov 03 22:53:24.508998 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:53:24.508958 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75a0d61a_c8e8_4925_941b_d729b4b21114.slice/crio-c3b449549c5a8f0f844df63eb66f9fcb4e4432e000caffdc87f4f9b01fa38103 WatchSource:0}: Error finding container c3b449549c5a8f0f844df63eb66f9fcb4e4432e000caffdc87f4f9b01fa38103: Status 404 returned error can't find the container with id c3b449549c5a8f0f844df63eb66f9fcb4e4432e000caffdc87f4f9b01fa38103 Nov 03 22:53:24.512062 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:24.512023 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:53:24.512167 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:24.512100 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:53:24.512167 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:24.512140 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:53:24.738574 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:24.738534 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:25.062484 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:25.062448 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:25.304835 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:25.304798 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" event={"ID":"75a0d61a-c8e8-4925-941b-d729b4b21114","Type":"ContainerStarted","Data":"5cdece8ad478f12547a659c127254ce222eb9bd255aa13471ce7c1b26a104e84"} Nov 03 22:53:25.306492 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:25.306461 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" event={"ID":"75a0d61a-c8e8-4925-941b-d729b4b21114","Type":"ContainerStarted","Data":"c3b449549c5a8f0f844df63eb66f9fcb4e4432e000caffdc87f4f9b01fa38103"} Nov 03 22:53:25.447236 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:25.447181 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" podStartSLOduration=2.447161537 podStartE2EDuration="2.447161537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:53:25.445580039 +0000 UTC m=+1834.755139333" watchObservedRunningTime="2026-11-03 22:53:25.447161537 +0000 UTC m=+1834.756720899" Nov 03 22:53:25.919694 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:53:25.919657 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:53:26.070430 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:26.070393 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:53:26.076149 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:26.076125 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:53:26.312262 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:26.312165 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:53:26.313143 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:26.313126 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:53:27.039668 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:27.039645 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" Nov 03 22:53:27.183788 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:27.183756 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/441335de-a4e8-4d44-8728-d16ba47b88ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/441335de-a4e8-4d44-8728-d16ba47b88ab-istiod-ca-cert\") pod \"441335de-a4e8-4d44-8728-d16ba47b88ab\" (UID: \"441335de-a4e8-4d44-8728-d16ba47b88ab\") " Nov 03 22:53:27.184279 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:27.183798 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/441335de-a4e8-4d44-8728-d16ba47b88ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/441335de-a4e8-4d44-8728-d16ba47b88ab-istio-podinfo\") pod \"441335de-a4e8-4d44-8728-d16ba47b88ab\" (UID: \"441335de-a4e8-4d44-8728-d16ba47b88ab\") " Nov 03 22:53:27.184279 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:27.183891 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/441335de-a4e8-4d44-8728-d16ba47b88ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/441335de-a4e8-4d44-8728-d16ba47b88ab-workload-socket\") pod \"441335de-a4e8-4d44-8728-d16ba47b88ab\" (UID: \"441335de-a4e8-4d44-8728-d16ba47b88ab\") " Nov 03 22:53:27.184279 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:27.183933 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/441335de-a4e8-4d44-8728-d16ba47b88ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/441335de-a4e8-4d44-8728-d16ba47b88ab-workload-certs\") pod \"441335de-a4e8-4d44-8728-d16ba47b88ab\" (UID: \"441335de-a4e8-4d44-8728-d16ba47b88ab\") " Nov 03 22:53:27.184279 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:27.183999 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/441335de-a4e8-4d44-8728-d16ba47b88ab-istio-token\" (UniqueName: \"kubernetes.io/projected/441335de-a4e8-4d44-8728-d16ba47b88ab-istio-token\") pod \"441335de-a4e8-4d44-8728-d16ba47b88ab\" (UID: \"441335de-a4e8-4d44-8728-d16ba47b88ab\") " Nov 03 22:53:27.184279 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:27.184114 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/441335de-a4e8-4d44-8728-d16ba47b88ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/441335de-a4e8-4d44-8728-d16ba47b88ab-istio-envoy\") pod \"441335de-a4e8-4d44-8728-d16ba47b88ab\" (UID: \"441335de-a4e8-4d44-8728-d16ba47b88ab\") " Nov 03 22:53:27.184279 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:27.184190 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/441335de-a4e8-4d44-8728-d16ba47b88ab-kube-api-access-2c76b\" (UniqueName: \"kubernetes.io/projected/441335de-a4e8-4d44-8728-d16ba47b88ab-kube-api-access-2c76b\") pod \"441335de-a4e8-4d44-8728-d16ba47b88ab\" (UID: \"441335de-a4e8-4d44-8728-d16ba47b88ab\") " Nov 03 22:53:27.184279 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:27.184225 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/441335de-a4e8-4d44-8728-d16ba47b88ab-istiod-ca-cert" pod "441335de-a4e8-4d44-8728-d16ba47b88ab" (UID: "441335de-a4e8-4d44-8728-d16ba47b88ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:53:27.184279 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:27.184247 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/441335de-a4e8-4d44-8728-d16ba47b88ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/441335de-a4e8-4d44-8728-d16ba47b88ab-credential-socket\") pod \"441335de-a4e8-4d44-8728-d16ba47b88ab\" (UID: \"441335de-a4e8-4d44-8728-d16ba47b88ab\") " Nov 03 22:53:27.184745 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:27.184318 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/441335de-a4e8-4d44-8728-d16ba47b88ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/441335de-a4e8-4d44-8728-d16ba47b88ab-istio-data\") pod \"441335de-a4e8-4d44-8728-d16ba47b88ab\" (UID: \"441335de-a4e8-4d44-8728-d16ba47b88ab\") " Nov 03 22:53:27.184745 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:27.184632 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441335de-a4e8-4d44-8728-d16ba47b88ab-workload-certs" pod "441335de-a4e8-4d44-8728-d16ba47b88ab" (UID: "441335de-a4e8-4d44-8728-d16ba47b88ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:27.184908 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:27.184846 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441335de-a4e8-4d44-8728-d16ba47b88ab-istio-data" pod "441335de-a4e8-4d44-8728-d16ba47b88ab" (UID: "441335de-a4e8-4d44-8728-d16ba47b88ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:27.185012 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:27.184992 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/441335de-a4e8-4d44-8728-d16ba47b88ab-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:27.186555 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:27.185434 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441335de-a4e8-4d44-8728-d16ba47b88ab-credential-socket" pod "441335de-a4e8-4d44-8728-d16ba47b88ab" (UID: "441335de-a4e8-4d44-8728-d16ba47b88ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:27.186555 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:27.186164 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/441335de-a4e8-4d44-8728-d16ba47b88ab-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:27.187479 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:27.187443 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441335de-a4e8-4d44-8728-d16ba47b88ab-istio-envoy" pod "441335de-a4e8-4d44-8728-d16ba47b88ab" (UID: "441335de-a4e8-4d44-8728-d16ba47b88ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:27.187767 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:27.187741 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/441335de-a4e8-4d44-8728-d16ba47b88ab-istio-podinfo" pod "441335de-a4e8-4d44-8728-d16ba47b88ab" (UID: "441335de-a4e8-4d44-8728-d16ba47b88ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:53:27.188007 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:27.187979 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441335de-a4e8-4d44-8728-d16ba47b88ab-workload-socket" pod "441335de-a4e8-4d44-8728-d16ba47b88ab" (UID: "441335de-a4e8-4d44-8728-d16ba47b88ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:27.188704 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:27.188671 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/441335de-a4e8-4d44-8728-d16ba47b88ab-istio-token" pod "441335de-a4e8-4d44-8728-d16ba47b88ab" (UID: "441335de-a4e8-4d44-8728-d16ba47b88ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:27.188796 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:27.188730 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/441335de-a4e8-4d44-8728-d16ba47b88ab-kube-api-access-2c76b" pod "441335de-a4e8-4d44-8728-d16ba47b88ab" (UID: "441335de-a4e8-4d44-8728-d16ba47b88ab"). InnerVolumeSpecName "kube-api-access-2c76b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:27.287901 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:27.287798 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/441335de-a4e8-4d44-8728-d16ba47b88ab-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:27.287901 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:27.287826 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/441335de-a4e8-4d44-8728-d16ba47b88ab-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:27.287901 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:27.287838 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/441335de-a4e8-4d44-8728-d16ba47b88ab-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:27.287901 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:27.287852 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c76b\" (UniqueName: \"kubernetes.io/projected/441335de-a4e8-4d44-8728-d16ba47b88ab-kube-api-access-2c76b\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:27.287901 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:27.287900 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/441335de-a4e8-4d44-8728-d16ba47b88ab-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:27.288207 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:27.287909 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/441335de-a4e8-4d44-8728-d16ba47b88ab-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:27.288207 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:27.287917 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/441335de-a4e8-4d44-8728-d16ba47b88ab-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:53:27.322075 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:27.322036 5196 generic.go:356] "Generic (PLEG): container finished" podID="441335de-a4e8-4d44-8728-d16ba47b88ab" containerID="369bd997835415dec09a2505d632b236752b9e2ba47a4298047563ecb83c62c7" exitCode=0 Nov 03 22:53:27.322237 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:27.322094 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" Nov 03 22:53:27.322237 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:27.322118 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" event={"ID":"441335de-a4e8-4d44-8728-d16ba47b88ab","Type":"ContainerDied","Data":"369bd997835415dec09a2505d632b236752b9e2ba47a4298047563ecb83c62c7"} Nov 03 22:53:27.322237 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:27.322174 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s" event={"ID":"441335de-a4e8-4d44-8728-d16ba47b88ab","Type":"ContainerDied","Data":"1e30247c395a91a7f257ad399bffba98b498beeebbad417c5762f69e84d38224"} Nov 03 22:53:27.322237 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:27.322200 5196 scope.go:122] "RemoveContainer" containerID="369bd997835415dec09a2505d632b236752b9e2ba47a4298047563ecb83c62c7" Nov 03 22:53:27.334355 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:27.333132 5196 scope.go:122] "RemoveContainer" containerID="369bd997835415dec09a2505d632b236752b9e2ba47a4298047563ecb83c62c7" Nov 03 22:53:27.334355 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:53:27.333468 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"369bd997835415dec09a2505d632b236752b9e2ba47a4298047563ecb83c62c7\": container with ID starting with 369bd997835415dec09a2505d632b236752b9e2ba47a4298047563ecb83c62c7 not found: ID does not exist" containerID="369bd997835415dec09a2505d632b236752b9e2ba47a4298047563ecb83c62c7" Nov 03 22:53:27.334355 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:27.333501 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"369bd997835415dec09a2505d632b236752b9e2ba47a4298047563ecb83c62c7"} err="failed to get container status \"369bd997835415dec09a2505d632b236752b9e2ba47a4298047563ecb83c62c7\": rpc error: code = NotFound desc = could not find container \"369bd997835415dec09a2505d632b236752b9e2ba47a4298047563ecb83c62c7\": container with ID starting with 369bd997835415dec09a2505d632b236752b9e2ba47a4298047563ecb83c62c7 not found: ID does not exist" Nov 03 22:53:27.490709 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:27.490681 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s"] Nov 03 22:53:27.494790 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:27.494758 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ebffae22-openshift-default-78c8d7888f-b9x4s"] Nov 03 22:53:28.283297 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:28.283228 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:28.455179 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:28.455148 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:28.598055 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:28.598028 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:28.752604 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:28.752561 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:28.815574 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:28.815539 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="441335de-a4e8-4d44-8728-d16ba47b88ab" path="/var/lib/kubelet/pods/441335de-a4e8-4d44-8728-d16ba47b88ab/volumes" Nov 03 22:53:32.141960 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:32.141929 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:32.324012 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:32.323977 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:33.627670 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:33.627637 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:38.054552 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.054518 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp"] Nov 03 22:53:38.055312 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.055293 5196 cpu_manager.go:414] "containerMap: removing container" podUID="441335de-a4e8-4d44-8728-d16ba47b88ab" containerName="istio-proxy" Nov 03 22:53:38.055312 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.055312 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="441335de-a4e8-4d44-8728-d16ba47b88ab" containerName="istio-proxy" Nov 03 22:53:38.055490 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.055475 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="441335de-a4e8-4d44-8728-d16ba47b88ab" containerName="istio-proxy" Nov 03 22:53:38.060558 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.060541 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp" Nov 03 22:53:38.114588 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.114541 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-efce83-serving-cert\"" Nov 03 22:53:38.168957 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.168925 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp"] Nov 03 22:53:38.210491 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.210462 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:38.232055 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.232027 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4-maas-api-tls\") pod \"maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp\" (UID: \"ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp" Nov 03 22:53:38.232055 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.232056 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4-metrics-certs\") pod \"maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp\" (UID: \"ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp" Nov 03 22:53:38.232247 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.232086 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4-openshift-service-ca\") pod \"maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp\" (UID: \"ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp" Nov 03 22:53:38.232328 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.232304 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dkg8\" (UniqueName: \"kubernetes.io/projected/ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4-kube-api-access-6dkg8\") pod \"maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp\" (UID: \"ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp" Nov 03 22:53:38.286150 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.286120 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt"] Nov 03 22:53:38.290832 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.290806 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" Nov 03 22:53:38.333585 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.333494 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4-openshift-service-ca\") pod \"maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp\" (UID: \"ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp" Nov 03 22:53:38.333761 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.333628 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-istio-token\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt\" (UID: \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" Nov 03 22:53:38.333761 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.333674 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-workload-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt\" (UID: \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" Nov 03 22:53:38.333761 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.333712 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-istio-envoy\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt\" (UID: \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" Nov 03 22:53:38.333761 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.333735 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-istio-data\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt\" (UID: \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" Nov 03 22:53:38.334089 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.333765 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-istio-podinfo\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt\" (UID: \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" Nov 03 22:53:38.334089 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.333804 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dkg8\" (UniqueName: \"kubernetes.io/projected/ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4-kube-api-access-6dkg8\") pod \"maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp\" (UID: \"ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp" Nov 03 22:53:38.334089 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.333850 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-credential-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt\" (UID: \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" Nov 03 22:53:38.334089 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.333900 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx8lm\" (UniqueName: \"kubernetes.io/projected/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-kube-api-access-qx8lm\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt\" (UID: \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" Nov 03 22:53:38.334089 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.333934 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-istiod-ca-cert\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt\" (UID: \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" Nov 03 22:53:38.334089 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.334022 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-workload-certs\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt\" (UID: \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" Nov 03 22:53:38.334089 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.334061 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4-maas-api-tls\") pod \"maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp\" (UID: \"ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp" Nov 03 22:53:38.334089 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.334085 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4-metrics-certs\") pod \"maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp\" (UID: \"ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp" Nov 03 22:53:38.335358 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.335285 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4-openshift-service-ca\") pod \"maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp\" (UID: \"ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp" Nov 03 22:53:38.338271 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.338248 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4-maas-api-tls\") pod \"maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp\" (UID: \"ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp" Nov 03 22:53:38.338390 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.338373 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4-metrics-certs\") pod \"maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp\" (UID: \"ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp" Nov 03 22:53:38.344068 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.344047 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a9ab23c4-openshift-default-dockercfg-rxq5w\"" Nov 03 22:53:38.351412 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.351389 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt"] Nov 03 22:53:38.380624 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.380600 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:38.397558 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.397538 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dkg8\" (UniqueName: \"kubernetes.io/projected/ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4-kube-api-access-6dkg8\") pod \"maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp\" (UID: \"ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp" Nov 03 22:53:38.435087 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.435054 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-workload-certs\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt\" (UID: \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" Nov 03 22:53:38.435293 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.435180 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-istio-token\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt\" (UID: \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" Nov 03 22:53:38.435293 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.435218 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-workload-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt\" (UID: \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" Nov 03 22:53:38.435293 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.435254 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-istio-envoy\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt\" (UID: \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" Nov 03 22:53:38.435293 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.435282 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-istio-data\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt\" (UID: \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" Nov 03 22:53:38.435512 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.435312 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-istio-podinfo\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt\" (UID: \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" Nov 03 22:53:38.435512 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.435379 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-credential-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt\" (UID: \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" Nov 03 22:53:38.435902 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.435509 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx8lm\" (UniqueName: \"kubernetes.io/projected/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-kube-api-access-qx8lm\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt\" (UID: \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" Nov 03 22:53:38.435902 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.435565 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-istiod-ca-cert\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt\" (UID: \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" Nov 03 22:53:38.436439 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.436401 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-istio-data\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt\" (UID: \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" Nov 03 22:53:38.436634 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.436475 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-workload-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt\" (UID: \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" Nov 03 22:53:38.436738 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.436667 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-credential-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt\" (UID: \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" Nov 03 22:53:38.437194 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.437171 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-istiod-ca-cert\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt\" (UID: \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" Nov 03 22:53:38.437290 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.437211 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-workload-certs\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt\" (UID: \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" Nov 03 22:53:38.443669 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.443648 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-istio-podinfo\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt\" (UID: \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" Nov 03 22:53:38.446165 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.446138 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-istio-envoy\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt\" (UID: \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" Nov 03 22:53:38.498285 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.498250 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx8lm\" (UniqueName: \"kubernetes.io/projected/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-kube-api-access-qx8lm\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt\" (UID: \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" Nov 03 22:53:38.498285 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.498265 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-istio-token\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt\" (UID: \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" Nov 03 22:53:38.527415 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.527375 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:38.610218 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.610187 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" Nov 03 22:53:38.681715 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:38.681631 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp" Nov 03 22:53:39.044833 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:39.044756 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt"] Nov 03 22:53:39.082520 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:53:39.082490 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod625eb3b0_400d_4fa7_b5e7_80a08e17a42e.slice/crio-0ace3c0fdee35a380ff36952d7c921cd7f16ade10af93dedb7d67af13970fe32 WatchSource:0}: Error finding container 0ace3c0fdee35a380ff36952d7c921cd7f16ade10af93dedb7d67af13970fe32: Status 404 returned error can't find the container with id 0ace3c0fdee35a380ff36952d7c921cd7f16ade10af93dedb7d67af13970fe32 Nov 03 22:53:39.085393 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:39.085361 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:53:39.085466 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:39.085431 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:53:39.085466 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:39.085462 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:53:39.101204 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:39.101186 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp"] Nov 03 22:53:39.102606 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:53:39.102582 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee80b8b6_4f98_41da_b3d0_9f3c9121c0b4.slice/crio-c6f7bcdefb3753f30df5633a51807a87029d924073f15659ee9b529b141ad9c7 WatchSource:0}: Error finding container c6f7bcdefb3753f30df5633a51807a87029d924073f15659ee9b529b141ad9c7: Status 404 returned error can't find the container with id c6f7bcdefb3753f30df5633a51807a87029d924073f15659ee9b529b141ad9c7 Nov 03 22:53:39.405610 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:39.405573 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" event={"ID":"625eb3b0-400d-4fa7-b5e7-80a08e17a42e","Type":"ContainerStarted","Data":"90d15a672be3ebd5c87943d5230353a982153933c55a33009bdc1af10c7b5517"} Nov 03 22:53:39.405828 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:39.405633 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" event={"ID":"625eb3b0-400d-4fa7-b5e7-80a08e17a42e","Type":"ContainerStarted","Data":"0ace3c0fdee35a380ff36952d7c921cd7f16ade10af93dedb7d67af13970fe32"} Nov 03 22:53:39.407091 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:39.407067 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp" event={"ID":"ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4","Type":"ContainerStarted","Data":"c6f7bcdefb3753f30df5633a51807a87029d924073f15659ee9b529b141ad9c7"} Nov 03 22:53:39.530661 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:39.530599 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" podStartSLOduration=1.5305829370000001 podStartE2EDuration="1.530582937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:53:39.52671586 +0000 UTC m=+1848.836275153" watchObservedRunningTime="2026-11-03 22:53:39.530582937 +0000 UTC m=+1848.840142229" Nov 03 22:53:39.611007 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:39.610973 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" Nov 03 22:53:40.428194 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:40.428152 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp" event={"ID":"ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4","Type":"ContainerStarted","Data":"a9a942e4d022a7d71e95511b2e20233eb3848f98d194a0837fbfa236a5162fe7"} Nov 03 22:53:40.428696 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:40.428355 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp" Nov 03 22:53:40.550104 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:40.550049 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp" podStartSLOduration=1.698395702 podStartE2EDuration="2.550033486s" totalImagesPullingTime="851.637784ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:53:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:53:40.547431767 +0000 UTC m=+1849.856991061" watchObservedRunningTime="2026-11-03 22:53:40.550033486 +0000 UTC m=+1849.859592806" Nov 03 22:53:40.616489 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:40.616446 5196 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 03 22:53:40.616722 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:40.616518 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" podUID="625eb3b0-400d-4fa7-b5e7-80a08e17a42e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 03 22:53:41.614885 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:41.614239 5196 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 03 22:53:41.614885 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:41.614297 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" podUID="625eb3b0-400d-4fa7-b5e7-80a08e17a42e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 03 22:53:42.615905 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:42.611243 5196 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 03 22:53:42.615905 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:42.611310 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" podUID="625eb3b0-400d-4fa7-b5e7-80a08e17a42e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 03 22:53:43.267617 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:43.267587 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:43.611878 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:43.611268 5196 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 03 22:53:43.611878 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:43.611333 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" podUID="625eb3b0-400d-4fa7-b5e7-80a08e17a42e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 03 22:53:44.611290 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:44.611253 5196 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 03 22:53:44.611921 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:44.611889 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" podUID="625eb3b0-400d-4fa7-b5e7-80a08e17a42e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 03 22:53:45.614721 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:45.614694 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" Nov 03 22:53:45.615151 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:45.615064 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" Nov 03 22:53:45.615764 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:45.615742 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" Nov 03 22:53:46.439884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:46.439821 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp" Nov 03 22:53:48.012251 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:48.012213 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:52.904769 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:52.904733 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:53:55.918504 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:53:55.918463 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:53:57.616236 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:53:57.616198 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:02.067577 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:02.067530 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-26444f-8676cfdd67-tp9js"] Nov 03 22:54:02.071934 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:02.071916 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26444f-8676cfdd67-tp9js" Nov 03 22:54:02.092979 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:02.092955 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbe0a3aa-0540-40df-ba43-9196bf53b562-maas-api-tls\") pod \"maas-api-e2e-api-a-26444f-8676cfdd67-tp9js\" (UID: \"bbe0a3aa-0540-40df-ba43-9196bf53b562\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26444f-8676cfdd67-tp9js" Nov 03 22:54:02.093120 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:02.093108 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7cwc\" (UniqueName: \"kubernetes.io/projected/bbe0a3aa-0540-40df-ba43-9196bf53b562-kube-api-access-s7cwc\") pod \"maas-api-e2e-api-a-26444f-8676cfdd67-tp9js\" (UID: \"bbe0a3aa-0540-40df-ba43-9196bf53b562\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26444f-8676cfdd67-tp9js" Nov 03 22:54:02.093175 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:02.093155 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbe0a3aa-0540-40df-ba43-9196bf53b562-metrics-certs\") pod \"maas-api-e2e-api-a-26444f-8676cfdd67-tp9js\" (UID: \"bbe0a3aa-0540-40df-ba43-9196bf53b562\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26444f-8676cfdd67-tp9js" Nov 03 22:54:02.093276 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:02.093264 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbe0a3aa-0540-40df-ba43-9196bf53b562-openshift-service-ca\") pod \"maas-api-e2e-api-a-26444f-8676cfdd67-tp9js\" (UID: \"bbe0a3aa-0540-40df-ba43-9196bf53b562\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26444f-8676cfdd67-tp9js" Nov 03 22:54:02.127160 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:02.127129 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-26444f-serving-cert\"" Nov 03 22:54:02.131585 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:02.131562 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-26444f-8676cfdd67-tp9js"] Nov 03 22:54:02.195027 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:02.194998 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7cwc\" (UniqueName: \"kubernetes.io/projected/bbe0a3aa-0540-40df-ba43-9196bf53b562-kube-api-access-s7cwc\") pod \"maas-api-e2e-api-a-26444f-8676cfdd67-tp9js\" (UID: \"bbe0a3aa-0540-40df-ba43-9196bf53b562\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26444f-8676cfdd67-tp9js" Nov 03 22:54:02.195198 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:02.195082 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbe0a3aa-0540-40df-ba43-9196bf53b562-metrics-certs\") pod \"maas-api-e2e-api-a-26444f-8676cfdd67-tp9js\" (UID: \"bbe0a3aa-0540-40df-ba43-9196bf53b562\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26444f-8676cfdd67-tp9js" Nov 03 22:54:02.195198 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:02.195157 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbe0a3aa-0540-40df-ba43-9196bf53b562-openshift-service-ca\") pod \"maas-api-e2e-api-a-26444f-8676cfdd67-tp9js\" (UID: \"bbe0a3aa-0540-40df-ba43-9196bf53b562\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26444f-8676cfdd67-tp9js" Nov 03 22:54:02.195707 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:02.195466 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbe0a3aa-0540-40df-ba43-9196bf53b562-maas-api-tls\") pod \"maas-api-e2e-api-a-26444f-8676cfdd67-tp9js\" (UID: \"bbe0a3aa-0540-40df-ba43-9196bf53b562\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26444f-8676cfdd67-tp9js" Nov 03 22:54:02.195707 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:54:02.195631 5196 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-26444f-serving-cert: secret "maas-api-e2e-api-a-26444f-serving-cert" not found Nov 03 22:54:02.195808 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:54:02.195724 5196 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbe0a3aa-0540-40df-ba43-9196bf53b562-maas-api-tls podName:bbe0a3aa-0540-40df-ba43-9196bf53b562 nodeName:}" failed. No retries permitted until 2026-11-03 22:54:02.695703752 +0000 UTC m=+1872.005263037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bbe0a3aa-0540-40df-ba43-9196bf53b562-maas-api-tls") pod "maas-api-e2e-api-a-26444f-8676cfdd67-tp9js" (UID: "bbe0a3aa-0540-40df-ba43-9196bf53b562") : secret "maas-api-e2e-api-a-26444f-serving-cert" not found Nov 03 22:54:02.195981 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:02.195957 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbe0a3aa-0540-40df-ba43-9196bf53b562-openshift-service-ca\") pod \"maas-api-e2e-api-a-26444f-8676cfdd67-tp9js\" (UID: \"bbe0a3aa-0540-40df-ba43-9196bf53b562\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26444f-8676cfdd67-tp9js" Nov 03 22:54:02.198230 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:02.198208 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbe0a3aa-0540-40df-ba43-9196bf53b562-metrics-certs\") pod \"maas-api-e2e-api-a-26444f-8676cfdd67-tp9js\" (UID: \"bbe0a3aa-0540-40df-ba43-9196bf53b562\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26444f-8676cfdd67-tp9js" Nov 03 22:54:02.257670 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:02.257642 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7cwc\" (UniqueName: \"kubernetes.io/projected/bbe0a3aa-0540-40df-ba43-9196bf53b562-kube-api-access-s7cwc\") pod \"maas-api-e2e-api-a-26444f-8676cfdd67-tp9js\" (UID: \"bbe0a3aa-0540-40df-ba43-9196bf53b562\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26444f-8676cfdd67-tp9js" Nov 03 22:54:02.266174 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:02.266143 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:02.448034 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:02.448003 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:02.599106 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:02.599072 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:02.704398 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:02.704322 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbe0a3aa-0540-40df-ba43-9196bf53b562-maas-api-tls\") pod \"maas-api-e2e-api-a-26444f-8676cfdd67-tp9js\" (UID: \"bbe0a3aa-0540-40df-ba43-9196bf53b562\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26444f-8676cfdd67-tp9js" Nov 03 22:54:02.706961 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:02.706934 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbe0a3aa-0540-40df-ba43-9196bf53b562-maas-api-tls\") pod \"maas-api-e2e-api-a-26444f-8676cfdd67-tp9js\" (UID: \"bbe0a3aa-0540-40df-ba43-9196bf53b562\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26444f-8676cfdd67-tp9js" Nov 03 22:54:02.985764 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:02.985688 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26444f-8676cfdd67-tp9js" Nov 03 22:54:03.367440 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:03.367415 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-26444f-8676cfdd67-tp9js"] Nov 03 22:54:03.368449 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:54:03.368421 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbe0a3aa_0540_40df_ba43_9196bf53b562.slice/crio-36e860e8642033e7474b9214410d547e7c5f661d5ce7a52906dfbc0b2e59ada4 WatchSource:0}: Error finding container 36e860e8642033e7474b9214410d547e7c5f661d5ce7a52906dfbc0b2e59ada4: Status 404 returned error can't find the container with id 36e860e8642033e7474b9214410d547e7c5f661d5ce7a52906dfbc0b2e59ada4 Nov 03 22:54:03.582964 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:03.582925 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26444f-8676cfdd67-tp9js" event={"ID":"bbe0a3aa-0540-40df-ba43-9196bf53b562","Type":"ContainerStarted","Data":"36e860e8642033e7474b9214410d547e7c5f661d5ce7a52906dfbc0b2e59ada4"} Nov 03 22:54:04.590920 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:04.590886 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26444f-8676cfdd67-tp9js" event={"ID":"bbe0a3aa-0540-40df-ba43-9196bf53b562","Type":"ContainerStarted","Data":"1bd37d93403a9e2d5f54a641a1f0941f8b6410f23b150b9125b98d349aabfa12"} Nov 03 22:54:04.591315 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:04.590966 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26444f-8676cfdd67-tp9js" Nov 03 22:54:04.712255 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:04.712208 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26444f-8676cfdd67-tp9js" podStartSLOduration=1.899089214 podStartE2EDuration="2.712193026s" totalImagesPullingTime="813.103812ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:54:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:54:04.709659935 +0000 UTC m=+1874.019219230" watchObservedRunningTime="2026-11-03 22:54:04.712193026 +0000 UTC m=+1874.021752331" Nov 03 22:54:06.926577 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:06.926547 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:07.217397 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:07.217314 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:10.602470 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:10.602441 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26444f-8676cfdd67-tp9js" Nov 03 22:54:12.019225 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:12.019177 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:12.409201 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:12.409155 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:16.248026 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:16.247987 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6977-0fb5f5"] Nov 03 22:54:16.252587 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:16.252564 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6977-0fb5f5" Nov 03 22:54:16.308643 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:16.308611 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6977-0fb5f5"] Nov 03 22:54:16.356035 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:16.356008 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jgmv2\"" Nov 03 22:54:16.402762 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:16.402730 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wjct\" (UniqueName: \"kubernetes.io/projected/b17c2861-d3a0-4301-a6de-4b44c5f59f65-kube-api-access-7wjct\") pod \"test-curl-6977-0fb5f5\" (UID: \"b17c2861-d3a0-4301-a6de-4b44c5f59f65\") " pod="opendatahub/test-curl-6977-0fb5f5" Nov 03 22:54:16.504587 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:16.504510 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wjct\" (UniqueName: \"kubernetes.io/projected/b17c2861-d3a0-4301-a6de-4b44c5f59f65-kube-api-access-7wjct\") pod \"test-curl-6977-0fb5f5\" (UID: \"b17c2861-d3a0-4301-a6de-4b44c5f59f65\") " pod="opendatahub/test-curl-6977-0fb5f5" Nov 03 22:54:16.566960 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:16.566929 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wjct\" (UniqueName: \"kubernetes.io/projected/b17c2861-d3a0-4301-a6de-4b44c5f59f65-kube-api-access-7wjct\") pod \"test-curl-6977-0fb5f5\" (UID: \"b17c2861-d3a0-4301-a6de-4b44c5f59f65\") " pod="opendatahub/test-curl-6977-0fb5f5" Nov 03 22:54:16.866610 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:16.866582 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6977-0fb5f5" Nov 03 22:54:16.899520 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:16.899489 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:17.243950 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:54:17.243923 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb17c2861_d3a0_4301_a6de_4b44c5f59f65.slice/crio-51b989f6c8a13ba5e2caa1a37c3816bbf996a1a822b8f314a63dd3cb651731b0 WatchSource:0}: Error finding container 51b989f6c8a13ba5e2caa1a37c3816bbf996a1a822b8f314a63dd3cb651731b0: Status 404 returned error can't find the container with id 51b989f6c8a13ba5e2caa1a37c3816bbf996a1a822b8f314a63dd3cb651731b0 Nov 03 22:54:17.244701 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:17.244674 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6977-0fb5f5"] Nov 03 22:54:17.671925 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:17.671852 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6977-0fb5f5" event={"ID":"b17c2861-d3a0-4301-a6de-4b44c5f59f65","Type":"ContainerStarted","Data":"6345eb9a6a30b916a937c7e99a879fabd42ad1ba8657f8fd888b5bdb30edfef5"} Nov 03 22:54:17.671925 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:17.671932 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6977-0fb5f5" event={"ID":"b17c2861-d3a0-4301-a6de-4b44c5f59f65","Type":"ContainerStarted","Data":"51b989f6c8a13ba5e2caa1a37c3816bbf996a1a822b8f314a63dd3cb651731b0"} Nov 03 22:54:17.792217 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:17.792161 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6977-0fb5f5" podStartSLOduration=1.79214659 podStartE2EDuration="1.79214659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:54:17.790404751 +0000 UTC m=+1887.099964045" watchObservedRunningTime="2026-11-03 22:54:17.79214659 +0000 UTC m=+1887.101705884" Nov 03 22:54:19.281212 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:19.281163 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6977-0fb5f5"] Nov 03 22:54:19.286222 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:19.286191 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6977-0fb5f5"] Nov 03 22:54:19.684768 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:19.684721 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6977-0fb5f5" podUID="b17c2861-d3a0-4301-a6de-4b44c5f59f65" containerName="test-curl-6977-0fb5f5" containerID="cri-o://6345eb9a6a30b916a937c7e99a879fabd42ad1ba8657f8fd888b5bdb30edfef5" gracePeriod=2 Nov 03 22:54:19.736966 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:54:19.736932 5196 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6977-0fb5f5\" is forbidden: User \"system:node:ip-10-0-143-132.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-132.us-west-1.compute.internal' and this object" podUID="b17c2861-d3a0-4301-a6de-4b44c5f59f65" pod="opendatahub/test-curl-6977-0fb5f5" Nov 03 22:54:20.864314 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:54:20.864273 5196 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6977-0fb5f5\" is forbidden: User \"system:node:ip-10-0-143-132.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-132.us-west-1.compute.internal' and this object" podUID="b17c2861-d3a0-4301-a6de-4b44c5f59f65" pod="opendatahub/test-curl-6977-0fb5f5" Nov 03 22:54:21.955640 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:21.955579 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6977-0fb5f5" Nov 03 22:54:21.973118 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:21.973082 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b17c2861-d3a0-4301-a6de-4b44c5f59f65-kube-api-access-7wjct\" (UniqueName: \"kubernetes.io/projected/b17c2861-d3a0-4301-a6de-4b44c5f59f65-kube-api-access-7wjct\") pod \"b17c2861-d3a0-4301-a6de-4b44c5f59f65\" (UID: \"b17c2861-d3a0-4301-a6de-4b44c5f59f65\") " Nov 03 22:54:21.975301 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:21.975274 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b17c2861-d3a0-4301-a6de-4b44c5f59f65-kube-api-access-7wjct" pod "b17c2861-d3a0-4301-a6de-4b44c5f59f65" (UID: "b17c2861-d3a0-4301-a6de-4b44c5f59f65"). InnerVolumeSpecName "kube-api-access-7wjct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:22.000670 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:22.000602 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:22.074784 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:22.074755 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wjct\" (UniqueName: \"kubernetes.io/projected/b17c2861-d3a0-4301-a6de-4b44c5f59f65-kube-api-access-7wjct\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:54:22.151818 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:22.151781 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:22.704232 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:22.704191 5196 generic.go:356] "Generic (PLEG): container finished" podID="b17c2861-d3a0-4301-a6de-4b44c5f59f65" containerID="6345eb9a6a30b916a937c7e99a879fabd42ad1ba8657f8fd888b5bdb30edfef5" exitCode=137 Nov 03 22:54:22.704417 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:22.704239 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6977-0fb5f5" Nov 03 22:54:22.704417 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:22.704263 5196 scope.go:122] "RemoveContainer" containerID="6345eb9a6a30b916a937c7e99a879fabd42ad1ba8657f8fd888b5bdb30edfef5" Nov 03 22:54:22.714059 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:22.714044 5196 scope.go:122] "RemoveContainer" containerID="6345eb9a6a30b916a937c7e99a879fabd42ad1ba8657f8fd888b5bdb30edfef5" Nov 03 22:54:22.714313 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:54:22.714290 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6345eb9a6a30b916a937c7e99a879fabd42ad1ba8657f8fd888b5bdb30edfef5\": container with ID starting with 6345eb9a6a30b916a937c7e99a879fabd42ad1ba8657f8fd888b5bdb30edfef5 not found: ID does not exist" containerID="6345eb9a6a30b916a937c7e99a879fabd42ad1ba8657f8fd888b5bdb30edfef5" Nov 03 22:54:22.714385 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:22.714325 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6345eb9a6a30b916a937c7e99a879fabd42ad1ba8657f8fd888b5bdb30edfef5"} err="failed to get container status \"6345eb9a6a30b916a937c7e99a879fabd42ad1ba8657f8fd888b5bdb30edfef5\": rpc error: code = NotFound desc = could not find container \"6345eb9a6a30b916a937c7e99a879fabd42ad1ba8657f8fd888b5bdb30edfef5\": container with ID starting with 6345eb9a6a30b916a937c7e99a879fabd42ad1ba8657f8fd888b5bdb30edfef5 not found: ID does not exist" Nov 03 22:54:22.765328 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:54:22.765302 5196 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6977-0fb5f5\" is forbidden: User \"system:node:ip-10-0-143-132.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-132.us-west-1.compute.internal' and this object" podUID="b17c2861-d3a0-4301-a6de-4b44c5f59f65" pod="opendatahub/test-curl-6977-0fb5f5" Nov 03 22:54:22.815304 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:22.815276 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b17c2861-d3a0-4301-a6de-4b44c5f59f65" path="/var/lib/kubelet/pods/b17c2861-d3a0-4301-a6de-4b44c5f59f65/volumes" Nov 03 22:54:22.998381 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:22.998313 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6983-eb9b14"] Nov 03 22:54:22.998891 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:22.998848 5196 cpu_manager.go:414] "containerMap: removing container" podUID="b17c2861-d3a0-4301-a6de-4b44c5f59f65" containerName="test-curl-6977-0fb5f5" Nov 03 22:54:22.998954 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:22.998892 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b17c2861-d3a0-4301-a6de-4b44c5f59f65" containerName="test-curl-6977-0fb5f5" Nov 03 22:54:22.999008 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:22.998998 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="b17c2861-d3a0-4301-a6de-4b44c5f59f65" containerName="test-curl-6977-0fb5f5" Nov 03 22:54:23.003447 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:23.003432 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6983-eb9b14" Nov 03 22:54:23.058341 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:23.058316 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6983-eb9b14"] Nov 03 22:54:23.086105 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:23.086078 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt5l5\" (UniqueName: \"kubernetes.io/projected/9d629262-a501-4a4e-8eed-86096be183a9-kube-api-access-vt5l5\") pod \"test-curl-6983-eb9b14\" (UID: \"9d629262-a501-4a4e-8eed-86096be183a9\") " pod="opendatahub/test-curl-6983-eb9b14" Nov 03 22:54:23.105491 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:23.105468 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jgmv2\"" Nov 03 22:54:23.187633 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:23.187598 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt5l5\" (UniqueName: \"kubernetes.io/projected/9d629262-a501-4a4e-8eed-86096be183a9-kube-api-access-vt5l5\") pod \"test-curl-6983-eb9b14\" (UID: \"9d629262-a501-4a4e-8eed-86096be183a9\") " pod="opendatahub/test-curl-6983-eb9b14" Nov 03 22:54:23.254975 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:23.254907 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt5l5\" (UniqueName: \"kubernetes.io/projected/9d629262-a501-4a4e-8eed-86096be183a9-kube-api-access-vt5l5\") pod \"test-curl-6983-eb9b14\" (UID: \"9d629262-a501-4a4e-8eed-86096be183a9\") " pod="opendatahub/test-curl-6983-eb9b14" Nov 03 22:54:23.315608 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:23.315582 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6983-eb9b14" Nov 03 22:54:23.613901 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:23.613850 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:23.698585 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:23.698557 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6983-eb9b14"] Nov 03 22:54:23.699586 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:54:23.699554 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d629262_a501_4a4e_8eed_86096be183a9.slice/crio-7c2dabb38c4492791573ad9974a4259ab5f42e6db81878a36b9eb8ca22487c19 WatchSource:0}: Error finding container 7c2dabb38c4492791573ad9974a4259ab5f42e6db81878a36b9eb8ca22487c19: Status 404 returned error can't find the container with id 7c2dabb38c4492791573ad9974a4259ab5f42e6db81878a36b9eb8ca22487c19 Nov 03 22:54:23.713428 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:23.713404 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6983-eb9b14" event={"ID":"9d629262-a501-4a4e-8eed-86096be183a9","Type":"ContainerStarted","Data":"7c2dabb38c4492791573ad9974a4259ab5f42e6db81878a36b9eb8ca22487c19"} Nov 03 22:54:23.754225 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:23.754203 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:24.091638 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:24.091597 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f"] Nov 03 22:54:24.092048 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:24.091783 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" podUID="038354d8-e0d0-4f26-8370-183f575f5710" containerName="istio-proxy" containerID="cri-o://7718b72a0226f15298f2b384c46969941c3a29233f4f503e0b7d0f7a55e4c198" gracePeriod=30 Nov 03 22:54:24.721455 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:24.721422 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6983-eb9b14" event={"ID":"9d629262-a501-4a4e-8eed-86096be183a9","Type":"ContainerStarted","Data":"5757174fba1bc1fc5d0c0340eb8ba6e78e9d0ce605eb308981305d38b3f24cc2"} Nov 03 22:54:24.838473 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:24.838425 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6983-eb9b14" podStartSLOduration=2.838410663 podStartE2EDuration="2.838410663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:54:24.835733543 +0000 UTC m=+1894.145292858" watchObservedRunningTime="2026-11-03 22:54:24.838410663 +0000 UTC m=+1894.147969956" Nov 03 22:54:25.919986 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:54:25.919943 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:54:26.266018 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:26.265908 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6983-eb9b14"] Nov 03 22:54:26.270109 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:26.270084 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6983-eb9b14"] Nov 03 22:54:26.430142 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:26.430111 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6983-7bbfeb"] Nov 03 22:54:26.430684 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:26.430670 5196 cpu_manager.go:414] "containerMap: removing container" podUID="9d629262-a501-4a4e-8eed-86096be183a9" containerName="test-curl-6983-eb9b14" Nov 03 22:54:26.430744 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:26.430684 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d629262-a501-4a4e-8eed-86096be183a9" containerName="test-curl-6983-eb9b14" Nov 03 22:54:26.430791 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:26.430763 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d629262-a501-4a4e-8eed-86096be183a9" containerName="test-curl-6983-eb9b14" Nov 03 22:54:26.434249 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:26.434233 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6983-7bbfeb" Nov 03 22:54:26.489479 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:26.489450 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6983-7bbfeb"] Nov 03 22:54:26.532513 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:26.532440 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnsw6\" (UniqueName: \"kubernetes.io/projected/6007e5b5-8907-4d13-986e-ab96768e4443-kube-api-access-rnsw6\") pod \"test-curl-6983-7bbfeb\" (UID: \"6007e5b5-8907-4d13-986e-ab96768e4443\") " pod="opendatahub/test-curl-6983-7bbfeb" Nov 03 22:54:26.634381 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:26.634345 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnsw6\" (UniqueName: \"kubernetes.io/projected/6007e5b5-8907-4d13-986e-ab96768e4443-kube-api-access-rnsw6\") pod \"test-curl-6983-7bbfeb\" (UID: \"6007e5b5-8907-4d13-986e-ab96768e4443\") " pod="opendatahub/test-curl-6983-7bbfeb" Nov 03 22:54:26.696039 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:26.696009 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnsw6\" (UniqueName: \"kubernetes.io/projected/6007e5b5-8907-4d13-986e-ab96768e4443-kube-api-access-rnsw6\") pod \"test-curl-6983-7bbfeb\" (UID: \"6007e5b5-8907-4d13-986e-ab96768e4443\") " pod="opendatahub/test-curl-6983-7bbfeb" Nov 03 22:54:26.733745 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:26.733688 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6983-eb9b14" podUID="9d629262-a501-4a4e-8eed-86096be183a9" containerName="test-curl-6983-eb9b14" containerID="cri-o://5757174fba1bc1fc5d0c0340eb8ba6e78e9d0ce605eb308981305d38b3f24cc2" gracePeriod=2 Nov 03 22:54:26.747350 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:26.747322 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6983-7bbfeb" Nov 03 22:54:26.785770 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:54:26.785691 5196 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6983-eb9b14\" is forbidden: User \"system:node:ip-10-0-143-132.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-132.us-west-1.compute.internal' and this object" podUID="9d629262-a501-4a4e-8eed-86096be183a9" pod="opendatahub/test-curl-6983-eb9b14" Nov 03 22:54:27.158017 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:27.157988 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6983-7bbfeb"] Nov 03 22:54:27.158382 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:54:27.158064 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6007e5b5_8907_4d13_986e_ab96768e4443.slice/crio-ad873f9e1252f99ec75789a9570c3e3d04b7f9f02a5a49a73468b0c02712129d WatchSource:0}: Error finding container ad873f9e1252f99ec75789a9570c3e3d04b7f9f02a5a49a73468b0c02712129d: Status 404 returned error can't find the container with id ad873f9e1252f99ec75789a9570c3e3d04b7f9f02a5a49a73468b0c02712129d Nov 03 22:54:27.259721 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:27.259691 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:27.741518 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:27.741487 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6983-7bbfeb" event={"ID":"6007e5b5-8907-4d13-986e-ab96768e4443","Type":"ContainerStarted","Data":"b920090f4576c3aee261c8ceec9cf5ecf5e80cacb1e4a91f194787ef0d528638"} Nov 03 22:54:27.741695 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:27.741529 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6983-7bbfeb" event={"ID":"6007e5b5-8907-4d13-986e-ab96768e4443","Type":"ContainerStarted","Data":"ad873f9e1252f99ec75789a9570c3e3d04b7f9f02a5a49a73468b0c02712129d"} Nov 03 22:54:27.856453 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:27.856396 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6983-7bbfeb" podStartSLOduration=1.85637819 podStartE2EDuration="1.85637819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:54:27.855881552 +0000 UTC m=+1897.165440858" watchObservedRunningTime="2026-11-03 22:54:27.85637819 +0000 UTC m=+1897.165937484" Nov 03 22:54:28.991770 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:28.991708 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6983-eb9b14" Nov 03 22:54:29.072114 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.072082 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d629262-a501-4a4e-8eed-86096be183a9-kube-api-access-vt5l5\" (UniqueName: \"kubernetes.io/projected/9d629262-a501-4a4e-8eed-86096be183a9-kube-api-access-vt5l5\") pod \"9d629262-a501-4a4e-8eed-86096be183a9\" (UID: \"9d629262-a501-4a4e-8eed-86096be183a9\") " Nov 03 22:54:29.074208 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.074186 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d629262-a501-4a4e-8eed-86096be183a9-kube-api-access-vt5l5" pod "9d629262-a501-4a4e-8eed-86096be183a9" (UID: "9d629262-a501-4a4e-8eed-86096be183a9"). InnerVolumeSpecName "kube-api-access-vt5l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:29.174077 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.174046 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt5l5\" (UniqueName: \"kubernetes.io/projected/9d629262-a501-4a4e-8eed-86096be183a9-kube-api-access-vt5l5\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:54:29.351710 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.351684 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" Nov 03 22:54:29.373009 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.372986 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6983-7bbfeb"] Nov 03 22:54:29.379774 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.379749 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6983-7bbfeb"] Nov 03 22:54:29.478275 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.478231 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/038354d8-e0d0-4f26-8370-183f575f5710-kube-api-access-djz54\" (UniqueName: \"kubernetes.io/projected/038354d8-e0d0-4f26-8370-183f575f5710-kube-api-access-djz54\") pod \"038354d8-e0d0-4f26-8370-183f575f5710\" (UID: \"038354d8-e0d0-4f26-8370-183f575f5710\") " Nov 03 22:54:29.478458 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.478289 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/038354d8-e0d0-4f26-8370-183f575f5710-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/038354d8-e0d0-4f26-8370-183f575f5710-istio-envoy\") pod \"038354d8-e0d0-4f26-8370-183f575f5710\" (UID: \"038354d8-e0d0-4f26-8370-183f575f5710\") " Nov 03 22:54:29.478458 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.478347 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/038354d8-e0d0-4f26-8370-183f575f5710-istio-token\" (UniqueName: \"kubernetes.io/projected/038354d8-e0d0-4f26-8370-183f575f5710-istio-token\") pod \"038354d8-e0d0-4f26-8370-183f575f5710\" (UID: \"038354d8-e0d0-4f26-8370-183f575f5710\") " Nov 03 22:54:29.478595 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.478492 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/038354d8-e0d0-4f26-8370-183f575f5710-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/038354d8-e0d0-4f26-8370-183f575f5710-istiod-ca-cert\") pod \"038354d8-e0d0-4f26-8370-183f575f5710\" (UID: \"038354d8-e0d0-4f26-8370-183f575f5710\") " Nov 03 22:54:29.478658 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.478642 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/038354d8-e0d0-4f26-8370-183f575f5710-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/038354d8-e0d0-4f26-8370-183f575f5710-istio-podinfo\") pod \"038354d8-e0d0-4f26-8370-183f575f5710\" (UID: \"038354d8-e0d0-4f26-8370-183f575f5710\") " Nov 03 22:54:29.478713 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.478692 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/038354d8-e0d0-4f26-8370-183f575f5710-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/038354d8-e0d0-4f26-8370-183f575f5710-workload-certs\") pod \"038354d8-e0d0-4f26-8370-183f575f5710\" (UID: \"038354d8-e0d0-4f26-8370-183f575f5710\") " Nov 03 22:54:29.478772 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.478723 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/038354d8-e0d0-4f26-8370-183f575f5710-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/038354d8-e0d0-4f26-8370-183f575f5710-credential-socket\") pod \"038354d8-e0d0-4f26-8370-183f575f5710\" (UID: \"038354d8-e0d0-4f26-8370-183f575f5710\") " Nov 03 22:54:29.479032 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.479007 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/038354d8-e0d0-4f26-8370-183f575f5710-credential-socket" pod "038354d8-e0d0-4f26-8370-183f575f5710" (UID: "038354d8-e0d0-4f26-8370-183f575f5710"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:29.479219 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.479190 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/038354d8-e0d0-4f26-8370-183f575f5710-istiod-ca-cert" pod "038354d8-e0d0-4f26-8370-183f575f5710" (UID: "038354d8-e0d0-4f26-8370-183f575f5710"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:54:29.479327 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.479303 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/038354d8-e0d0-4f26-8370-183f575f5710-workload-certs" pod "038354d8-e0d0-4f26-8370-183f575f5710" (UID: "038354d8-e0d0-4f26-8370-183f575f5710"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:29.479413 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.479213 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/038354d8-e0d0-4f26-8370-183f575f5710-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/038354d8-e0d0-4f26-8370-183f575f5710-workload-socket\") pod \"038354d8-e0d0-4f26-8370-183f575f5710\" (UID: \"038354d8-e0d0-4f26-8370-183f575f5710\") " Nov 03 22:54:29.479413 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.479389 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/038354d8-e0d0-4f26-8370-183f575f5710-workload-socket" pod "038354d8-e0d0-4f26-8370-183f575f5710" (UID: "038354d8-e0d0-4f26-8370-183f575f5710"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:29.479525 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.479455 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/038354d8-e0d0-4f26-8370-183f575f5710-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/038354d8-e0d0-4f26-8370-183f575f5710-istio-data\") pod \"038354d8-e0d0-4f26-8370-183f575f5710\" (UID: \"038354d8-e0d0-4f26-8370-183f575f5710\") " Nov 03 22:54:29.479704 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.479685 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/038354d8-e0d0-4f26-8370-183f575f5710-istio-data" pod "038354d8-e0d0-4f26-8370-183f575f5710" (UID: "038354d8-e0d0-4f26-8370-183f575f5710"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:29.480299 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.480277 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/038354d8-e0d0-4f26-8370-183f575f5710-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:54:29.480364 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.480308 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/038354d8-e0d0-4f26-8370-183f575f5710-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:54:29.480364 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.480324 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/038354d8-e0d0-4f26-8370-183f575f5710-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:54:29.480364 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.480339 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/038354d8-e0d0-4f26-8370-183f575f5710-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:54:29.480364 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.480355 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/038354d8-e0d0-4f26-8370-183f575f5710-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:54:29.481041 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.481015 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/038354d8-e0d0-4f26-8370-183f575f5710-kube-api-access-djz54" pod "038354d8-e0d0-4f26-8370-183f575f5710" (UID: "038354d8-e0d0-4f26-8370-183f575f5710"). InnerVolumeSpecName "kube-api-access-djz54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:29.481320 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.481293 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/038354d8-e0d0-4f26-8370-183f575f5710-istio-token" pod "038354d8-e0d0-4f26-8370-183f575f5710" (UID: "038354d8-e0d0-4f26-8370-183f575f5710"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:29.481390 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.481340 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/038354d8-e0d0-4f26-8370-183f575f5710-istio-podinfo" pod "038354d8-e0d0-4f26-8370-183f575f5710" (UID: "038354d8-e0d0-4f26-8370-183f575f5710"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:54:29.481551 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.481532 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/038354d8-e0d0-4f26-8370-183f575f5710-istio-envoy" pod "038354d8-e0d0-4f26-8370-183f575f5710" (UID: "038354d8-e0d0-4f26-8370-183f575f5710"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:29.526460 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.526389 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6983-977fa4"] Nov 03 22:54:29.527011 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.526994 5196 cpu_manager.go:414] "containerMap: removing container" podUID="6007e5b5-8907-4d13-986e-ab96768e4443" containerName="test-curl-6983-7bbfeb" Nov 03 22:54:29.527011 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.527011 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6007e5b5-8907-4d13-986e-ab96768e4443" containerName="test-curl-6983-7bbfeb" Nov 03 22:54:29.527201 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.527031 5196 cpu_manager.go:414] "containerMap: removing container" podUID="038354d8-e0d0-4f26-8370-183f575f5710" containerName="istio-proxy" Nov 03 22:54:29.527201 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.527036 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="038354d8-e0d0-4f26-8370-183f575f5710" containerName="istio-proxy" Nov 03 22:54:29.527201 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.527134 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="6007e5b5-8907-4d13-986e-ab96768e4443" containerName="test-curl-6983-7bbfeb" Nov 03 22:54:29.527201 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.527149 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="038354d8-e0d0-4f26-8370-183f575f5710" containerName="istio-proxy" Nov 03 22:54:29.530746 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.530725 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6983-977fa4" Nov 03 22:54:29.581256 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.581230 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djz54\" (UniqueName: \"kubernetes.io/projected/038354d8-e0d0-4f26-8370-183f575f5710-kube-api-access-djz54\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:54:29.581256 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.581260 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/038354d8-e0d0-4f26-8370-183f575f5710-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:54:29.581437 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.581273 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/038354d8-e0d0-4f26-8370-183f575f5710-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:54:29.581437 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.581286 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/038354d8-e0d0-4f26-8370-183f575f5710-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:54:29.586988 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.586963 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6983-977fa4"] Nov 03 22:54:29.682826 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.682790 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhr42\" (UniqueName: \"kubernetes.io/projected/0346bacd-48a7-463d-b278-42b6d86f4b96-kube-api-access-nhr42\") pod \"test-curl-6983-977fa4\" (UID: \"0346bacd-48a7-463d-b278-42b6d86f4b96\") " pod="opendatahub/test-curl-6983-977fa4" Nov 03 22:54:29.757975 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.757943 5196 generic.go:356] "Generic (PLEG): container finished" podID="9d629262-a501-4a4e-8eed-86096be183a9" containerID="5757174fba1bc1fc5d0c0340eb8ba6e78e9d0ce605eb308981305d38b3f24cc2" exitCode=137 Nov 03 22:54:29.758128 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.757986 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6983-eb9b14" Nov 03 22:54:29.758128 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.758038 5196 scope.go:122] "RemoveContainer" containerID="5757174fba1bc1fc5d0c0340eb8ba6e78e9d0ce605eb308981305d38b3f24cc2" Nov 03 22:54:29.759521 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.759499 5196 generic.go:356] "Generic (PLEG): container finished" podID="038354d8-e0d0-4f26-8370-183f575f5710" containerID="7718b72a0226f15298f2b384c46969941c3a29233f4f503e0b7d0f7a55e4c198" exitCode=0 Nov 03 22:54:29.759605 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.759568 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" Nov 03 22:54:29.759605 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.759575 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" event={"ID":"038354d8-e0d0-4f26-8370-183f575f5710","Type":"ContainerDied","Data":"7718b72a0226f15298f2b384c46969941c3a29233f4f503e0b7d0f7a55e4c198"} Nov 03 22:54:29.759696 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.759631 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f" event={"ID":"038354d8-e0d0-4f26-8370-183f575f5710","Type":"ContainerDied","Data":"3d1e3ead188c1d3a750ddc5e7065c9416c7d65115c412a65d85c0f8e0ca462a8"} Nov 03 22:54:29.760074 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.760050 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6983-7bbfeb" podUID="6007e5b5-8907-4d13-986e-ab96768e4443" containerName="test-curl-6983-7bbfeb" containerID="cri-o://b920090f4576c3aee261c8ceec9cf5ecf5e80cacb1e4a91f194787ef0d528638" gracePeriod=2 Nov 03 22:54:29.768964 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.768944 5196 scope.go:122] "RemoveContainer" containerID="5757174fba1bc1fc5d0c0340eb8ba6e78e9d0ce605eb308981305d38b3f24cc2" Nov 03 22:54:29.769343 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:54:29.769248 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5757174fba1bc1fc5d0c0340eb8ba6e78e9d0ce605eb308981305d38b3f24cc2\": container with ID starting with 5757174fba1bc1fc5d0c0340eb8ba6e78e9d0ce605eb308981305d38b3f24cc2 not found: ID does not exist" containerID="5757174fba1bc1fc5d0c0340eb8ba6e78e9d0ce605eb308981305d38b3f24cc2" Nov 03 22:54:29.769343 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.769285 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5757174fba1bc1fc5d0c0340eb8ba6e78e9d0ce605eb308981305d38b3f24cc2"} err="failed to get container status \"5757174fba1bc1fc5d0c0340eb8ba6e78e9d0ce605eb308981305d38b3f24cc2\": rpc error: code = NotFound desc = could not find container \"5757174fba1bc1fc5d0c0340eb8ba6e78e9d0ce605eb308981305d38b3f24cc2\": container with ID starting with 5757174fba1bc1fc5d0c0340eb8ba6e78e9d0ce605eb308981305d38b3f24cc2 not found: ID does not exist" Nov 03 22:54:29.769343 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.769309 5196 scope.go:122] "RemoveContainer" containerID="7718b72a0226f15298f2b384c46969941c3a29233f4f503e0b7d0f7a55e4c198" Nov 03 22:54:29.778632 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.778614 5196 scope.go:122] "RemoveContainer" containerID="7718b72a0226f15298f2b384c46969941c3a29233f4f503e0b7d0f7a55e4c198" Nov 03 22:54:29.778923 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:54:29.778900 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7718b72a0226f15298f2b384c46969941c3a29233f4f503e0b7d0f7a55e4c198\": container with ID starting with 7718b72a0226f15298f2b384c46969941c3a29233f4f503e0b7d0f7a55e4c198 not found: ID does not exist" containerID="7718b72a0226f15298f2b384c46969941c3a29233f4f503e0b7d0f7a55e4c198" Nov 03 22:54:29.778993 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.778933 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7718b72a0226f15298f2b384c46969941c3a29233f4f503e0b7d0f7a55e4c198"} err="failed to get container status \"7718b72a0226f15298f2b384c46969941c3a29233f4f503e0b7d0f7a55e4c198\": rpc error: code = NotFound desc = could not find container \"7718b72a0226f15298f2b384c46969941c3a29233f4f503e0b7d0f7a55e4c198\": container with ID starting with 7718b72a0226f15298f2b384c46969941c3a29233f4f503e0b7d0f7a55e4c198 not found: ID does not exist" Nov 03 22:54:29.784657 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.784637 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhr42\" (UniqueName: \"kubernetes.io/projected/0346bacd-48a7-463d-b278-42b6d86f4b96-kube-api-access-nhr42\") pod \"test-curl-6983-977fa4\" (UID: \"0346bacd-48a7-463d-b278-42b6d86f4b96\") " pod="opendatahub/test-curl-6983-977fa4" Nov 03 22:54:29.846687 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.846667 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhr42\" (UniqueName: \"kubernetes.io/projected/0346bacd-48a7-463d-b278-42b6d86f4b96-kube-api-access-nhr42\") pod \"test-curl-6983-977fa4\" (UID: \"0346bacd-48a7-463d-b278-42b6d86f4b96\") " pod="opendatahub/test-curl-6983-977fa4" Nov 03 22:54:29.929407 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.929371 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f"] Nov 03 22:54:29.932339 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:29.932312 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-tww6f"] Nov 03 22:54:29.979110 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:54:29.979064 5196 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6983-7bbfeb\" is forbidden: User \"system:node:ip-10-0-143-132.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-132.us-west-1.compute.internal' and this object" podUID="6007e5b5-8907-4d13-986e-ab96768e4443" pod="opendatahub/test-curl-6983-7bbfeb" Nov 03 22:54:30.031088 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:54:30.031007 5196 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6983-eb9b14\" is forbidden: User \"system:node:ip-10-0-143-132.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-132.us-west-1.compute.internal' and this object" podUID="9d629262-a501-4a4e-8eed-86096be183a9" pod="opendatahub/test-curl-6983-eb9b14" Nov 03 22:54:30.144011 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:30.143985 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6983-977fa4" Nov 03 22:54:30.523553 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:30.523525 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6983-977fa4"] Nov 03 22:54:30.524167 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:54:30.524134 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0346bacd_48a7_463d_b278_42b6d86f4b96.slice/crio-460e32d4c70b65bf743a17687710490ee3289ec1a5b468ed8d694088ab7d241d WatchSource:0}: Error finding container 460e32d4c70b65bf743a17687710490ee3289ec1a5b468ed8d694088ab7d241d: Status 404 returned error can't find the container with id 460e32d4c70b65bf743a17687710490ee3289ec1a5b468ed8d694088ab7d241d Nov 03 22:54:30.771034 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:30.770961 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6983-977fa4" event={"ID":"0346bacd-48a7-463d-b278-42b6d86f4b96","Type":"ContainerStarted","Data":"74d729058a3f473e1cd5e58bab285c4fb474b6ecd09285d1135dffcc8a75daa1"} Nov 03 22:54:30.771034 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:30.770998 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6983-977fa4" event={"ID":"0346bacd-48a7-463d-b278-42b6d86f4b96","Type":"ContainerStarted","Data":"460e32d4c70b65bf743a17687710490ee3289ec1a5b468ed8d694088ab7d241d"} Nov 03 22:54:30.816142 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:30.816114 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="038354d8-e0d0-4f26-8370-183f575f5710" path="/var/lib/kubelet/pods/038354d8-e0d0-4f26-8370-183f575f5710/volumes" Nov 03 22:54:30.816552 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:30.816537 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d629262-a501-4a4e-8eed-86096be183a9" path="/var/lib/kubelet/pods/9d629262-a501-4a4e-8eed-86096be183a9/volumes" Nov 03 22:54:30.886750 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:30.886704 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6983-977fa4" podStartSLOduration=1.886691747 podStartE2EDuration="1.886691747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:54:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:54:30.885760564 +0000 UTC m=+1900.195319856" watchObservedRunningTime="2026-11-03 22:54:30.886691747 +0000 UTC m=+1900.196251040" Nov 03 22:54:30.937583 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:54:30.937552 5196 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6983-eb9b14\" is forbidden: User \"system:node:ip-10-0-143-132.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-132.us-west-1.compute.internal' and this object" podUID="9d629262-a501-4a4e-8eed-86096be183a9" pod="opendatahub/test-curl-6983-eb9b14" Nov 03 22:54:30.989398 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:54:30.989364 5196 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6983-7bbfeb\" is forbidden: User \"system:node:ip-10-0-143-132.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-132.us-west-1.compute.internal' and this object" podUID="6007e5b5-8907-4d13-986e-ab96768e4443" pod="opendatahub/test-curl-6983-7bbfeb" Nov 03 22:54:32.057071 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:32.057038 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6983-7bbfeb" Nov 03 22:54:32.073809 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:32.073776 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:32.109132 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:54:32.109096 5196 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6983-7bbfeb\" is forbidden: User \"system:node:ip-10-0-143-132.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-132.us-west-1.compute.internal' and this object" podUID="6007e5b5-8907-4d13-986e-ab96768e4443" pod="opendatahub/test-curl-6983-7bbfeb" Nov 03 22:54:32.226327 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:32.226298 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6007e5b5-8907-4d13-986e-ab96768e4443-kube-api-access-rnsw6\" (UniqueName: \"kubernetes.io/projected/6007e5b5-8907-4d13-986e-ab96768e4443-kube-api-access-rnsw6\") pod \"6007e5b5-8907-4d13-986e-ab96768e4443\" (UID: \"6007e5b5-8907-4d13-986e-ab96768e4443\") " Nov 03 22:54:32.228510 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:32.228485 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6007e5b5-8907-4d13-986e-ab96768e4443-kube-api-access-rnsw6" pod "6007e5b5-8907-4d13-986e-ab96768e4443" (UID: "6007e5b5-8907-4d13-986e-ab96768e4443"). InnerVolumeSpecName "kube-api-access-rnsw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:32.240818 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:32.240781 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:32.305989 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:32.305917 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6983-977fa4"] Nov 03 22:54:32.328195 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:32.328162 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnsw6\" (UniqueName: \"kubernetes.io/projected/6007e5b5-8907-4d13-986e-ab96768e4443-kube-api-access-rnsw6\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:54:32.332710 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:32.332688 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6983-977fa4"] Nov 03 22:54:32.459133 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:32.459098 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6983-bebf84"] Nov 03 22:54:32.459800 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:32.459751 5196 cpu_manager.go:414] "containerMap: removing container" podUID="0346bacd-48a7-463d-b278-42b6d86f4b96" containerName="test-curl-6983-977fa4" Nov 03 22:54:32.459800 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:32.459773 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0346bacd-48a7-463d-b278-42b6d86f4b96" containerName="test-curl-6983-977fa4" Nov 03 22:54:32.460002 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:32.459893 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="0346bacd-48a7-463d-b278-42b6d86f4b96" containerName="test-curl-6983-977fa4" Nov 03 22:54:32.463182 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:32.463164 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6983-bebf84" Nov 03 22:54:32.519597 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:32.519567 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6983-bebf84"] Nov 03 22:54:32.522534 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:32.522509 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:32.628492 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:54:32.628440 5196 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6983-7bbfeb\" is forbidden: User \"system:node:ip-10-0-143-132.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-132.us-west-1.compute.internal' and this object" podUID="6007e5b5-8907-4d13-986e-ab96768e4443" pod="opendatahub/test-curl-6983-7bbfeb" Nov 03 22:54:32.632426 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:32.632387 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n8rk\" (UniqueName: \"kubernetes.io/projected/defd206f-fcd0-46cc-9f24-4ec2e2f50aae-kube-api-access-2n8rk\") pod \"test-curl-6983-bebf84\" (UID: \"defd206f-fcd0-46cc-9f24-4ec2e2f50aae\") " pod="opendatahub/test-curl-6983-bebf84" Nov 03 22:54:32.699067 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:32.699033 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:32.734210 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:32.734179 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n8rk\" (UniqueName: \"kubernetes.io/projected/defd206f-fcd0-46cc-9f24-4ec2e2f50aae-kube-api-access-2n8rk\") pod \"test-curl-6983-bebf84\" (UID: \"defd206f-fcd0-46cc-9f24-4ec2e2f50aae\") " pod="opendatahub/test-curl-6983-bebf84" Nov 03 22:54:32.793433 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:32.793392 5196 generic.go:356] "Generic (PLEG): container finished" podID="6007e5b5-8907-4d13-986e-ab96768e4443" containerID="b920090f4576c3aee261c8ceec9cf5ecf5e80cacb1e4a91f194787ef0d528638" exitCode=137 Nov 03 22:54:32.793631 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:32.793457 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6983-7bbfeb" Nov 03 22:54:32.793631 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:32.793572 5196 scope.go:122] "RemoveContainer" containerID="b920090f4576c3aee261c8ceec9cf5ecf5e80cacb1e4a91f194787ef0d528638" Nov 03 22:54:32.794021 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:32.793992 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6983-977fa4" podUID="0346bacd-48a7-463d-b278-42b6d86f4b96" containerName="test-curl-6983-977fa4" containerID="cri-o://74d729058a3f473e1cd5e58bab285c4fb474b6ecd09285d1135dffcc8a75daa1" gracePeriod=2 Nov 03 22:54:32.802469 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:32.802434 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n8rk\" (UniqueName: \"kubernetes.io/projected/defd206f-fcd0-46cc-9f24-4ec2e2f50aae-kube-api-access-2n8rk\") pod \"test-curl-6983-bebf84\" (UID: \"defd206f-fcd0-46cc-9f24-4ec2e2f50aae\") " pod="opendatahub/test-curl-6983-bebf84" Nov 03 22:54:32.817029 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:32.816994 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6007e5b5-8907-4d13-986e-ab96768e4443" path="/var/lib/kubelet/pods/6007e5b5-8907-4d13-986e-ab96768e4443/volumes" Nov 03 22:54:32.822093 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:32.822017 5196 scope.go:122] "RemoveContainer" containerID="b920090f4576c3aee261c8ceec9cf5ecf5e80cacb1e4a91f194787ef0d528638" Nov 03 22:54:32.822547 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:54:32.822509 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b920090f4576c3aee261c8ceec9cf5ecf5e80cacb1e4a91f194787ef0d528638\": container with ID starting with b920090f4576c3aee261c8ceec9cf5ecf5e80cacb1e4a91f194787ef0d528638 not found: ID does not exist" containerID="b920090f4576c3aee261c8ceec9cf5ecf5e80cacb1e4a91f194787ef0d528638" Nov 03 22:54:32.822630 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:32.822542 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b920090f4576c3aee261c8ceec9cf5ecf5e80cacb1e4a91f194787ef0d528638"} err="failed to get container status \"b920090f4576c3aee261c8ceec9cf5ecf5e80cacb1e4a91f194787ef0d528638\": rpc error: code = NotFound desc = could not find container \"b920090f4576c3aee261c8ceec9cf5ecf5e80cacb1e4a91f194787ef0d528638\": container with ID starting with b920090f4576c3aee261c8ceec9cf5ecf5e80cacb1e4a91f194787ef0d528638 not found: ID does not exist" Nov 03 22:54:32.845641 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:54:32.845601 5196 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6983-7bbfeb\" is forbidden: User \"system:node:ip-10-0-143-132.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-132.us-west-1.compute.internal' and this object" podUID="6007e5b5-8907-4d13-986e-ab96768e4443" pod="opendatahub/test-curl-6983-7bbfeb" Nov 03 22:54:32.858164 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:32.858136 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:32.897498 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:54:32.897431 5196 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6983-7bbfeb\" is forbidden: User \"system:node:ip-10-0-143-132.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-132.us-west-1.compute.internal' and this object" podUID="6007e5b5-8907-4d13-986e-ab96768e4443" pod="opendatahub/test-curl-6983-7bbfeb" Nov 03 22:54:33.075196 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:33.075169 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6983-bebf84" Nov 03 22:54:33.465665 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:33.465633 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6983-bebf84"] Nov 03 22:54:33.467326 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:54:33.467297 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddefd206f_fcd0_46cc_9f24_4ec2e2f50aae.slice/crio-3666ace4c3b85b8bce277cbf701b77243f04d65e2f8f9e0ab6b4c1b6434b4060 WatchSource:0}: Error finding container 3666ace4c3b85b8bce277cbf701b77243f04d65e2f8f9e0ab6b4c1b6434b4060: Status 404 returned error can't find the container with id 3666ace4c3b85b8bce277cbf701b77243f04d65e2f8f9e0ab6b4c1b6434b4060 Nov 03 22:54:33.805200 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:33.805121 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6983-bebf84" event={"ID":"defd206f-fcd0-46cc-9f24-4ec2e2f50aae","Type":"ContainerStarted","Data":"4d40605712657ba7ff27a1876c193a8de1a138f4221bf0213f551e9829e124d8"} Nov 03 22:54:33.805200 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:33.805160 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6983-bebf84" event={"ID":"defd206f-fcd0-46cc-9f24-4ec2e2f50aae","Type":"ContainerStarted","Data":"3666ace4c3b85b8bce277cbf701b77243f04d65e2f8f9e0ab6b4c1b6434b4060"} Nov 03 22:54:33.920851 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:33.920791 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6983-bebf84" podStartSLOduration=1.920773719 podStartE2EDuration="1.920773719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:54:33.919419484 +0000 UTC m=+1903.228978776" watchObservedRunningTime="2026-11-03 22:54:33.920773719 +0000 UTC m=+1903.230333013" Nov 03 22:54:33.971603 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:54:33.971567 5196 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6983-7bbfeb\" is forbidden: User \"system:node:ip-10-0-143-132.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-132.us-west-1.compute.internal' and this object" podUID="6007e5b5-8907-4d13-986e-ab96768e4443" pod="opendatahub/test-curl-6983-7bbfeb" Nov 03 22:54:35.068564 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:35.068494 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6983-977fa4" Nov 03 22:54:35.188445 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:35.188408 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0346bacd-48a7-463d-b278-42b6d86f4b96-kube-api-access-nhr42\" (UniqueName: \"kubernetes.io/projected/0346bacd-48a7-463d-b278-42b6d86f4b96-kube-api-access-nhr42\") pod \"0346bacd-48a7-463d-b278-42b6d86f4b96\" (UID: \"0346bacd-48a7-463d-b278-42b6d86f4b96\") " Nov 03 22:54:35.190522 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:35.190495 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0346bacd-48a7-463d-b278-42b6d86f4b96-kube-api-access-nhr42" pod "0346bacd-48a7-463d-b278-42b6d86f4b96" (UID: "0346bacd-48a7-463d-b278-42b6d86f4b96"). InnerVolumeSpecName "kube-api-access-nhr42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:35.290200 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:35.290167 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhr42\" (UniqueName: \"kubernetes.io/projected/0346bacd-48a7-463d-b278-42b6d86f4b96-kube-api-access-nhr42\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:54:35.395582 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:35.392949 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6983-bebf84"] Nov 03 22:54:35.396359 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:35.396332 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6983-bebf84"] Nov 03 22:54:35.822378 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:35.822298 5196 generic.go:356] "Generic (PLEG): container finished" podID="0346bacd-48a7-463d-b278-42b6d86f4b96" containerID="74d729058a3f473e1cd5e58bab285c4fb474b6ecd09285d1135dffcc8a75daa1" exitCode=137 Nov 03 22:54:35.822378 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:35.822338 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6983-977fa4" Nov 03 22:54:35.822597 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:35.822392 5196 scope.go:122] "RemoveContainer" containerID="74d729058a3f473e1cd5e58bab285c4fb474b6ecd09285d1135dffcc8a75daa1" Nov 03 22:54:35.822716 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:35.822671 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6983-bebf84" podUID="defd206f-fcd0-46cc-9f24-4ec2e2f50aae" containerName="test-curl-6983-bebf84" containerID="cri-o://4d40605712657ba7ff27a1876c193a8de1a138f4221bf0213f551e9829e124d8" gracePeriod=2 Nov 03 22:54:35.832096 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:35.832075 5196 scope.go:122] "RemoveContainer" containerID="74d729058a3f473e1cd5e58bab285c4fb474b6ecd09285d1135dffcc8a75daa1" Nov 03 22:54:35.832370 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:54:35.832348 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74d729058a3f473e1cd5e58bab285c4fb474b6ecd09285d1135dffcc8a75daa1\": container with ID starting with 74d729058a3f473e1cd5e58bab285c4fb474b6ecd09285d1135dffcc8a75daa1 not found: ID does not exist" containerID="74d729058a3f473e1cd5e58bab285c4fb474b6ecd09285d1135dffcc8a75daa1" Nov 03 22:54:35.832446 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:35.832383 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74d729058a3f473e1cd5e58bab285c4fb474b6ecd09285d1135dffcc8a75daa1"} err="failed to get container status \"74d729058a3f473e1cd5e58bab285c4fb474b6ecd09285d1135dffcc8a75daa1\": rpc error: code = NotFound desc = could not find container \"74d729058a3f473e1cd5e58bab285c4fb474b6ecd09285d1135dffcc8a75daa1\": container with ID starting with 74d729058a3f473e1cd5e58bab285c4fb474b6ecd09285d1135dffcc8a75daa1 not found: ID does not exist" Nov 03 22:54:35.875083 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:54:35.875041 5196 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6983-bebf84\" is forbidden: User \"system:node:ip-10-0-143-132.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-132.us-west-1.compute.internal' and this object" podUID="defd206f-fcd0-46cc-9f24-4ec2e2f50aae" pod="opendatahub/test-curl-6983-bebf84" Nov 03 22:54:35.927239 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:54:35.927202 5196 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6983-977fa4\" is forbidden: User \"system:node:ip-10-0-143-132.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-132.us-west-1.compute.internal' and this object" podUID="0346bacd-48a7-463d-b278-42b6d86f4b96" pod="opendatahub/test-curl-6983-977fa4" Nov 03 22:54:35.979121 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:54:35.979082 5196 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6983-bebf84\" is forbidden: User \"system:node:ip-10-0-143-132.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-132.us-west-1.compute.internal' and this object" podUID="defd206f-fcd0-46cc-9f24-4ec2e2f50aae" pod="opendatahub/test-curl-6983-bebf84" Nov 03 22:54:36.052442 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:36.052412 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:36.232881 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:36.232374 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:36.407119 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:36.407067 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:36.816701 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:36.816665 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0346bacd-48a7-463d-b278-42b6d86f4b96" path="/var/lib/kubelet/pods/0346bacd-48a7-463d-b278-42b6d86f4b96/volumes" Nov 03 22:54:37.623756 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:37.623720 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:37.774639 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:37.774604 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:38.013233 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:38.013201 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:38.103512 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:38.103484 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6983-bebf84" Nov 03 22:54:38.228658 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:38.228578 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/defd206f-fcd0-46cc-9f24-4ec2e2f50aae-kube-api-access-2n8rk\" (UniqueName: \"kubernetes.io/projected/defd206f-fcd0-46cc-9f24-4ec2e2f50aae-kube-api-access-2n8rk\") pod \"defd206f-fcd0-46cc-9f24-4ec2e2f50aae\" (UID: \"defd206f-fcd0-46cc-9f24-4ec2e2f50aae\") " Nov 03 22:54:38.230770 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:38.230742 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/defd206f-fcd0-46cc-9f24-4ec2e2f50aae-kube-api-access-2n8rk" pod "defd206f-fcd0-46cc-9f24-4ec2e2f50aae" (UID: "defd206f-fcd0-46cc-9f24-4ec2e2f50aae"). InnerVolumeSpecName "kube-api-access-2n8rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:38.330310 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:38.330278 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n8rk\" (UniqueName: \"kubernetes.io/projected/defd206f-fcd0-46cc-9f24-4ec2e2f50aae-kube-api-access-2n8rk\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:54:38.815190 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:38.815156 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="defd206f-fcd0-46cc-9f24-4ec2e2f50aae" path="/var/lib/kubelet/pods/defd206f-fcd0-46cc-9f24-4ec2e2f50aae/volumes" Nov 03 22:54:38.843348 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:38.843323 5196 generic.go:356] "Generic (PLEG): container finished" podID="defd206f-fcd0-46cc-9f24-4ec2e2f50aae" containerID="4d40605712657ba7ff27a1876c193a8de1a138f4221bf0213f551e9829e124d8" exitCode=137 Nov 03 22:54:38.843452 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:38.843362 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6983-bebf84" Nov 03 22:54:38.843452 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:38.843380 5196 scope.go:122] "RemoveContainer" containerID="4d40605712657ba7ff27a1876c193a8de1a138f4221bf0213f551e9829e124d8" Nov 03 22:54:38.852647 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:38.852628 5196 scope.go:122] "RemoveContainer" containerID="4d40605712657ba7ff27a1876c193a8de1a138f4221bf0213f551e9829e124d8" Nov 03 22:54:38.852957 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:54:38.852932 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d40605712657ba7ff27a1876c193a8de1a138f4221bf0213f551e9829e124d8\": container with ID starting with 4d40605712657ba7ff27a1876c193a8de1a138f4221bf0213f551e9829e124d8 not found: ID does not exist" containerID="4d40605712657ba7ff27a1876c193a8de1a138f4221bf0213f551e9829e124d8" Nov 03 22:54:38.853048 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:38.852968 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d40605712657ba7ff27a1876c193a8de1a138f4221bf0213f551e9829e124d8"} err="failed to get container status \"4d40605712657ba7ff27a1876c193a8de1a138f4221bf0213f551e9829e124d8\": rpc error: code = NotFound desc = could not find container \"4d40605712657ba7ff27a1876c193a8de1a138f4221bf0213f551e9829e124d8\": container with ID starting with 4d40605712657ba7ff27a1876c193a8de1a138f4221bf0213f551e9829e124d8 not found: ID does not exist" Nov 03 22:54:43.044023 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:43.043971 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:47.851671 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:47.851636 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:50.818446 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:50.818420 5196 patch_prober.go:29] interesting pod/e2e-api-a-26444f-openshift-default-695897496f-mcdxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:54:50.818794 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:50.818479 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" podUID="0474bb77-105a-4009-9f7e-2e19a4068fc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 22:54:50.955826 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:50.955757 5196 generic.go:356] "Generic (PLEG): container finished" podID="0474bb77-105a-4009-9f7e-2e19a4068fc9" containerID="fb80c20770872da17ebaa7493076c9373630cbd3d9aa70d087b5070180aafdb9" exitCode=0 Nov 03 22:54:50.955826 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:50.955812 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" event={"ID":"0474bb77-105a-4009-9f7e-2e19a4068fc9","Type":"ContainerDied","Data":"fb80c20770872da17ebaa7493076c9373630cbd3d9aa70d087b5070180aafdb9"} Nov 03 22:54:50.956209 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:50.956195 5196 scope.go:122] "RemoveContainer" containerID="fb80c20770872da17ebaa7493076c9373630cbd3d9aa70d087b5070180aafdb9" Nov 03 22:54:50.957904 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:50.957851 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:54:50.957987 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:50.957946 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:54:50.957987 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:50.957979 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:54:51.964424 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:51.964392 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" event={"ID":"0474bb77-105a-4009-9f7e-2e19a4068fc9","Type":"ContainerStarted","Data":"af6e3c7027e1ea8724e0f847c1964538b3c227ca425714dd6c0791dee7bd29f3"} Nov 03 22:54:52.321882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:52.321761 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:54:52.322042 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:52.322006 5196 patch_prober.go:29] interesting pod/e2e-api-a-26444f-openshift-default-695897496f-mcdxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:54:52.322087 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:52.322063 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" podUID="0474bb77-105a-4009-9f7e-2e19a4068fc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 22:54:52.455365 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:52.455323 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w"] Nov 03 22:54:52.456011 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:52.455992 5196 cpu_manager.go:414] "containerMap: removing container" podUID="defd206f-fcd0-46cc-9f24-4ec2e2f50aae" containerName="test-curl-6983-bebf84" Nov 03 22:54:52.456154 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:52.456012 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="defd206f-fcd0-46cc-9f24-4ec2e2f50aae" containerName="test-curl-6983-bebf84" Nov 03 22:54:52.456154 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:52.456137 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="defd206f-fcd0-46cc-9f24-4ec2e2f50aae" containerName="test-curl-6983-bebf84" Nov 03 22:54:52.459435 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:52.459414 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w" Nov 03 22:54:52.517829 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:52.517742 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-aacab9-serving-cert\"" Nov 03 22:54:52.521920 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:52.521833 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w"] Nov 03 22:54:52.547441 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:52.547411 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w\" (UID: \"9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w" Nov 03 22:54:52.547554 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:52.547532 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdvzh\" (UniqueName: \"kubernetes.io/projected/9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7-kube-api-access-qdvzh\") pod \"maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w\" (UID: \"9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w" Nov 03 22:54:52.547616 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:52.547601 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w\" (UID: \"9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w" Nov 03 22:54:52.547663 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:52.547645 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w\" (UID: \"9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w" Nov 03 22:54:52.648997 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:52.648966 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w\" (UID: \"9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w" Nov 03 22:54:52.649192 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:52.649019 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdvzh\" (UniqueName: \"kubernetes.io/projected/9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7-kube-api-access-qdvzh\") pod \"maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w\" (UID: \"9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w" Nov 03 22:54:52.649192 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:52.649061 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w\" (UID: \"9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w" Nov 03 22:54:52.649192 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:52.649090 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w\" (UID: \"9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w" Nov 03 22:54:52.649377 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:54:52.649196 5196 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-aacab9-serving-cert: secret "maas-api-e2e-shared-b-w4-aacab9-serving-cert" not found Nov 03 22:54:52.649377 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:54:52.649259 5196 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7-maas-api-tls podName:9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7 nodeName:}" failed. No retries permitted until 2026-11-03 22:54:53.149243466 +0000 UTC m=+1922.458802755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7-maas-api-tls") pod "maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w" (UID: "9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7") : secret "maas-api-e2e-shared-b-w4-aacab9-serving-cert" not found Nov 03 22:54:52.649731 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:52.649694 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w\" (UID: \"9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w" Nov 03 22:54:52.652058 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:52.652031 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w\" (UID: \"9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w" Nov 03 22:54:52.652506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:52.652456 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:52.718880 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:52.718567 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdvzh\" (UniqueName: \"kubernetes.io/projected/9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7-kube-api-access-qdvzh\") pod \"maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w\" (UID: \"9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w" Nov 03 22:54:52.812551 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:52.812488 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:52.960309 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:52.960227 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:53.156145 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:53.156106 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w\" (UID: \"9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w" Nov 03 22:54:53.158655 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:53.158632 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w\" (UID: \"9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w" Nov 03 22:54:53.322529 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:53.322447 5196 patch_prober.go:29] interesting pod/e2e-api-a-26444f-openshift-default-695897496f-mcdxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:54:53.322529 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:53.322508 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" podUID="0474bb77-105a-4009-9f7e-2e19a4068fc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 22:54:53.373078 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:53.373045 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w" Nov 03 22:54:53.783792 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:53.783756 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w"] Nov 03 22:54:53.784802 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:54:53.784772 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e0c1f9c_4b12_42d2_bc9f_8caaffacb2b7.slice/crio-a4ae9313252465c0cb71b20b0ddbd2e5b09973e11106b6d7266835500decbf15 WatchSource:0}: Error finding container a4ae9313252465c0cb71b20b0ddbd2e5b09973e11106b6d7266835500decbf15: Status 404 returned error can't find the container with id a4ae9313252465c0cb71b20b0ddbd2e5b09973e11106b6d7266835500decbf15 Nov 03 22:54:53.786477 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:53.786457 5196 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:54:53.977910 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:53.977842 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w" event={"ID":"9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7","Type":"ContainerStarted","Data":"a4ae9313252465c0cb71b20b0ddbd2e5b09973e11106b6d7266835500decbf15"} Nov 03 22:54:54.322724 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:54.322687 5196 patch_prober.go:29] interesting pod/e2e-api-a-26444f-openshift-default-695897496f-mcdxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:54:54.323195 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:54.322741 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" podUID="0474bb77-105a-4009-9f7e-2e19a4068fc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 22:54:54.985148 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:54.985112 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w" event={"ID":"9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7","Type":"ContainerStarted","Data":"cb7e033ae66f070148ca45f9182b60397ad0ec9e33d90a8cc406cae6df0e87a9"} Nov 03 22:54:54.985307 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:54.985213 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w" Nov 03 22:54:55.117284 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:55.117224 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w" podStartSLOduration=2.164569193 podStartE2EDuration="3.117204346s" totalImagesPullingTime="952.635153ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:54:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:54:55.11111387 +0000 UTC m=+1924.420673164" watchObservedRunningTime="2026-11-03 22:54:55.117204346 +0000 UTC m=+1924.426763641" Nov 03 22:54:55.265393 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:55.265357 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:55.322685 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:55.322629 5196 patch_prober.go:29] interesting pod/e2e-api-a-26444f-openshift-default-695897496f-mcdxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:54:55.322884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:55.322710 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" podUID="0474bb77-105a-4009-9f7e-2e19a4068fc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 22:54:55.416681 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:55.416605 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:55.571072 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:55.571040 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:54:55.943807 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:54:55.943771 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:54:56.322446 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:56.322364 5196 patch_prober.go:29] interesting pod/e2e-api-a-26444f-openshift-default-695897496f-mcdxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:54:56.322446 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:56.322420 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" podUID="0474bb77-105a-4009-9f7e-2e19a4068fc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 22:54:57.327591 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:57.327559 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:54:57.328080 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:57.328065 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:54:57.328813 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:57.328791 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:54:57.719146 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:54:57.719114 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:55:01.016643 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:01.016612 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w" Nov 03 22:55:01.618754 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:01.618723 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:55:02.312112 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:02.312080 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:55:02.475410 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:02.475381 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:55:02.621390 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:02.621356 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:55:07.310881 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:07.308736 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:55:07.457412 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:07.457379 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:55:07.816443 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:07.816405 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:55:12.238657 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:12.238624 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:55:12.792774 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:12.792735 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:55:15.151252 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:15.151221 5196 generic.go:356] "Generic (PLEG): container finished" podID="9fde3ea1-b20e-4828-b8e6-5ffd537294f6" containerID="a48ffae424ef174332893917bac4904aad4fbab46d0da2576e210e54c8c3352f" exitCode=0 Nov 03 22:55:15.151771 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:15.151292 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" event={"ID":"9fde3ea1-b20e-4828-b8e6-5ffd537294f6","Type":"ContainerDied","Data":"a48ffae424ef174332893917bac4904aad4fbab46d0da2576e210e54c8c3352f"} Nov 03 22:55:15.151771 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:15.151681 5196 scope.go:122] "RemoveContainer" containerID="a48ffae424ef174332893917bac4904aad4fbab46d0da2576e210e54c8c3352f" Nov 03 22:55:15.153396 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:15.153367 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:55:15.153474 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:15.153435 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:55:15.153474 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:15.153465 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:55:16.161057 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:16.161024 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" event={"ID":"9fde3ea1-b20e-4828-b8e6-5ffd537294f6","Type":"ContainerStarted","Data":"7036bd1effaa4e0ac338ddb97f2e51a6bd0e6d9aacb1a4b14b59bcae9d216231"} Nov 03 22:55:16.865388 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:16.865361 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:55:16.866347 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:16.866322 5196 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 03 22:55:16.866465 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:16.866368 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" podUID="9fde3ea1-b20e-4828-b8e6-5ffd537294f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 03 22:55:17.057482 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:17.057434 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:55:17.078053 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:17.078012 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:55:17.865637 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:17.865604 5196 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 03 22:55:17.866160 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:17.865666 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" podUID="9fde3ea1-b20e-4828-b8e6-5ffd537294f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 03 22:55:18.865476 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:18.865440 5196 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 03 22:55:18.865652 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:18.865500 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" podUID="9fde3ea1-b20e-4828-b8e6-5ffd537294f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 03 22:55:19.865831 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:19.865796 5196 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 03 22:55:19.866281 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:19.865876 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" podUID="9fde3ea1-b20e-4828-b8e6-5ffd537294f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 03 22:55:20.865340 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:20.865303 5196 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 03 22:55:20.865508 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:20.865358 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" podUID="9fde3ea1-b20e-4828-b8e6-5ffd537294f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 03 22:55:21.870130 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:21.870093 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:55:21.871196 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:21.871180 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:55:22.260397 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:22.260309 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:55:25.918627 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:55:25.918588 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:55:27.446136 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:27.446107 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:55:28.509577 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.509537 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx"] Nov 03 22:55:28.512735 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.512717 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" Nov 03 22:55:28.533619 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.530980 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kpmpx\" (UID: \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" Nov 03 22:55:28.533619 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.531095 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kpmpx\" (UID: \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" Nov 03 22:55:28.533619 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.531136 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kpmpx\" (UID: \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" Nov 03 22:55:28.533619 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.531186 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kpmpx\" (UID: \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" Nov 03 22:55:28.533619 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.531255 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kpmpx\" (UID: \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" Nov 03 22:55:28.533619 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.531287 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kpmpx\" (UID: \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" Nov 03 22:55:28.533619 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.531339 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kpmpx\" (UID: \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" Nov 03 22:55:28.533619 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.531405 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpt2g\" (UniqueName: \"kubernetes.io/projected/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-kube-api-access-dpt2g\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kpmpx\" (UID: \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" Nov 03 22:55:28.533619 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.531436 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kpmpx\" (UID: \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" Nov 03 22:55:28.559716 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.559690 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz"] Nov 03 22:55:28.562540 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.562524 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj"] Nov 03 22:55:28.562658 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.562638 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" Nov 03 22:55:28.565282 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.565264 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" Nov 03 22:55:28.607483 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.607463 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-89xv7\"" Nov 03 22:55:28.616976 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.616956 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx"] Nov 03 22:55:28.632571 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.632547 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f644685-a2d7-48ee-833d-d49c9959f598-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-njmdj\" (UID: \"9f644685-a2d7-48ee-833d-d49c9959f598\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" Nov 03 22:55:28.632708 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.632575 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f644685-a2d7-48ee-833d-d49c9959f598-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-njmdj\" (UID: \"9f644685-a2d7-48ee-833d-d49c9959f598\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" Nov 03 22:55:28.632708 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.632604 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kpmpx\" (UID: \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" Nov 03 22:55:28.632708 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.632627 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f644685-a2d7-48ee-833d-d49c9959f598-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-njmdj\" (UID: \"9f644685-a2d7-48ee-833d-d49c9959f598\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" Nov 03 22:55:28.633283 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.632883 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ef06915-cf1c-45cb-b61b-8c80c45f1dcd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8q2rz\" (UID: \"6ef06915-cf1c-45cb-b61b-8c80c45f1dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" Nov 03 22:55:28.633520 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.632978 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kpmpx\" (UID: \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" Nov 03 22:55:28.633607 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.633549 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f644685-a2d7-48ee-833d-d49c9959f598-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-njmdj\" (UID: \"9f644685-a2d7-48ee-833d-d49c9959f598\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" Nov 03 22:55:28.633607 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.633594 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kpmpx\" (UID: \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" Nov 03 22:55:28.633720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.633612 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kpmpx\" (UID: \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" Nov 03 22:55:28.633720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.633656 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kpmpx\" (UID: \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" Nov 03 22:55:28.633720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.633705 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gxgn\" (UniqueName: \"kubernetes.io/projected/9f644685-a2d7-48ee-833d-d49c9959f598-kube-api-access-7gxgn\") pod \"maas-default-gateway-openshift-default-7f5667f86c-njmdj\" (UID: \"9f644685-a2d7-48ee-833d-d49c9959f598\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" Nov 03 22:55:28.633908 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.633737 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f644685-a2d7-48ee-833d-d49c9959f598-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-njmdj\" (UID: \"9f644685-a2d7-48ee-833d-d49c9959f598\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" Nov 03 22:55:28.633908 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.633779 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kpmpx\" (UID: \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" Nov 03 22:55:28.633908 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.633805 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ef06915-cf1c-45cb-b61b-8c80c45f1dcd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8q2rz\" (UID: \"6ef06915-cf1c-45cb-b61b-8c80c45f1dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" Nov 03 22:55:28.633908 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.633836 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kpmpx\" (UID: \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" Nov 03 22:55:28.634189 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.633914 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kpmpx\" (UID: \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" Nov 03 22:55:28.634189 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.633951 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef06915-cf1c-45cb-b61b-8c80c45f1dcd-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8q2rz\" (UID: \"6ef06915-cf1c-45cb-b61b-8c80c45f1dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" Nov 03 22:55:28.634189 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.633975 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef06915-cf1c-45cb-b61b-8c80c45f1dcd-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8q2rz\" (UID: \"6ef06915-cf1c-45cb-b61b-8c80c45f1dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" Nov 03 22:55:28.634189 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.634001 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nj4j\" (UniqueName: \"kubernetes.io/projected/6ef06915-cf1c-45cb-b61b-8c80c45f1dcd-kube-api-access-9nj4j\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8q2rz\" (UID: \"6ef06915-cf1c-45cb-b61b-8c80c45f1dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" Nov 03 22:55:28.634189 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.634033 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kpmpx\" (UID: \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" Nov 03 22:55:28.634189 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.634037 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f644685-a2d7-48ee-833d-d49c9959f598-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-njmdj\" (UID: \"9f644685-a2d7-48ee-833d-d49c9959f598\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" Nov 03 22:55:28.634189 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.634103 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ef06915-cf1c-45cb-b61b-8c80c45f1dcd-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8q2rz\" (UID: \"6ef06915-cf1c-45cb-b61b-8c80c45f1dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" Nov 03 22:55:28.634189 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.634134 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpt2g\" (UniqueName: \"kubernetes.io/projected/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-kube-api-access-dpt2g\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kpmpx\" (UID: \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" Nov 03 22:55:28.634189 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.634181 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kpmpx\" (UID: \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" Nov 03 22:55:28.634683 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.634192 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kpmpx\" (UID: \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" Nov 03 22:55:28.634683 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.634268 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ef06915-cf1c-45cb-b61b-8c80c45f1dcd-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8q2rz\" (UID: \"6ef06915-cf1c-45cb-b61b-8c80c45f1dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" Nov 03 22:55:28.634683 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.634305 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f644685-a2d7-48ee-833d-d49c9959f598-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-njmdj\" (UID: \"9f644685-a2d7-48ee-833d-d49c9959f598\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" Nov 03 22:55:28.634683 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.634333 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f644685-a2d7-48ee-833d-d49c9959f598-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-njmdj\" (UID: \"9f644685-a2d7-48ee-833d-d49c9959f598\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" Nov 03 22:55:28.634683 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.634384 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ef06915-cf1c-45cb-b61b-8c80c45f1dcd-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8q2rz\" (UID: \"6ef06915-cf1c-45cb-b61b-8c80c45f1dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" Nov 03 22:55:28.634683 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.634415 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ef06915-cf1c-45cb-b61b-8c80c45f1dcd-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8q2rz\" (UID: \"6ef06915-cf1c-45cb-b61b-8c80c45f1dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" Nov 03 22:55:28.635035 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.634779 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kpmpx\" (UID: \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" Nov 03 22:55:28.635035 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.635026 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kpmpx\" (UID: \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" Nov 03 22:55:28.636547 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.636525 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kpmpx\" (UID: \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" Nov 03 22:55:28.637196 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.637177 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kpmpx\" (UID: \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" Nov 03 22:55:28.681507 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.681472 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj"] Nov 03 22:55:28.685360 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.685331 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz"] Nov 03 22:55:28.697578 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.697548 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpt2g\" (UniqueName: \"kubernetes.io/projected/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-kube-api-access-dpt2g\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kpmpx\" (UID: \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" Nov 03 22:55:28.697835 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.697811 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kpmpx\" (UID: \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" Nov 03 22:55:28.735766 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.735738 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f644685-a2d7-48ee-833d-d49c9959f598-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-njmdj\" (UID: \"9f644685-a2d7-48ee-833d-d49c9959f598\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" Nov 03 22:55:28.735766 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.735771 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f644685-a2d7-48ee-833d-d49c9959f598-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-njmdj\" (UID: \"9f644685-a2d7-48ee-833d-d49c9959f598\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" Nov 03 22:55:28.735931 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.735813 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f644685-a2d7-48ee-833d-d49c9959f598-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-njmdj\" (UID: \"9f644685-a2d7-48ee-833d-d49c9959f598\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" Nov 03 22:55:28.735931 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.735851 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ef06915-cf1c-45cb-b61b-8c80c45f1dcd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8q2rz\" (UID: \"6ef06915-cf1c-45cb-b61b-8c80c45f1dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" Nov 03 22:55:28.735931 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.735916 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f644685-a2d7-48ee-833d-d49c9959f598-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-njmdj\" (UID: \"9f644685-a2d7-48ee-833d-d49c9959f598\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" Nov 03 22:55:28.736130 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.735985 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gxgn\" (UniqueName: \"kubernetes.io/projected/9f644685-a2d7-48ee-833d-d49c9959f598-kube-api-access-7gxgn\") pod \"maas-default-gateway-openshift-default-7f5667f86c-njmdj\" (UID: \"9f644685-a2d7-48ee-833d-d49c9959f598\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" Nov 03 22:55:28.736130 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.736016 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f644685-a2d7-48ee-833d-d49c9959f598-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-njmdj\" (UID: \"9f644685-a2d7-48ee-833d-d49c9959f598\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" Nov 03 22:55:28.736130 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.736060 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ef06915-cf1c-45cb-b61b-8c80c45f1dcd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8q2rz\" (UID: \"6ef06915-cf1c-45cb-b61b-8c80c45f1dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" Nov 03 22:55:28.736130 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.736117 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef06915-cf1c-45cb-b61b-8c80c45f1dcd-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8q2rz\" (UID: \"6ef06915-cf1c-45cb-b61b-8c80c45f1dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" Nov 03 22:55:28.736387 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.736140 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef06915-cf1c-45cb-b61b-8c80c45f1dcd-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8q2rz\" (UID: \"6ef06915-cf1c-45cb-b61b-8c80c45f1dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" Nov 03 22:55:28.736387 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.736166 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nj4j\" (UniqueName: \"kubernetes.io/projected/6ef06915-cf1c-45cb-b61b-8c80c45f1dcd-kube-api-access-9nj4j\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8q2rz\" (UID: \"6ef06915-cf1c-45cb-b61b-8c80c45f1dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" Nov 03 22:55:28.736387 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.736200 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f644685-a2d7-48ee-833d-d49c9959f598-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-njmdj\" (UID: \"9f644685-a2d7-48ee-833d-d49c9959f598\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" Nov 03 22:55:28.736387 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.736240 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ef06915-cf1c-45cb-b61b-8c80c45f1dcd-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8q2rz\" (UID: \"6ef06915-cf1c-45cb-b61b-8c80c45f1dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" Nov 03 22:55:28.736387 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.736262 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f644685-a2d7-48ee-833d-d49c9959f598-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-njmdj\" (UID: \"9f644685-a2d7-48ee-833d-d49c9959f598\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" Nov 03 22:55:28.736387 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.736292 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ef06915-cf1c-45cb-b61b-8c80c45f1dcd-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8q2rz\" (UID: \"6ef06915-cf1c-45cb-b61b-8c80c45f1dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" Nov 03 22:55:28.736387 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.736309 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f644685-a2d7-48ee-833d-d49c9959f598-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-njmdj\" (UID: \"9f644685-a2d7-48ee-833d-d49c9959f598\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" Nov 03 22:55:28.736387 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.736323 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f644685-a2d7-48ee-833d-d49c9959f598-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-njmdj\" (UID: \"9f644685-a2d7-48ee-833d-d49c9959f598\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" Nov 03 22:55:28.736387 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.736349 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f644685-a2d7-48ee-833d-d49c9959f598-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-njmdj\" (UID: \"9f644685-a2d7-48ee-833d-d49c9959f598\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" Nov 03 22:55:28.736387 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.736385 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ef06915-cf1c-45cb-b61b-8c80c45f1dcd-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8q2rz\" (UID: \"6ef06915-cf1c-45cb-b61b-8c80c45f1dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" Nov 03 22:55:28.736992 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.736407 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ef06915-cf1c-45cb-b61b-8c80c45f1dcd-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8q2rz\" (UID: \"6ef06915-cf1c-45cb-b61b-8c80c45f1dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" Nov 03 22:55:28.736992 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.736543 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f644685-a2d7-48ee-833d-d49c9959f598-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-njmdj\" (UID: \"9f644685-a2d7-48ee-833d-d49c9959f598\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" Nov 03 22:55:28.736992 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.736603 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef06915-cf1c-45cb-b61b-8c80c45f1dcd-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8q2rz\" (UID: \"6ef06915-cf1c-45cb-b61b-8c80c45f1dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" Nov 03 22:55:28.736992 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.736618 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ef06915-cf1c-45cb-b61b-8c80c45f1dcd-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8q2rz\" (UID: \"6ef06915-cf1c-45cb-b61b-8c80c45f1dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" Nov 03 22:55:28.736992 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.736688 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f644685-a2d7-48ee-833d-d49c9959f598-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-njmdj\" (UID: \"9f644685-a2d7-48ee-833d-d49c9959f598\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" Nov 03 22:55:28.736992 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.736910 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef06915-cf1c-45cb-b61b-8c80c45f1dcd-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8q2rz\" (UID: \"6ef06915-cf1c-45cb-b61b-8c80c45f1dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" Nov 03 22:55:28.736992 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.736960 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ef06915-cf1c-45cb-b61b-8c80c45f1dcd-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8q2rz\" (UID: \"6ef06915-cf1c-45cb-b61b-8c80c45f1dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" Nov 03 22:55:28.737338 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.737005 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ef06915-cf1c-45cb-b61b-8c80c45f1dcd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8q2rz\" (UID: \"6ef06915-cf1c-45cb-b61b-8c80c45f1dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" Nov 03 22:55:28.737415 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.737385 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f644685-a2d7-48ee-833d-d49c9959f598-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-njmdj\" (UID: \"9f644685-a2d7-48ee-833d-d49c9959f598\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" Nov 03 22:55:28.738783 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.738764 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ef06915-cf1c-45cb-b61b-8c80c45f1dcd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8q2rz\" (UID: \"6ef06915-cf1c-45cb-b61b-8c80c45f1dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" Nov 03 22:55:28.738932 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.738915 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f644685-a2d7-48ee-833d-d49c9959f598-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-njmdj\" (UID: \"9f644685-a2d7-48ee-833d-d49c9959f598\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" Nov 03 22:55:28.739467 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.739451 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ef06915-cf1c-45cb-b61b-8c80c45f1dcd-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8q2rz\" (UID: \"6ef06915-cf1c-45cb-b61b-8c80c45f1dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" Nov 03 22:55:28.739629 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.739613 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f644685-a2d7-48ee-833d-d49c9959f598-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-njmdj\" (UID: \"9f644685-a2d7-48ee-833d-d49c9959f598\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" Nov 03 22:55:28.798505 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.798434 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ef06915-cf1c-45cb-b61b-8c80c45f1dcd-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8q2rz\" (UID: \"6ef06915-cf1c-45cb-b61b-8c80c45f1dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" Nov 03 22:55:28.798505 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.798435 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gxgn\" (UniqueName: \"kubernetes.io/projected/9f644685-a2d7-48ee-833d-d49c9959f598-kube-api-access-7gxgn\") pod \"maas-default-gateway-openshift-default-7f5667f86c-njmdj\" (UID: \"9f644685-a2d7-48ee-833d-d49c9959f598\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" Nov 03 22:55:28.798718 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.798618 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f644685-a2d7-48ee-833d-d49c9959f598-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-njmdj\" (UID: \"9f644685-a2d7-48ee-833d-d49c9959f598\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" Nov 03 22:55:28.798943 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.798923 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nj4j\" (UniqueName: \"kubernetes.io/projected/6ef06915-cf1c-45cb-b61b-8c80c45f1dcd-kube-api-access-9nj4j\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8q2rz\" (UID: \"6ef06915-cf1c-45cb-b61b-8c80c45f1dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" Nov 03 22:55:28.825589 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.825564 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" Nov 03 22:55:28.876950 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.876785 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" Nov 03 22:55:28.882518 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:28.882472 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" Nov 03 22:55:29.260512 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:29.260486 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx"] Nov 03 22:55:29.262099 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:55:29.262073 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f2ded9_7d5f_4ea5_a2c2_fb9c6c1f99cf.slice/crio-d80e477abc32c79038988bf21470c8334287e28221ed11c7c55770e14dfc55d2 WatchSource:0}: Error finding container d80e477abc32c79038988bf21470c8334287e28221ed11c7c55770e14dfc55d2: Status 404 returned error can't find the container with id d80e477abc32c79038988bf21470c8334287e28221ed11c7c55770e14dfc55d2 Nov 03 22:55:29.264592 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:29.264558 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:55:29.264659 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:29.264627 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:55:29.264696 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:29.264656 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:55:29.312375 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:55:29.312352 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f644685_a2d7_48ee_833d_d49c9959f598.slice/crio-e469706c68de92805b29a42b5fd81c50e7fd51a73da64e2505f30c541e521d16 WatchSource:0}: Error finding container e469706c68de92805b29a42b5fd81c50e7fd51a73da64e2505f30c541e521d16: Status 404 returned error can't find the container with id e469706c68de92805b29a42b5fd81c50e7fd51a73da64e2505f30c541e521d16 Nov 03 22:55:29.314008 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:29.313987 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj"] Nov 03 22:55:29.314930 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:29.314900 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:55:29.314992 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:29.314960 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:55:29.315027 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:29.314994 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:55:29.363427 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:55:29.363395 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef06915_cf1c_45cb_b61b_8c80c45f1dcd.slice/crio-0e1206214b6f93ff3cabc68e5cd3c7068a4ce86bb1f1f17bac00148c98b932fc WatchSource:0}: Error finding container 0e1206214b6f93ff3cabc68e5cd3c7068a4ce86bb1f1f17bac00148c98b932fc: Status 404 returned error can't find the container with id 0e1206214b6f93ff3cabc68e5cd3c7068a4ce86bb1f1f17bac00148c98b932fc Nov 03 22:55:29.364835 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:29.364813 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz"] Nov 03 22:55:29.366341 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:29.366302 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:55:29.366458 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:29.366395 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:55:29.366458 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:29.366443 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:55:30.268278 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:30.268237 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" event={"ID":"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf","Type":"ContainerStarted","Data":"e9af8ca6fbec64658df6f30faf22ec5b3b7945328e10b5c2cb0d9c1d20c97a66"} Nov 03 22:55:30.268891 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:30.268306 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" event={"ID":"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf","Type":"ContainerStarted","Data":"d80e477abc32c79038988bf21470c8334287e28221ed11c7c55770e14dfc55d2"} Nov 03 22:55:30.270065 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:30.270035 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" event={"ID":"9f644685-a2d7-48ee-833d-d49c9959f598","Type":"ContainerStarted","Data":"69999c03b37eb7c66c108ca16b427823b3a62dc965b10c01df9aa99ba081b177"} Nov 03 22:55:30.270212 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:30.270079 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" event={"ID":"9f644685-a2d7-48ee-833d-d49c9959f598","Type":"ContainerStarted","Data":"e469706c68de92805b29a42b5fd81c50e7fd51a73da64e2505f30c541e521d16"} Nov 03 22:55:30.271752 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:30.271728 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" event={"ID":"6ef06915-cf1c-45cb-b61b-8c80c45f1dcd","Type":"ContainerStarted","Data":"02197941751f6c99522c146cc2a646f197a990d7e5375ba4f00f7384861cbe26"} Nov 03 22:55:30.271884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:30.271773 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" event={"ID":"6ef06915-cf1c-45cb-b61b-8c80c45f1dcd","Type":"ContainerStarted","Data":"0e1206214b6f93ff3cabc68e5cd3c7068a4ce86bb1f1f17bac00148c98b932fc"} Nov 03 22:55:30.510883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:30.501548 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" podStartSLOduration=2.501528306 podStartE2EDuration="2.501528306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:55:30.444304319 +0000 UTC m=+1959.753863616" watchObservedRunningTime="2026-11-03 22:55:30.501528306 +0000 UTC m=+1959.811087600" Nov 03 22:55:30.614765 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:30.614663 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" podStartSLOduration=2.614638524 podStartE2EDuration="2.614638524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:55:30.579905042 +0000 UTC m=+1959.889464341" watchObservedRunningTime="2026-11-03 22:55:30.614638524 +0000 UTC m=+1959.924197817" Nov 03 22:55:30.752064 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:30.747493 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" podStartSLOduration=2.747469298 podStartE2EDuration="2.747469298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:55:30.724290448 +0000 UTC m=+1960.033849742" watchObservedRunningTime="2026-11-03 22:55:30.747469298 +0000 UTC m=+1960.057028593" Nov 03 22:55:30.885653 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:30.880844 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8q2rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 22:55:30.885653 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:30.881965 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" podUID="6ef06915-cf1c-45cb-b61b-8c80c45f1dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 22:55:30.885653 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:30.882443 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-kpmpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 03 22:55:30.885653 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:30.882496 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" podUID="e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 03 22:55:30.885653 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:30.884609 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-njmdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 03 22:55:30.885653 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:30.884674 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" podUID="9f644685-a2d7-48ee-833d-d49c9959f598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 03 22:55:31.003886 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:31.003481 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" Nov 03 22:55:31.003886 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:31.003554 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" Nov 03 22:55:31.003886 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:31.003584 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" Nov 03 22:55:31.175588 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:31.175466 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:55:31.386997 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:31.341627 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:55:31.837828 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:31.837795 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-kpmpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 03 22:55:31.838098 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:31.838069 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" podUID="e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 03 22:55:31.878757 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:31.878715 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8q2rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 22:55:31.879003 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:31.878983 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" podUID="6ef06915-cf1c-45cb-b61b-8c80c45f1dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 22:55:31.883461 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:31.883437 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-njmdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 03 22:55:31.883547 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:31.883481 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" podUID="9f644685-a2d7-48ee-833d-d49c9959f598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 03 22:55:32.848889 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:32.848308 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-kpmpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 03 22:55:32.848889 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:32.848378 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" podUID="e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 03 22:55:32.888028 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:32.887992 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8q2rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 22:55:32.888647 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:32.888619 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" podUID="6ef06915-cf1c-45cb-b61b-8c80c45f1dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 22:55:32.888845 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:32.888545 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-njmdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 03 22:55:32.889023 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:32.889004 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" podUID="9f644685-a2d7-48ee-833d-d49c9959f598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 03 22:55:33.738156 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:33.738120 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:55:33.839732 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:33.839701 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-kpmpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 03 22:55:33.841410 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:33.840108 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" podUID="e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 03 22:55:33.881439 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:33.881403 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8q2rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 22:55:33.882126 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:33.882075 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" podUID="6ef06915-cf1c-45cb-b61b-8c80c45f1dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 22:55:33.886448 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:33.886415 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-njmdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 03 22:55:33.886677 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:33.886648 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" podUID="9f644685-a2d7-48ee-833d-d49c9959f598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 03 22:55:34.556690 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:34.555997 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" podUID="9d8b4fed-b4b0-49ce-9238-e23175e90b70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:55:34.793893 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:34.793157 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" podUID="82749c55-0fca-4a14-b6ba-7df122a6a407" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:55:34.838511 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:34.838481 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-kpmpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 03 22:55:34.838793 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:34.838761 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" podUID="e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 03 22:55:34.879139 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:34.879098 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8q2rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 22:55:34.879446 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:34.879417 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" podUID="6ef06915-cf1c-45cb-b61b-8c80c45f1dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 22:55:34.898291 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:34.898253 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-njmdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 03 22:55:34.898906 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:34.898871 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" podUID="9f644685-a2d7-48ee-833d-d49c9959f598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 03 22:55:34.981367 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:34.980968 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" podUID="26fa51c2-0f2e-482f-9c67-0ea7924d7b3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:55:35.069005 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:35.068582 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" podUID="34e9cf3a-3456-4835-8933-102ce735cccc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:55:35.161068 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:35.160970 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" podUID="9f2e3f52-0664-434f-919b-32736cbf0602" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:55:35.528094 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:35.527985 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" podUID="abb461c4-9707-4b5c-aba0-793f7bb993e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:55:35.557015 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:35.556959 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" podUID="9d8b4fed-b4b0-49ce-9238-e23175e90b70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:55:35.557748 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:35.557722 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" Nov 03 22:55:35.835950 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:35.831647 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-kpmpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 03 22:55:35.835950 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:35.831773 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" podUID="e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 03 22:55:35.835950 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:35.832740 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" podUID="82749c55-0fca-4a14-b6ba-7df122a6a407" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:55:35.835950 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:35.833275 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" Nov 03 22:55:35.948908 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:35.899956 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-njmdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 03 22:55:36.257245 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:36.257205 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" podUID="9f644685-a2d7-48ee-833d-d49c9959f598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 03 22:55:36.275171 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:35.957768 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:55:36.275518 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:36.010576 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" podUID="26fa51c2-0f2e-482f-9c67-0ea7924d7b3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 03 22:55:36.275798 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:36.160010 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" podUID="34e9cf3a-3456-4835-8933-102ce735cccc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 03 22:55:36.279903 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:36.278323 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" Nov 03 22:55:36.279903 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:36.278418 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" Nov 03 22:55:36.279903 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:35.928679 5196 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" podUID="9d8b4fed-b4b0-49ce-9238-e23175e90b70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:55:36.280177 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:35.915251 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8q2rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 22:55:36.283909 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:36.280322 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" podUID="6ef06915-cf1c-45cb-b61b-8c80c45f1dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 22:55:36.284099 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:36.172994 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" podUID="9f2e3f52-0664-434f-919b-32736cbf0602" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:55:36.293662 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:36.293640 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" Nov 03 22:55:36.434015 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:36.433955 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:55:36.601089 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:36.536297 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" podUID="abb461c4-9707-4b5c-aba0-793f7bb993e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 03 22:55:36.604523 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:36.604499 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" Nov 03 22:55:36.678880 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:36.575318 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" podUID="9d8b4fed-b4b0-49ce-9238-e23175e90b70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:55:36.835096 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:36.833286 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" podUID="82749c55-0fca-4a14-b6ba-7df122a6a407" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:55:36.835716 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:36.834773 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-kpmpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 03 22:55:36.835928 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:36.835895 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" podUID="e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 03 22:55:36.958295 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:36.958181 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8q2rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 22:55:36.958898 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:36.958825 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" podUID="6ef06915-cf1c-45cb-b61b-8c80c45f1dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 22:55:36.959164 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:36.959145 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-njmdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 03 22:55:36.959330 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:36.959306 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" podUID="9f644685-a2d7-48ee-833d-d49c9959f598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 03 22:55:37.207885 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:37.187163 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" Nov 03 22:55:37.281376 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:37.281282 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" podUID="26fa51c2-0f2e-482f-9c67-0ea7924d7b3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:55:37.282930 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:37.282898 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" podUID="34e9cf3a-3456-4835-8933-102ce735cccc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:55:37.321225 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:37.304156 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" podUID="9f2e3f52-0664-434f-919b-32736cbf0602" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 22:55:37.344316 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:37.331007 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" Nov 03 22:55:37.711799 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:37.711697 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" podUID="9d8b4fed-b4b0-49ce-9238-e23175e90b70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:55:37.833885 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:37.831998 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-kpmpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 03 22:55:37.833885 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:37.832062 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" podUID="e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 03 22:55:37.841886 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:37.836074 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" podUID="82749c55-0fca-4a14-b6ba-7df122a6a407" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 03 22:55:37.905188 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:37.901924 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8q2rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 22:55:37.905188 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:37.901985 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" podUID="6ef06915-cf1c-45cb-b61b-8c80c45f1dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 22:55:37.905188 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:37.902393 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-njmdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 03 22:55:37.905188 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:37.902429 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" podUID="9f644685-a2d7-48ee-833d-d49c9959f598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 03 22:55:38.081005 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:38.079905 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" Nov 03 22:55:38.083561 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:38.082159 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" Nov 03 22:55:38.095040 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:38.095008 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" podUID="abb461c4-9707-4b5c-aba0-793f7bb993e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:55:38.190006 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:38.189975 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" Nov 03 22:55:38.268948 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:38.268660 5196 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" podUID="82749c55-0fca-4a14-b6ba-7df122a6a407" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:55:38.826585 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:38.826546 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-kpmpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 03 22:55:38.826956 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:38.826925 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" podUID="e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 03 22:55:38.836704 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:38.836673 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" podUID="82749c55-0fca-4a14-b6ba-7df122a6a407" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:55:38.906884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:38.895330 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8q2rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 22:55:38.906884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:38.895388 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" podUID="6ef06915-cf1c-45cb-b61b-8c80c45f1dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 22:55:38.953017 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:38.952975 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-njmdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 03 22:55:38.953225 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:38.953065 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" podUID="9f644685-a2d7-48ee-833d-d49c9959f598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 03 22:55:39.698882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:39.685539 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" Nov 03 22:55:39.852506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:39.852472 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-kpmpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 03 22:55:39.852823 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:39.852759 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" podUID="e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 03 22:55:39.881385 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:39.881253 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8q2rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 22:55:39.881385 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:39.881305 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" podUID="6ef06915-cf1c-45cb-b61b-8c80c45f1dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 22:55:39.895328 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:39.895217 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-njmdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Nov 03 22:55:39.895328 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:39.895269 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" podUID="9f644685-a2d7-48ee-833d-d49c9959f598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Nov 03 22:55:40.136885 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:40.120691 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:55:40.353061 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:40.352995 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" podUID="26fa51c2-0f2e-482f-9c67-0ea7924d7b3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:55:40.904883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:40.904469 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8q2rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Nov 03 22:55:40.904883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:40.904529 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" podUID="6ef06915-cf1c-45cb-b61b-8c80c45f1dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Nov 03 22:55:41.873079 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:41.873036 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-kpmpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:55:41.873417 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:41.873385 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" podUID="e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:55:42.045883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:42.045358 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-njmdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:55:42.045883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:42.045435 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" podUID="9f644685-a2d7-48ee-833d-d49c9959f598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:55:42.345541 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:42.343691 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" Nov 03 22:55:42.345541 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:42.345032 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" Nov 03 22:55:42.348403 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:42.346568 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" Nov 03 22:55:42.477772 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:42.477741 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" Nov 03 22:55:42.478577 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:42.478436 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" Nov 03 22:55:42.479933 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:42.479908 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" Nov 03 22:55:42.833146 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:42.831276 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" Nov 03 22:55:42.833146 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:42.832995 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" Nov 03 22:55:42.834780 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:42.834714 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" Nov 03 22:55:45.763884 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:45.762316 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:55:46.414367 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:46.414335 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:55:46.562328 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:46.562294 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:55:48.687542 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:48.687509 5196 patch_prober.go:29] interesting pod/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": read tcp 10.133.0.2:51210->10.133.0.96:15021: read: connection reset by peer" start-of-body= Nov 03 22:55:48.688178 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:48.687569 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" podUID="9a97695c-47ae-48f9-ad2e-f97625e48234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": read tcp 10.133.0.2:51210->10.133.0.96:15021: read: connection reset by peer" Nov 03 22:55:49.351314 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:49.351279 5196 generic.go:356] "Generic (PLEG): container finished" podID="75a0d61a-c8e8-4925-941b-d729b4b21114" containerID="5cdece8ad478f12547a659c127254ce222eb9bd255aa13471ce7c1b26a104e84" exitCode=0 Nov 03 22:55:49.351502 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:49.351362 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" event={"ID":"75a0d61a-c8e8-4925-941b-d729b4b21114","Type":"ContainerDied","Data":"5cdece8ad478f12547a659c127254ce222eb9bd255aa13471ce7c1b26a104e84"} Nov 03 22:55:49.351780 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:49.351763 5196 scope.go:122] "RemoveContainer" containerID="5cdece8ad478f12547a659c127254ce222eb9bd255aa13471ce7c1b26a104e84" Nov 03 22:55:49.353265 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:49.353237 5196 generic.go:356] "Generic (PLEG): container finished" podID="9a97695c-47ae-48f9-ad2e-f97625e48234" containerID="6b274616ced7135811db4386afb9aa23676ae1e48aa028c422227d91e16644f8" exitCode=0 Nov 03 22:55:49.353361 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:49.353285 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" event={"ID":"9a97695c-47ae-48f9-ad2e-f97625e48234","Type":"ContainerDied","Data":"6b274616ced7135811db4386afb9aa23676ae1e48aa028c422227d91e16644f8"} Nov 03 22:55:49.353498 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:49.353466 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:55:49.353559 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:49.353546 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:55:49.353598 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:49.353582 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:55:49.353671 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:49.353654 5196 scope.go:122] "RemoveContainer" containerID="6b274616ced7135811db4386afb9aa23676ae1e48aa028c422227d91e16644f8" Nov 03 22:55:49.355120 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:49.355094 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:55:49.355195 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:49.355158 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:55:49.355269 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:49.355191 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:55:50.362616 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:50.362577 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" event={"ID":"75a0d61a-c8e8-4925-941b-d729b4b21114","Type":"ContainerStarted","Data":"fef06f8ebf273399f4651503c28956673497a315df6bec21d2be11b4e043f3ec"} Nov 03 22:55:50.364899 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:50.364872 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" event={"ID":"9a97695c-47ae-48f9-ad2e-f97625e48234","Type":"ContainerStarted","Data":"3ad56f90432d3b07d261befb78877326ffefe902a254d307a6334578f5f76bfa"} Nov 03 22:55:50.636823 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:50.636725 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:55:51.051644 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:51.051545 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:55:51.051644 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:51.051546 5196 patch_prober.go:29] interesting pod/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 22:55:51.051847 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:51.051665 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" podUID="9a97695c-47ae-48f9-ad2e-f97625e48234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 22:55:51.070454 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:51.070422 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:55:51.070631 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:51.070611 5196 patch_prober.go:29] interesting pod/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 22:55:51.070681 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:51.070660 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" podUID="75a0d61a-c8e8-4925-941b-d729b4b21114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 22:55:52.051393 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:52.051352 5196 patch_prober.go:29] interesting pod/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 22:55:52.051893 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:52.051408 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" podUID="9a97695c-47ae-48f9-ad2e-f97625e48234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 22:55:52.071019 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:52.070987 5196 patch_prober.go:29] interesting pod/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 22:55:52.071269 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:52.071243 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" podUID="75a0d61a-c8e8-4925-941b-d729b4b21114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 22:55:53.051876 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:53.051230 5196 patch_prober.go:29] interesting pod/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 22:55:53.051876 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:53.051291 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" podUID="9a97695c-47ae-48f9-ad2e-f97625e48234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 22:55:53.071203 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:53.071167 5196 patch_prober.go:29] interesting pod/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 22:55:53.071377 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:53.071235 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" podUID="75a0d61a-c8e8-4925-941b-d729b4b21114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 22:55:54.045895 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:54.045844 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk"] Nov 03 22:55:54.046312 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:54.046290 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" podUID="25d699f6-aaae-480e-9efb-45afb8297cce" containerName="istio-proxy" containerID="cri-o://b6ebc6d92b6c75854e3b164675a8823dfd49bc87618514b10a26d81a95e7a675" gracePeriod=30 Nov 03 22:55:54.051654 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:54.051628 5196 patch_prober.go:29] interesting pod/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 22:55:54.051740 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:54.051678 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" podUID="9a97695c-47ae-48f9-ad2e-f97625e48234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 22:55:54.071467 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:54.071444 5196 patch_prober.go:29] interesting pod/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 22:55:54.071887 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:54.071852 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" podUID="75a0d61a-c8e8-4925-941b-d729b4b21114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 22:55:55.052881 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:55.052578 5196 patch_prober.go:29] interesting pod/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 22:55:55.052881 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:55.052657 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" podUID="9a97695c-47ae-48f9-ad2e-f97625e48234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 22:55:55.071154 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:55.071068 5196 patch_prober.go:29] interesting pod/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 22:55:55.071469 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:55.071438 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" podUID="75a0d61a-c8e8-4925-941b-d729b4b21114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 22:55:55.528876 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:55.528816 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v"] Nov 03 22:55:55.529122 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:55.529089 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" podUID="f0ca4784-6fe2-4001-ac15-25675fc41a02" containerName="istio-proxy" containerID="cri-o://4b8f9e503d3700c74c892a7657e38946c9882fbfe1d07d5054b5c29be7bc4a55" gracePeriod=30 Nov 03 22:55:55.791629 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:55.791545 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:55:55.943980 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:55:55.943947 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:55:56.055877 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:56.055789 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:55:56.056254 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:56.056230 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:55:56.057165 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:56.057138 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:55:56.075669 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:56.075643 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:55:56.076033 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:56.075889 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:55:56.076606 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:56.076591 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:55:56.259784 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:56.259751 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n"] Nov 03 22:55:56.259988 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:56.259966 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" podUID="05bd48b9-ff9d-4511-9c48-9878b3ae40b3" containerName="istio-proxy" containerID="cri-o://e17b0e300b97020c9d7cc98e22a5f4587e929ee39d67dbc057cf160aabe36a17" gracePeriod=30 Nov 03 22:55:56.263138 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:56.263098 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c"] Nov 03 22:55:56.263535 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:56.263500 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" podUID="734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0" containerName="istio-proxy" containerID="cri-o://c4da88202d281925af5ce892dda476ca0edd932079194e3ce9a72b2b5aa40adf" gracePeriod=30 Nov 03 22:55:56.998805 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:56.998767 5196 patch_prober.go:29] interesting pod/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:55:56.999009 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:56.998830 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" podUID="f0ca4784-6fe2-4001-ac15-25675fc41a02" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:55:57.697899 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:57.697840 5196 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:55:57.698289 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:57.697913 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" podUID="05bd48b9-ff9d-4511-9c48-9878b3ae40b3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:55:58.422685 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:58.422646 5196 patch_prober.go:29] interesting pod/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:55:58.422916 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:58.422700 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" podUID="25d699f6-aaae-480e-9efb-45afb8297cce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:55:58.670254 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:58.670217 5196 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:55:58.670409 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:58.670284 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" podUID="734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:55:59.326272 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:59.326215 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" Nov 03 22:55:59.334707 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:59.334648 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d699f6-aaae-480e-9efb-45afb8297cce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d699f6-aaae-480e-9efb-45afb8297cce-workload-certs\") pod \"25d699f6-aaae-480e-9efb-45afb8297cce\" (UID: \"25d699f6-aaae-480e-9efb-45afb8297cce\") " Nov 03 22:55:59.334931 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:59.334911 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d699f6-aaae-480e-9efb-45afb8297cce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d699f6-aaae-480e-9efb-45afb8297cce-workload-socket\") pod \"25d699f6-aaae-480e-9efb-45afb8297cce\" (UID: \"25d699f6-aaae-480e-9efb-45afb8297cce\") " Nov 03 22:55:59.335010 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:59.334946 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d699f6-aaae-480e-9efb-45afb8297cce-workload-certs" pod "25d699f6-aaae-480e-9efb-45afb8297cce" (UID: "25d699f6-aaae-480e-9efb-45afb8297cce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:59.335010 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:59.334958 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d699f6-aaae-480e-9efb-45afb8297cce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d699f6-aaae-480e-9efb-45afb8297cce-credential-socket\") pod \"25d699f6-aaae-480e-9efb-45afb8297cce\" (UID: \"25d699f6-aaae-480e-9efb-45afb8297cce\") " Nov 03 22:55:59.335129 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:59.335011 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25d699f6-aaae-480e-9efb-45afb8297cce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d699f6-aaae-480e-9efb-45afb8297cce-istio-podinfo\") pod \"25d699f6-aaae-480e-9efb-45afb8297cce\" (UID: \"25d699f6-aaae-480e-9efb-45afb8297cce\") " Nov 03 22:55:59.335129 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:59.335050 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25d699f6-aaae-480e-9efb-45afb8297cce-kube-api-access-bcfxl\" (UniqueName: \"kubernetes.io/projected/25d699f6-aaae-480e-9efb-45afb8297cce-kube-api-access-bcfxl\") pod \"25d699f6-aaae-480e-9efb-45afb8297cce\" (UID: \"25d699f6-aaae-480e-9efb-45afb8297cce\") " Nov 03 22:55:59.335129 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:59.335092 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25d699f6-aaae-480e-9efb-45afb8297cce-istio-token\" (UniqueName: \"kubernetes.io/projected/25d699f6-aaae-480e-9efb-45afb8297cce-istio-token\") pod \"25d699f6-aaae-480e-9efb-45afb8297cce\" (UID: \"25d699f6-aaae-480e-9efb-45afb8297cce\") " Nov 03 22:55:59.335290 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:59.335232 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d699f6-aaae-480e-9efb-45afb8297cce-workload-socket" pod "25d699f6-aaae-480e-9efb-45afb8297cce" (UID: "25d699f6-aaae-480e-9efb-45afb8297cce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:59.335290 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:59.335244 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d699f6-aaae-480e-9efb-45afb8297cce-credential-socket" pod "25d699f6-aaae-480e-9efb-45afb8297cce" (UID: "25d699f6-aaae-480e-9efb-45afb8297cce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:59.335290 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:59.335244 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d699f6-aaae-480e-9efb-45afb8297cce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d699f6-aaae-480e-9efb-45afb8297cce-istio-envoy\") pod \"25d699f6-aaae-480e-9efb-45afb8297cce\" (UID: \"25d699f6-aaae-480e-9efb-45afb8297cce\") " Nov 03 22:55:59.335437 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:59.335311 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d699f6-aaae-480e-9efb-45afb8297cce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d699f6-aaae-480e-9efb-45afb8297cce-istio-data\") pod \"25d699f6-aaae-480e-9efb-45afb8297cce\" (UID: \"25d699f6-aaae-480e-9efb-45afb8297cce\") " Nov 03 22:55:59.335437 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:59.335354 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25d699f6-aaae-480e-9efb-45afb8297cce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d699f6-aaae-480e-9efb-45afb8297cce-istiod-ca-cert\") pod \"25d699f6-aaae-480e-9efb-45afb8297cce\" (UID: \"25d699f6-aaae-480e-9efb-45afb8297cce\") " Nov 03 22:55:59.335580 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:59.335560 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d699f6-aaae-480e-9efb-45afb8297cce-istio-data" pod "25d699f6-aaae-480e-9efb-45afb8297cce" (UID: "25d699f6-aaae-480e-9efb-45afb8297cce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:59.335745 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:59.335725 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25d699f6-aaae-480e-9efb-45afb8297cce-istiod-ca-cert" pod "25d699f6-aaae-480e-9efb-45afb8297cce" (UID: "25d699f6-aaae-480e-9efb-45afb8297cce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:55:59.336163 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:59.336070 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d699f6-aaae-480e-9efb-45afb8297cce-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:55:59.336163 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:59.336090 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d699f6-aaae-480e-9efb-45afb8297cce-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:55:59.336163 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:59.336104 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d699f6-aaae-480e-9efb-45afb8297cce-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:55:59.336163 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:59.336117 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d699f6-aaae-480e-9efb-45afb8297cce-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:55:59.336163 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:59.336132 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d699f6-aaae-480e-9efb-45afb8297cce-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:55:59.337603 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:59.337578 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d699f6-aaae-480e-9efb-45afb8297cce-istio-envoy" pod "25d699f6-aaae-480e-9efb-45afb8297cce" (UID: "25d699f6-aaae-480e-9efb-45afb8297cce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:59.337867 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:59.337828 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d699f6-aaae-480e-9efb-45afb8297cce-istio-token" pod "25d699f6-aaae-480e-9efb-45afb8297cce" (UID: "25d699f6-aaae-480e-9efb-45afb8297cce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:55:59.338049 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:59.338022 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25d699f6-aaae-480e-9efb-45afb8297cce-istio-podinfo" pod "25d699f6-aaae-480e-9efb-45afb8297cce" (UID: "25d699f6-aaae-480e-9efb-45afb8297cce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:55:59.338049 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:59.338033 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d699f6-aaae-480e-9efb-45afb8297cce-kube-api-access-bcfxl" pod "25d699f6-aaae-480e-9efb-45afb8297cce" (UID: "25d699f6-aaae-480e-9efb-45afb8297cce"). InnerVolumeSpecName "kube-api-access-bcfxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:55:59.437701 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:59.437673 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d699f6-aaae-480e-9efb-45afb8297cce-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:55:59.437701 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:59.437699 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d699f6-aaae-480e-9efb-45afb8297cce-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:55:59.437916 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:59.437709 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d699f6-aaae-480e-9efb-45afb8297cce-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:55:59.437916 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:59.437718 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcfxl\" (UniqueName: \"kubernetes.io/projected/25d699f6-aaae-480e-9efb-45afb8297cce-kube-api-access-bcfxl\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:55:59.489759 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:59.489724 5196 generic.go:356] "Generic (PLEG): container finished" podID="25d699f6-aaae-480e-9efb-45afb8297cce" containerID="b6ebc6d92b6c75854e3b164675a8823dfd49bc87618514b10a26d81a95e7a675" exitCode=0 Nov 03 22:55:59.489911 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:59.489827 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" Nov 03 22:55:59.489911 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:59.489827 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" event={"ID":"25d699f6-aaae-480e-9efb-45afb8297cce","Type":"ContainerDied","Data":"b6ebc6d92b6c75854e3b164675a8823dfd49bc87618514b10a26d81a95e7a675"} Nov 03 22:55:59.490001 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:59.489950 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk" event={"ID":"25d699f6-aaae-480e-9efb-45afb8297cce","Type":"ContainerDied","Data":"92ebaeba25a33c19b5ec8f5495e5e58889f8d57f4789925e98cd06d49f23043a"} Nov 03 22:55:59.490001 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:59.489985 5196 scope.go:122] "RemoveContainer" containerID="b6ebc6d92b6c75854e3b164675a8823dfd49bc87618514b10a26d81a95e7a675" Nov 03 22:55:59.509108 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:59.509083 5196 scope.go:122] "RemoveContainer" containerID="b6ebc6d92b6c75854e3b164675a8823dfd49bc87618514b10a26d81a95e7a675" Nov 03 22:55:59.509419 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:55:59.509401 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6ebc6d92b6c75854e3b164675a8823dfd49bc87618514b10a26d81a95e7a675\": container with ID starting with b6ebc6d92b6c75854e3b164675a8823dfd49bc87618514b10a26d81a95e7a675 not found: ID does not exist" containerID="b6ebc6d92b6c75854e3b164675a8823dfd49bc87618514b10a26d81a95e7a675" Nov 03 22:55:59.509470 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:59.509431 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6ebc6d92b6c75854e3b164675a8823dfd49bc87618514b10a26d81a95e7a675"} err="failed to get container status \"b6ebc6d92b6c75854e3b164675a8823dfd49bc87618514b10a26d81a95e7a675\": rpc error: code = NotFound desc = could not find container \"b6ebc6d92b6c75854e3b164675a8823dfd49bc87618514b10a26d81a95e7a675\": container with ID starting with b6ebc6d92b6c75854e3b164675a8823dfd49bc87618514b10a26d81a95e7a675 not found: ID does not exist" Nov 03 22:55:59.671437 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:59.671398 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk"] Nov 03 22:55:59.675469 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:55:59.675443 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-shfnk"] Nov 03 22:56:00.498623 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:00.498587 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:00.816427 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:00.816401 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25d699f6-aaae-480e-9efb-45afb8297cce" path="/var/lib/kubelet/pods/25d699f6-aaae-480e-9efb-45afb8297cce/volumes" Nov 03 22:56:00.820192 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:00.820172 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" Nov 03 22:56:00.857081 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:00.857053 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0ca4784-6fe2-4001-ac15-25675fc41a02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ca4784-6fe2-4001-ac15-25675fc41a02-istio-podinfo\") pod \"f0ca4784-6fe2-4001-ac15-25675fc41a02\" (UID: \"f0ca4784-6fe2-4001-ac15-25675fc41a02\") " Nov 03 22:56:00.857213 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:00.857108 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0ca4784-6fe2-4001-ac15-25675fc41a02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ca4784-6fe2-4001-ac15-25675fc41a02-workload-socket\") pod \"f0ca4784-6fe2-4001-ac15-25675fc41a02\" (UID: \"f0ca4784-6fe2-4001-ac15-25675fc41a02\") " Nov 03 22:56:00.857213 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:00.857159 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0ca4784-6fe2-4001-ac15-25675fc41a02-kube-api-access-hpmrb\" (UniqueName: \"kubernetes.io/projected/f0ca4784-6fe2-4001-ac15-25675fc41a02-kube-api-access-hpmrb\") pod \"f0ca4784-6fe2-4001-ac15-25675fc41a02\" (UID: \"f0ca4784-6fe2-4001-ac15-25675fc41a02\") " Nov 03 22:56:00.857368 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:00.857309 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0ca4784-6fe2-4001-ac15-25675fc41a02-istio-token\" (UniqueName: \"kubernetes.io/projected/f0ca4784-6fe2-4001-ac15-25675fc41a02-istio-token\") pod \"f0ca4784-6fe2-4001-ac15-25675fc41a02\" (UID: \"f0ca4784-6fe2-4001-ac15-25675fc41a02\") " Nov 03 22:56:00.857430 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:00.857406 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0ca4784-6fe2-4001-ac15-25675fc41a02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ca4784-6fe2-4001-ac15-25675fc41a02-credential-socket\") pod \"f0ca4784-6fe2-4001-ac15-25675fc41a02\" (UID: \"f0ca4784-6fe2-4001-ac15-25675fc41a02\") " Nov 03 22:56:00.857490 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:00.857428 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ca4784-6fe2-4001-ac15-25675fc41a02-workload-socket" pod "f0ca4784-6fe2-4001-ac15-25675fc41a02" (UID: "f0ca4784-6fe2-4001-ac15-25675fc41a02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:00.857490 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:00.857464 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0ca4784-6fe2-4001-ac15-25675fc41a02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ca4784-6fe2-4001-ac15-25675fc41a02-istiod-ca-cert\") pod \"f0ca4784-6fe2-4001-ac15-25675fc41a02\" (UID: \"f0ca4784-6fe2-4001-ac15-25675fc41a02\") " Nov 03 22:56:00.857596 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:00.857498 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0ca4784-6fe2-4001-ac15-25675fc41a02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ca4784-6fe2-4001-ac15-25675fc41a02-istio-data\") pod \"f0ca4784-6fe2-4001-ac15-25675fc41a02\" (UID: \"f0ca4784-6fe2-4001-ac15-25675fc41a02\") " Nov 03 22:56:00.857596 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:00.857560 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0ca4784-6fe2-4001-ac15-25675fc41a02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ca4784-6fe2-4001-ac15-25675fc41a02-workload-certs\") pod \"f0ca4784-6fe2-4001-ac15-25675fc41a02\" (UID: \"f0ca4784-6fe2-4001-ac15-25675fc41a02\") " Nov 03 22:56:00.857705 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:00.857690 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ca4784-6fe2-4001-ac15-25675fc41a02-credential-socket" pod "f0ca4784-6fe2-4001-ac15-25675fc41a02" (UID: "f0ca4784-6fe2-4001-ac15-25675fc41a02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:00.857752 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:00.857705 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0ca4784-6fe2-4001-ac15-25675fc41a02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ca4784-6fe2-4001-ac15-25675fc41a02-istio-envoy\") pod \"f0ca4784-6fe2-4001-ac15-25675fc41a02\" (UID: \"f0ca4784-6fe2-4001-ac15-25675fc41a02\") " Nov 03 22:56:00.858182 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:00.857891 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ca4784-6fe2-4001-ac15-25675fc41a02-istio-data" pod "f0ca4784-6fe2-4001-ac15-25675fc41a02" (UID: "f0ca4784-6fe2-4001-ac15-25675fc41a02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:00.858316 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:00.858221 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ca4784-6fe2-4001-ac15-25675fc41a02-workload-certs" pod "f0ca4784-6fe2-4001-ac15-25675fc41a02" (UID: "f0ca4784-6fe2-4001-ac15-25675fc41a02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:00.858394 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:00.858312 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0ca4784-6fe2-4001-ac15-25675fc41a02-istiod-ca-cert" pod "f0ca4784-6fe2-4001-ac15-25675fc41a02" (UID: "f0ca4784-6fe2-4001-ac15-25675fc41a02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:56:00.858926 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:00.858901 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ca4784-6fe2-4001-ac15-25675fc41a02-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:00.859029 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:00.858931 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ca4784-6fe2-4001-ac15-25675fc41a02-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:00.859029 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:00.858948 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ca4784-6fe2-4001-ac15-25675fc41a02-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:00.859029 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:00.858963 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ca4784-6fe2-4001-ac15-25675fc41a02-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:00.859029 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:00.858977 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ca4784-6fe2-4001-ac15-25675fc41a02-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:00.859909 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:00.859844 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ca4784-6fe2-4001-ac15-25675fc41a02-istio-token" pod "f0ca4784-6fe2-4001-ac15-25675fc41a02" (UID: "f0ca4784-6fe2-4001-ac15-25675fc41a02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:00.860130 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:00.860110 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0ca4784-6fe2-4001-ac15-25675fc41a02-istio-podinfo" pod "f0ca4784-6fe2-4001-ac15-25675fc41a02" (UID: "f0ca4784-6fe2-4001-ac15-25675fc41a02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:56:00.860176 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:00.860148 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ca4784-6fe2-4001-ac15-25675fc41a02-kube-api-access-hpmrb" pod "f0ca4784-6fe2-4001-ac15-25675fc41a02" (UID: "f0ca4784-6fe2-4001-ac15-25675fc41a02"). InnerVolumeSpecName "kube-api-access-hpmrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:00.860256 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:00.860235 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ca4784-6fe2-4001-ac15-25675fc41a02-istio-envoy" pod "f0ca4784-6fe2-4001-ac15-25675fc41a02" (UID: "f0ca4784-6fe2-4001-ac15-25675fc41a02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:00.960793 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:00.960759 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ca4784-6fe2-4001-ac15-25675fc41a02-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:00.960793 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:00.960789 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ca4784-6fe2-4001-ac15-25675fc41a02-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:00.961057 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:00.960820 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpmrb\" (UniqueName: \"kubernetes.io/projected/f0ca4784-6fe2-4001-ac15-25675fc41a02-kube-api-access-hpmrb\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:00.961057 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:00.960830 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0ca4784-6fe2-4001-ac15-25675fc41a02-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:01.506847 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.506809 5196 generic.go:356] "Generic (PLEG): container finished" podID="05bd48b9-ff9d-4511-9c48-9878b3ae40b3" containerID="e17b0e300b97020c9d7cc98e22a5f4587e929ee39d67dbc057cf160aabe36a17" exitCode=0 Nov 03 22:56:01.507335 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.506913 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" event={"ID":"05bd48b9-ff9d-4511-9c48-9878b3ae40b3","Type":"ContainerDied","Data":"e17b0e300b97020c9d7cc98e22a5f4587e929ee39d67dbc057cf160aabe36a17"} Nov 03 22:56:01.508852 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.508824 5196 generic.go:356] "Generic (PLEG): container finished" podID="734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0" containerID="c4da88202d281925af5ce892dda476ca0edd932079194e3ce9a72b2b5aa40adf" exitCode=0 Nov 03 22:56:01.509015 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.508886 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" event={"ID":"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0","Type":"ContainerDied","Data":"c4da88202d281925af5ce892dda476ca0edd932079194e3ce9a72b2b5aa40adf"} Nov 03 22:56:01.510368 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.510342 5196 generic.go:356] "Generic (PLEG): container finished" podID="f0ca4784-6fe2-4001-ac15-25675fc41a02" containerID="4b8f9e503d3700c74c892a7657e38946c9882fbfe1d07d5054b5c29be7bc4a55" exitCode=0 Nov 03 22:56:01.510599 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.510385 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" event={"ID":"f0ca4784-6fe2-4001-ac15-25675fc41a02","Type":"ContainerDied","Data":"4b8f9e503d3700c74c892a7657e38946c9882fbfe1d07d5054b5c29be7bc4a55"} Nov 03 22:56:01.510700 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.510627 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" event={"ID":"f0ca4784-6fe2-4001-ac15-25675fc41a02","Type":"ContainerDied","Data":"3df1b738af557f12b6097649bc0fe68c76ffe4f2a9b1721440d1811d5e853244"} Nov 03 22:56:01.510700 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.510664 5196 scope.go:122] "RemoveContainer" containerID="4b8f9e503d3700c74c892a7657e38946c9882fbfe1d07d5054b5c29be7bc4a55" Nov 03 22:56:01.510822 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.510424 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v" Nov 03 22:56:01.525850 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.525825 5196 scope.go:122] "RemoveContainer" containerID="4b8f9e503d3700c74c892a7657e38946c9882fbfe1d07d5054b5c29be7bc4a55" Nov 03 22:56:01.526498 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:56:01.526470 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b8f9e503d3700c74c892a7657e38946c9882fbfe1d07d5054b5c29be7bc4a55\": container with ID starting with 4b8f9e503d3700c74c892a7657e38946c9882fbfe1d07d5054b5c29be7bc4a55 not found: ID does not exist" containerID="4b8f9e503d3700c74c892a7657e38946c9882fbfe1d07d5054b5c29be7bc4a55" Nov 03 22:56:01.526614 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.526512 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b8f9e503d3700c74c892a7657e38946c9882fbfe1d07d5054b5c29be7bc4a55"} err="failed to get container status \"4b8f9e503d3700c74c892a7657e38946c9882fbfe1d07d5054b5c29be7bc4a55\": rpc error: code = NotFound desc = could not find container \"4b8f9e503d3700c74c892a7657e38946c9882fbfe1d07d5054b5c29be7bc4a55\": container with ID starting with 4b8f9e503d3700c74c892a7657e38946c9882fbfe1d07d5054b5c29be7bc4a55 not found: ID does not exist" Nov 03 22:56:01.539843 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.539786 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" Nov 03 22:56:01.570069 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.570032 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-kube-api-access-vgwjh\" (UniqueName: \"kubernetes.io/projected/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-kube-api-access-vgwjh\") pod \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\" (UID: \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\") " Nov 03 22:56:01.570207 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.570166 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-workload-socket\") pod \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\" (UID: \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\") " Nov 03 22:56:01.570207 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.570201 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-istio-podinfo\") pod \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\" (UID: \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\") " Nov 03 22:56:01.570328 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.570283 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-istio-envoy\") pod \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\" (UID: \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\") " Nov 03 22:56:01.570402 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.570334 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-istiod-ca-cert\") pod \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\" (UID: \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\") " Nov 03 22:56:01.570402 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.570380 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-credential-socket\") pod \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\" (UID: \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\") " Nov 03 22:56:01.570513 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.570452 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-istio-data\") pod \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\" (UID: \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\") " Nov 03 22:56:01.570513 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.570485 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-workload-socket" pod "05bd48b9-ff9d-4511-9c48-9878b3ae40b3" (UID: "05bd48b9-ff9d-4511-9c48-9878b3ae40b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:01.570513 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.570508 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-istio-token\" (UniqueName: \"kubernetes.io/projected/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-istio-token\") pod \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\" (UID: \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\") " Nov 03 22:56:01.570680 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.570582 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-workload-certs\") pod \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\" (UID: \"05bd48b9-ff9d-4511-9c48-9878b3ae40b3\") " Nov 03 22:56:01.570819 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.570791 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-istio-data" pod "05bd48b9-ff9d-4511-9c48-9878b3ae40b3" (UID: "05bd48b9-ff9d-4511-9c48-9878b3ae40b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:01.571089 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.571063 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-credential-socket" pod "05bd48b9-ff9d-4511-9c48-9878b3ae40b3" (UID: "05bd48b9-ff9d-4511-9c48-9878b3ae40b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:01.571247 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.571220 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-istiod-ca-cert" pod "05bd48b9-ff9d-4511-9c48-9878b3ae40b3" (UID: "05bd48b9-ff9d-4511-9c48-9878b3ae40b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:56:01.571403 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.571282 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-workload-certs" pod "05bd48b9-ff9d-4511-9c48-9878b3ae40b3" (UID: "05bd48b9-ff9d-4511-9c48-9878b3ae40b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:01.571634 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.571611 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:01.571708 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.571640 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:01.571708 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.571656 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:01.571708 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.571670 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:01.571708 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.571684 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:01.572545 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.572526 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-kube-api-access-vgwjh" pod "05bd48b9-ff9d-4511-9c48-9878b3ae40b3" (UID: "05bd48b9-ff9d-4511-9c48-9878b3ae40b3"). InnerVolumeSpecName "kube-api-access-vgwjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:01.572823 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.572798 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-istio-envoy" pod "05bd48b9-ff9d-4511-9c48-9878b3ae40b3" (UID: "05bd48b9-ff9d-4511-9c48-9878b3ae40b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:01.572931 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.572913 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-istio-podinfo" pod "05bd48b9-ff9d-4511-9c48-9878b3ae40b3" (UID: "05bd48b9-ff9d-4511-9c48-9878b3ae40b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:56:01.573218 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.573193 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-istio-token" pod "05bd48b9-ff9d-4511-9c48-9878b3ae40b3" (UID: "05bd48b9-ff9d-4511-9c48-9878b3ae40b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:01.606146 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.606118 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" Nov 03 22:56:01.672628 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.672597 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-istio-token\" (UniqueName: \"kubernetes.io/projected/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-istio-token\") pod \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\" (UID: \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\") " Nov 03 22:56:01.672828 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.672647 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-workload-certs\") pod \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\" (UID: \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\") " Nov 03 22:56:01.672828 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.672713 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-istio-envoy\") pod \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\" (UID: \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\") " Nov 03 22:56:01.672828 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.672740 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-workload-socket\") pod \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\" (UID: \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\") " Nov 03 22:56:01.672828 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.672776 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-credential-socket\") pod \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\" (UID: \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\") " Nov 03 22:56:01.672828 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.672814 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-istiod-ca-cert\") pod \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\" (UID: \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\") " Nov 03 22:56:01.673128 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.672941 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-kube-api-access-shjhk\" (UniqueName: \"kubernetes.io/projected/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-kube-api-access-shjhk\") pod \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\" (UID: \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\") " Nov 03 22:56:01.673128 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.672988 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-istio-data\") pod \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\" (UID: \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\") " Nov 03 22:56:01.673128 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.673021 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-workload-certs" pod "734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0" (UID: "734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:01.673128 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.673061 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-workload-socket" pod "734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0" (UID: "734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:01.673128 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.673113 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-istio-podinfo\") pod \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\" (UID: \"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0\") " Nov 03 22:56:01.673369 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.673108 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-credential-socket" pod "734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0" (UID: "734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:01.673369 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.673306 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-istio-data" pod "734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0" (UID: "734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:01.673369 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.673339 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-istiod-ca-cert" pod "734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0" (UID: "734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:56:01.673741 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.673678 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:01.673741 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.673698 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:01.673741 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.673708 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:01.673741 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.673716 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:01.673741 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.673724 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:01.673741 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.673736 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:01.673741 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.673747 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:01.673741 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.673757 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:01.673741 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.673766 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgwjh\" (UniqueName: \"kubernetes.io/projected/05bd48b9-ff9d-4511-9c48-9878b3ae40b3-kube-api-access-vgwjh\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:01.675295 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.675271 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-istio-envoy" pod "734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0" (UID: "734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:56:01.675422 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.675375 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-istio-podinfo" pod "734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0" (UID: "734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:56:01.675719 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.675703 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-kube-api-access-shjhk" pod "734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0" (UID: "734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0"). InnerVolumeSpecName "kube-api-access-shjhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:01.675761 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.675726 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-istio-token" pod "734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0" (UID: "734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:01.775494 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.775444 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:01.775494 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.775486 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shjhk\" (UniqueName: \"kubernetes.io/projected/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-kube-api-access-shjhk\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:01.775758 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.775505 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:01.775758 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.775519 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:01.978999 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.978935 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v"] Nov 03 22:56:01.982273 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:01.982247 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-wlh8v"] Nov 03 22:56:02.518391 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:02.518357 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" Nov 03 22:56:02.518391 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:02.518387 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n" event={"ID":"05bd48b9-ff9d-4511-9c48-9878b3ae40b3","Type":"ContainerDied","Data":"f3441ae4fcdbc6e1a95aef74dfde2f640034db57b2d7659a35e88085f12c3959"} Nov 03 22:56:02.518983 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:02.518445 5196 scope.go:122] "RemoveContainer" containerID="e17b0e300b97020c9d7cc98e22a5f4587e929ee39d67dbc057cf160aabe36a17" Nov 03 22:56:02.519910 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:02.519894 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" Nov 03 22:56:02.519910 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:02.519902 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c" event={"ID":"734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0","Type":"ContainerDied","Data":"bea4d0e91dc3afd112d225c41b4a3e472b4f8fdf0b54b5b1246c4db0a6a7124e"} Nov 03 22:56:02.530604 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:02.530583 5196 scope.go:122] "RemoveContainer" containerID="c4da88202d281925af5ce892dda476ca0edd932079194e3ce9a72b2b5aa40adf" Nov 03 22:56:02.687288 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:02.687254 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c"] Nov 03 22:56:02.737406 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:02.737371 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-w6k2c"] Nov 03 22:56:02.801755 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:02.801675 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n"] Nov 03 22:56:02.816431 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:02.816401 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0" path="/var/lib/kubelet/pods/734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0/volumes" Nov 03 22:56:02.817238 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:02.817214 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0ca4784-6fe2-4001-ac15-25675fc41a02" path="/var/lib/kubelet/pods/f0ca4784-6fe2-4001-ac15-25675fc41a02/volumes" Nov 03 22:56:02.817555 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:02.817542 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-cb25n"] Nov 03 22:56:04.817798 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:04.817757 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05bd48b9-ff9d-4511-9c48-9878b3ae40b3" path="/var/lib/kubelet/pods/05bd48b9-ff9d-4511-9c48-9878b3ae40b3/volumes" Nov 03 22:56:05.508769 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:05.508735 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:06.436296 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:06.436257 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:06.631494 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:06.631460 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:07.629756 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:07.629720 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:10.698292 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:10.698263 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:13.135988 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:13.135955 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:14.955905 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:14.955852 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:15.150137 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:15.149972 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:15.790191 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:15.790154 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:20.755246 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:20.755192 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:25.598251 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:25.598213 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:25.919196 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:56:25.919118 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:56:30.710468 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:30.710436 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:35.598244 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:35.598204 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:37.042681 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:37.042646 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:38.390440 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:38.390407 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:38.536251 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:38.536217 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:39.898142 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:39.898105 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:40.046192 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:40.046153 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:40.212930 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:40.212828 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:40.963721 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:40.963688 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:41.111081 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:41.111045 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:41.951359 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:41.951326 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:43.314030 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:43.314000 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:46.081157 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:46.081119 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp"] Nov 03 22:56:46.081573 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:46.081324 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp" podUID="ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4" containerName="maas-api" containerID="cri-o://a9a942e4d022a7d71e95511b2e20233eb3848f98d194a0837fbfa236a5162fe7" gracePeriod=30 Nov 03 22:56:46.197676 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:46.197645 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:46.433248 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:46.433220 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp" Nov 03 22:56:46.489882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:46.480263 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4-maas-api-tls\") pod \"ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4\" (UID: \"ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4\") " Nov 03 22:56:46.489882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:46.480342 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4-openshift-service-ca\") pod \"ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4\" (UID: \"ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4\") " Nov 03 22:56:46.489882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:46.480427 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4-metrics-certs\") pod \"ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4\" (UID: \"ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4\") " Nov 03 22:56:46.489882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:46.480535 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4-kube-api-access-6dkg8\" (UniqueName: \"kubernetes.io/projected/ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4-kube-api-access-6dkg8\") pod \"ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4\" (UID: \"ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4\") " Nov 03 22:56:46.489882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:46.484262 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4-openshift-service-ca" pod "ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4" (UID: "ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:56:46.496368 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:46.496335 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4-metrics-certs" pod "ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4" (UID: "ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:56:46.498277 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:46.498236 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4-kube-api-access-6dkg8" pod "ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4" (UID: "ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4"). InnerVolumeSpecName "kube-api-access-6dkg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:46.498759 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:46.498725 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4-maas-api-tls" pod "ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4" (UID: "ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:56:46.584878 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:46.583240 5196 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4-metrics-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:46.584878 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:46.583278 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dkg8\" (UniqueName: \"kubernetes.io/projected/ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4-kube-api-access-6dkg8\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:46.584878 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:46.583295 5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4-maas-api-tls\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:46.584878 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:46.583309 5196 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4-openshift-service-ca\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:56:46.834113 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:46.834084 5196 generic.go:356] "Generic (PLEG): container finished" podID="ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4" containerID="a9a942e4d022a7d71e95511b2e20233eb3848f98d194a0837fbfa236a5162fe7" exitCode=0 Nov 03 22:56:46.834249 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:46.834132 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp" Nov 03 22:56:46.834249 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:46.834169 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp" event={"ID":"ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4","Type":"ContainerDied","Data":"a9a942e4d022a7d71e95511b2e20233eb3848f98d194a0837fbfa236a5162fe7"} Nov 03 22:56:46.834249 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:46.834223 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp" event={"ID":"ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4","Type":"ContainerDied","Data":"c6f7bcdefb3753f30df5633a51807a87029d924073f15659ee9b529b141ad9c7"} Nov 03 22:56:46.834353 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:46.834252 5196 scope.go:122] "RemoveContainer" containerID="a9a942e4d022a7d71e95511b2e20233eb3848f98d194a0837fbfa236a5162fe7" Nov 03 22:56:46.845274 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:46.845255 5196 scope.go:122] "RemoveContainer" containerID="a9a942e4d022a7d71e95511b2e20233eb3848f98d194a0837fbfa236a5162fe7" Nov 03 22:56:46.845548 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:56:46.845528 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9a942e4d022a7d71e95511b2e20233eb3848f98d194a0837fbfa236a5162fe7\": container with ID starting with a9a942e4d022a7d71e95511b2e20233eb3848f98d194a0837fbfa236a5162fe7 not found: ID does not exist" containerID="a9a942e4d022a7d71e95511b2e20233eb3848f98d194a0837fbfa236a5162fe7" Nov 03 22:56:46.845614 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:46.845558 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9a942e4d022a7d71e95511b2e20233eb3848f98d194a0837fbfa236a5162fe7"} err="failed to get container status \"a9a942e4d022a7d71e95511b2e20233eb3848f98d194a0837fbfa236a5162fe7\": rpc error: code = NotFound desc = could not find container \"a9a942e4d022a7d71e95511b2e20233eb3848f98d194a0837fbfa236a5162fe7\": container with ID starting with a9a942e4d022a7d71e95511b2e20233eb3848f98d194a0837fbfa236a5162fe7 not found: ID does not exist" Nov 03 22:56:47.006753 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:47.006724 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp"] Nov 03 22:56:47.086935 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:47.086295 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-efce83-857b6c76f5-nz8tp"] Nov 03 22:56:47.362521 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:47.362489 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:47.741936 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:47.739643 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:48.020214 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:48.020143 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:48.791726 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:48.791693 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:48.815635 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:48.815606 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4" path="/var/lib/kubelet/pods/ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4/volumes" Nov 03 22:56:48.985634 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:48.985606 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:49.199898 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:49.199829 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:49.397929 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:49.397896 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:53.546776 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:53.546742 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:54.312402 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:54.312367 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:55.896675 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:55.896639 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:55.920829 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:56:55.920337 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:56:56.076097 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:56.076054 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:56.271964 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:56.271892 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:57.050517 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:57.050455 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:57.229427 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:57.229374 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:58.339236 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:58.339188 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:58.973053 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:58.971912 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:59.384128 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:59.384079 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:59.537811 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:59.537781 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:59.686164 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:59.686079 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:56:59.839346 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:56:59.839302 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:00.031202 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:00.030048 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:00.182172 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:00.182139 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:03.533398 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:03.533365 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:03.713108 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:03.713073 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:03.859009 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:03.858979 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:04.899528 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:04.899488 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:06.748265 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:06.748232 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:07.007218 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:07.007137 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:08.041175 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:08.041112 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:09.237073 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:09.237016 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-26444f-8676cfdd67-tp9js"] Nov 03 22:57:09.239126 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:09.237841 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26444f-8676cfdd67-tp9js" podUID="bbe0a3aa-0540-40df-ba43-9196bf53b562" containerName="maas-api" containerID="cri-o://1bd37d93403a9e2d5f54a641a1f0941f8b6410f23b150b9125b98d349aabfa12" gracePeriod=30 Nov 03 22:57:09.643759 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:09.643643 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:09.818880 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:09.817881 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26444f-8676cfdd67-tp9js" Nov 03 22:57:09.922692 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:09.922589 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbe0a3aa-0540-40df-ba43-9196bf53b562-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbe0a3aa-0540-40df-ba43-9196bf53b562-openshift-service-ca\") pod \"bbe0a3aa-0540-40df-ba43-9196bf53b562\" (UID: \"bbe0a3aa-0540-40df-ba43-9196bf53b562\") " Nov 03 22:57:09.922692 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:09.922682 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbe0a3aa-0540-40df-ba43-9196bf53b562-kube-api-access-s7cwc\" (UniqueName: \"kubernetes.io/projected/bbe0a3aa-0540-40df-ba43-9196bf53b562-kube-api-access-s7cwc\") pod \"bbe0a3aa-0540-40df-ba43-9196bf53b562\" (UID: \"bbe0a3aa-0540-40df-ba43-9196bf53b562\") " Nov 03 22:57:09.923001 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:09.922737 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbe0a3aa-0540-40df-ba43-9196bf53b562-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbe0a3aa-0540-40df-ba43-9196bf53b562-metrics-certs\") pod \"bbe0a3aa-0540-40df-ba43-9196bf53b562\" (UID: \"bbe0a3aa-0540-40df-ba43-9196bf53b562\") " Nov 03 22:57:09.923001 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:09.922950 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbe0a3aa-0540-40df-ba43-9196bf53b562-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbe0a3aa-0540-40df-ba43-9196bf53b562-maas-api-tls\") pod \"bbe0a3aa-0540-40df-ba43-9196bf53b562\" (UID: \"bbe0a3aa-0540-40df-ba43-9196bf53b562\") " Nov 03 22:57:09.923137 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:09.923018 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbe0a3aa-0540-40df-ba43-9196bf53b562-openshift-service-ca" pod "bbe0a3aa-0540-40df-ba43-9196bf53b562" (UID: "bbe0a3aa-0540-40df-ba43-9196bf53b562"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:57:09.923737 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:09.923713 5196 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbe0a3aa-0540-40df-ba43-9196bf53b562-openshift-service-ca\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:57:09.926284 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:09.925908 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbe0a3aa-0540-40df-ba43-9196bf53b562-kube-api-access-s7cwc" pod "bbe0a3aa-0540-40df-ba43-9196bf53b562" (UID: "bbe0a3aa-0540-40df-ba43-9196bf53b562"). InnerVolumeSpecName "kube-api-access-s7cwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:57:09.926284 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:09.926024 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbe0a3aa-0540-40df-ba43-9196bf53b562-maas-api-tls" pod "bbe0a3aa-0540-40df-ba43-9196bf53b562" (UID: "bbe0a3aa-0540-40df-ba43-9196bf53b562"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:57:09.928307 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:09.928278 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbe0a3aa-0540-40df-ba43-9196bf53b562-metrics-certs" pod "bbe0a3aa-0540-40df-ba43-9196bf53b562" (UID: "bbe0a3aa-0540-40df-ba43-9196bf53b562"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:57:10.024876 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:10.024831 5196 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbe0a3aa-0540-40df-ba43-9196bf53b562-metrics-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:57:10.025132 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:10.024898 5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbe0a3aa-0540-40df-ba43-9196bf53b562-maas-api-tls\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:57:10.025132 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:10.024908 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7cwc\" (UniqueName: \"kubernetes.io/projected/bbe0a3aa-0540-40df-ba43-9196bf53b562-kube-api-access-s7cwc\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:57:10.172150 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:10.172110 5196 generic.go:356] "Generic (PLEG): container finished" podID="bbe0a3aa-0540-40df-ba43-9196bf53b562" containerID="1bd37d93403a9e2d5f54a641a1f0941f8b6410f23b150b9125b98d349aabfa12" exitCode=0 Nov 03 22:57:10.172356 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:10.172175 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26444f-8676cfdd67-tp9js" event={"ID":"bbe0a3aa-0540-40df-ba43-9196bf53b562","Type":"ContainerDied","Data":"1bd37d93403a9e2d5f54a641a1f0941f8b6410f23b150b9125b98d349aabfa12"} Nov 03 22:57:10.172356 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:10.172220 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26444f-8676cfdd67-tp9js" event={"ID":"bbe0a3aa-0540-40df-ba43-9196bf53b562","Type":"ContainerDied","Data":"36e860e8642033e7474b9214410d547e7c5f661d5ce7a52906dfbc0b2e59ada4"} Nov 03 22:57:10.172356 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:10.172242 5196 scope.go:122] "RemoveContainer" containerID="1bd37d93403a9e2d5f54a641a1f0941f8b6410f23b150b9125b98d349aabfa12" Nov 03 22:57:10.172356 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:10.172359 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-26444f-8676cfdd67-tp9js" Nov 03 22:57:10.188182 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:10.188155 5196 scope.go:122] "RemoveContainer" containerID="1bd37d93403a9e2d5f54a641a1f0941f8b6410f23b150b9125b98d349aabfa12" Nov 03 22:57:10.188678 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:57:10.188648 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bd37d93403a9e2d5f54a641a1f0941f8b6410f23b150b9125b98d349aabfa12\": container with ID starting with 1bd37d93403a9e2d5f54a641a1f0941f8b6410f23b150b9125b98d349aabfa12 not found: ID does not exist" containerID="1bd37d93403a9e2d5f54a641a1f0941f8b6410f23b150b9125b98d349aabfa12" Nov 03 22:57:10.188788 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:10.188688 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bd37d93403a9e2d5f54a641a1f0941f8b6410f23b150b9125b98d349aabfa12"} err="failed to get container status \"1bd37d93403a9e2d5f54a641a1f0941f8b6410f23b150b9125b98d349aabfa12\": rpc error: code = NotFound desc = could not find container \"1bd37d93403a9e2d5f54a641a1f0941f8b6410f23b150b9125b98d349aabfa12\": container with ID starting with 1bd37d93403a9e2d5f54a641a1f0941f8b6410f23b150b9125b98d349aabfa12 not found: ID does not exist" Nov 03 22:57:10.345778 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:10.345746 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-26444f-8676cfdd67-tp9js"] Nov 03 22:57:10.351670 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:10.350690 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-26444f-8676cfdd67-tp9js"] Nov 03 22:57:10.859998 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:10.859961 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbe0a3aa-0540-40df-ba43-9196bf53b562" path="/var/lib/kubelet/pods/bbe0a3aa-0540-40df-ba43-9196bf53b562/volumes" Nov 03 22:57:10.941095 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:10.941065 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:11.171077 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:11.170962 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:11.327883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:11.326902 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:11.929873 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:11.929772 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:13.487259 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:13.487225 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:16.218278 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:16.218239 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:17.992579 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:17.992545 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:18.141029 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:18.140997 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:18.305284 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:18.305189 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk"] Nov 03 22:57:18.305482 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:18.305457 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" podUID="0474bb77-105a-4009-9f7e-2e19a4068fc9" containerName="istio-proxy" containerID="cri-o://af6e3c7027e1ea8724e0f847c1964538b3c227ca425714dd6c0791dee7bd29f3" gracePeriod=30 Nov 03 22:57:18.315536 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:18.315508 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:19.610757 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:19.610723 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:19.759194 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:19.759155 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:20.409679 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:20.409645 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:20.563285 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:20.563252 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:21.157942 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:21.157842 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:21.616554 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:21.616524 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:21.974004 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:21.973920 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:22.238354 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:22.238275 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:22.394258 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:22.394224 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:22.874978 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:22.874945 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:23.562748 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:23.562724 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:57:23.616756 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:23.616726 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0474bb77-105a-4009-9f7e-2e19a4068fc9-istio-token\" (UniqueName: \"kubernetes.io/projected/0474bb77-105a-4009-9f7e-2e19a4068fc9-istio-token\") pod \"0474bb77-105a-4009-9f7e-2e19a4068fc9\" (UID: \"0474bb77-105a-4009-9f7e-2e19a4068fc9\") " Nov 03 22:57:23.616941 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:23.616768 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0474bb77-105a-4009-9f7e-2e19a4068fc9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0474bb77-105a-4009-9f7e-2e19a4068fc9-istio-podinfo\") pod \"0474bb77-105a-4009-9f7e-2e19a4068fc9\" (UID: \"0474bb77-105a-4009-9f7e-2e19a4068fc9\") " Nov 03 22:57:23.616941 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:23.616801 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0474bb77-105a-4009-9f7e-2e19a4068fc9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0474bb77-105a-4009-9f7e-2e19a4068fc9-credential-socket\") pod \"0474bb77-105a-4009-9f7e-2e19a4068fc9\" (UID: \"0474bb77-105a-4009-9f7e-2e19a4068fc9\") " Nov 03 22:57:23.617071 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:23.616941 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0474bb77-105a-4009-9f7e-2e19a4068fc9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0474bb77-105a-4009-9f7e-2e19a4068fc9-workload-certs\") pod \"0474bb77-105a-4009-9f7e-2e19a4068fc9\" (UID: \"0474bb77-105a-4009-9f7e-2e19a4068fc9\") " Nov 03 22:57:23.617071 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:23.616982 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0474bb77-105a-4009-9f7e-2e19a4068fc9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0474bb77-105a-4009-9f7e-2e19a4068fc9-istio-envoy\") pod \"0474bb77-105a-4009-9f7e-2e19a4068fc9\" (UID: \"0474bb77-105a-4009-9f7e-2e19a4068fc9\") " Nov 03 22:57:23.617071 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:23.617011 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0474bb77-105a-4009-9f7e-2e19a4068fc9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0474bb77-105a-4009-9f7e-2e19a4068fc9-istiod-ca-cert\") pod \"0474bb77-105a-4009-9f7e-2e19a4068fc9\" (UID: \"0474bb77-105a-4009-9f7e-2e19a4068fc9\") " Nov 03 22:57:23.617228 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:23.617069 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0474bb77-105a-4009-9f7e-2e19a4068fc9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0474bb77-105a-4009-9f7e-2e19a4068fc9-workload-socket\") pod \"0474bb77-105a-4009-9f7e-2e19a4068fc9\" (UID: \"0474bb77-105a-4009-9f7e-2e19a4068fc9\") " Nov 03 22:57:23.617228 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:23.617136 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0474bb77-105a-4009-9f7e-2e19a4068fc9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0474bb77-105a-4009-9f7e-2e19a4068fc9-istio-data\") pod \"0474bb77-105a-4009-9f7e-2e19a4068fc9\" (UID: \"0474bb77-105a-4009-9f7e-2e19a4068fc9\") " Nov 03 22:57:23.617228 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:23.617171 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0474bb77-105a-4009-9f7e-2e19a4068fc9-credential-socket" pod "0474bb77-105a-4009-9f7e-2e19a4068fc9" (UID: "0474bb77-105a-4009-9f7e-2e19a4068fc9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:57:23.617383 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:23.617222 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0474bb77-105a-4009-9f7e-2e19a4068fc9-workload-certs" pod "0474bb77-105a-4009-9f7e-2e19a4068fc9" (UID: "0474bb77-105a-4009-9f7e-2e19a4068fc9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:57:23.617383 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:23.617235 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0474bb77-105a-4009-9f7e-2e19a4068fc9-kube-api-access-c9cvn\" (UniqueName: \"kubernetes.io/projected/0474bb77-105a-4009-9f7e-2e19a4068fc9-kube-api-access-c9cvn\") pod \"0474bb77-105a-4009-9f7e-2e19a4068fc9\" (UID: \"0474bb77-105a-4009-9f7e-2e19a4068fc9\") " Nov 03 22:57:23.617499 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:23.617390 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0474bb77-105a-4009-9f7e-2e19a4068fc9-workload-socket" pod "0474bb77-105a-4009-9f7e-2e19a4068fc9" (UID: "0474bb77-105a-4009-9f7e-2e19a4068fc9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:57:23.617499 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:23.617403 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0474bb77-105a-4009-9f7e-2e19a4068fc9-istio-data" pod "0474bb77-105a-4009-9f7e-2e19a4068fc9" (UID: "0474bb77-105a-4009-9f7e-2e19a4068fc9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:57:23.617612 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:23.617553 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0474bb77-105a-4009-9f7e-2e19a4068fc9-istiod-ca-cert" pod "0474bb77-105a-4009-9f7e-2e19a4068fc9" (UID: "0474bb77-105a-4009-9f7e-2e19a4068fc9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:57:23.617830 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:23.617809 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0474bb77-105a-4009-9f7e-2e19a4068fc9-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:57:23.617972 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:23.617832 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0474bb77-105a-4009-9f7e-2e19a4068fc9-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:57:23.617972 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:23.617846 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0474bb77-105a-4009-9f7e-2e19a4068fc9-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:57:23.617972 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:23.617876 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0474bb77-105a-4009-9f7e-2e19a4068fc9-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:57:23.617972 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:23.617888 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0474bb77-105a-4009-9f7e-2e19a4068fc9-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:57:23.619286 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:23.619253 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0474bb77-105a-4009-9f7e-2e19a4068fc9-istio-envoy" pod "0474bb77-105a-4009-9f7e-2e19a4068fc9" (UID: "0474bb77-105a-4009-9f7e-2e19a4068fc9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:57:23.619584 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:23.619552 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0474bb77-105a-4009-9f7e-2e19a4068fc9-istio-podinfo" pod "0474bb77-105a-4009-9f7e-2e19a4068fc9" (UID: "0474bb77-105a-4009-9f7e-2e19a4068fc9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:57:23.619584 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:23.619565 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0474bb77-105a-4009-9f7e-2e19a4068fc9-istio-token" pod "0474bb77-105a-4009-9f7e-2e19a4068fc9" (UID: "0474bb77-105a-4009-9f7e-2e19a4068fc9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:57:23.619732 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:23.619642 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0474bb77-105a-4009-9f7e-2e19a4068fc9-kube-api-access-c9cvn" pod "0474bb77-105a-4009-9f7e-2e19a4068fc9" (UID: "0474bb77-105a-4009-9f7e-2e19a4068fc9"). InnerVolumeSpecName "kube-api-access-c9cvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:57:23.719501 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:23.719475 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9cvn\" (UniqueName: \"kubernetes.io/projected/0474bb77-105a-4009-9f7e-2e19a4068fc9-kube-api-access-c9cvn\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:57:23.719501 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:23.719500 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0474bb77-105a-4009-9f7e-2e19a4068fc9-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:57:23.719714 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:23.719511 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0474bb77-105a-4009-9f7e-2e19a4068fc9-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:57:23.719714 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:23.719520 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0474bb77-105a-4009-9f7e-2e19a4068fc9-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:57:24.352940 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:24.352905 5196 generic.go:356] "Generic (PLEG): container finished" podID="0474bb77-105a-4009-9f7e-2e19a4068fc9" containerID="af6e3c7027e1ea8724e0f847c1964538b3c227ca425714dd6c0791dee7bd29f3" exitCode=0 Nov 03 22:57:24.353148 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:24.352966 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" Nov 03 22:57:24.353148 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:24.352985 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" event={"ID":"0474bb77-105a-4009-9f7e-2e19a4068fc9","Type":"ContainerDied","Data":"af6e3c7027e1ea8724e0f847c1964538b3c227ca425714dd6c0791dee7bd29f3"} Nov 03 22:57:24.353148 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:24.353059 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk" event={"ID":"0474bb77-105a-4009-9f7e-2e19a4068fc9","Type":"ContainerDied","Data":"6ed1c7a54f2b912f3fa94f4ce7e3a697aac329fd4bd24f0055503366d8313b95"} Nov 03 22:57:24.353148 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:24.353092 5196 scope.go:122] "RemoveContainer" containerID="af6e3c7027e1ea8724e0f847c1964538b3c227ca425714dd6c0791dee7bd29f3" Nov 03 22:57:24.363648 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:24.363626 5196 scope.go:122] "RemoveContainer" containerID="fb80c20770872da17ebaa7493076c9373630cbd3d9aa70d087b5070180aafdb9" Nov 03 22:57:24.373414 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:24.373396 5196 scope.go:122] "RemoveContainer" containerID="af6e3c7027e1ea8724e0f847c1964538b3c227ca425714dd6c0791dee7bd29f3" Nov 03 22:57:24.373677 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:57:24.373660 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af6e3c7027e1ea8724e0f847c1964538b3c227ca425714dd6c0791dee7bd29f3\": container with ID starting with af6e3c7027e1ea8724e0f847c1964538b3c227ca425714dd6c0791dee7bd29f3 not found: ID does not exist" containerID="af6e3c7027e1ea8724e0f847c1964538b3c227ca425714dd6c0791dee7bd29f3" Nov 03 22:57:24.373757 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:24.373692 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af6e3c7027e1ea8724e0f847c1964538b3c227ca425714dd6c0791dee7bd29f3"} err="failed to get container status \"af6e3c7027e1ea8724e0f847c1964538b3c227ca425714dd6c0791dee7bd29f3\": rpc error: code = NotFound desc = could not find container \"af6e3c7027e1ea8724e0f847c1964538b3c227ca425714dd6c0791dee7bd29f3\": container with ID starting with af6e3c7027e1ea8724e0f847c1964538b3c227ca425714dd6c0791dee7bd29f3 not found: ID does not exist" Nov 03 22:57:24.373757 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:24.373715 5196 scope.go:122] "RemoveContainer" containerID="fb80c20770872da17ebaa7493076c9373630cbd3d9aa70d087b5070180aafdb9" Nov 03 22:57:24.374018 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:57:24.374000 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb80c20770872da17ebaa7493076c9373630cbd3d9aa70d087b5070180aafdb9\": container with ID starting with fb80c20770872da17ebaa7493076c9373630cbd3d9aa70d087b5070180aafdb9 not found: ID does not exist" containerID="fb80c20770872da17ebaa7493076c9373630cbd3d9aa70d087b5070180aafdb9" Nov 03 22:57:24.374070 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:24.374025 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb80c20770872da17ebaa7493076c9373630cbd3d9aa70d087b5070180aafdb9"} err="failed to get container status \"fb80c20770872da17ebaa7493076c9373630cbd3d9aa70d087b5070180aafdb9\": rpc error: code = NotFound desc = could not find container \"fb80c20770872da17ebaa7493076c9373630cbd3d9aa70d087b5070180aafdb9\": container with ID starting with fb80c20770872da17ebaa7493076c9373630cbd3d9aa70d087b5070180aafdb9 not found: ID does not exist" Nov 03 22:57:24.572900 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:24.572844 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk"] Nov 03 22:57:24.623799 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:24.623719 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-26444f-openshift-default-695897496f-mcdxk"] Nov 03 22:57:24.815181 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:24.815146 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0474bb77-105a-4009-9f7e-2e19a4068fc9" path="/var/lib/kubelet/pods/0474bb77-105a-4009-9f7e-2e19a4068fc9/volumes" Nov 03 22:57:25.733225 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:25.733199 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:25.886817 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:25.886787 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:25.919153 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:57:25.919121 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:57:27.504093 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:27.504063 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:29.166973 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:29.166940 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:29.346733 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:29.346646 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:29.516953 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:29.516830 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:29.748773 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:29.748742 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:32.633490 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:32.633452 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:36.890242 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:36.890196 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn"] Nov 03 22:57:36.890798 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:36.890442 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn" podUID="80c67970-a697-4de5-bc1e-d925bc67d7f5" containerName="maas-api" containerID="cri-o://b74a00e279182b1693a5f5720705d2e86d6154381590565c3dbcbcf308e8e3e5" gracePeriod=30 Nov 03 22:57:36.992713 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:36.992679 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:37.217380 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:37.217350 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn" Nov 03 22:57:37.224669 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:37.224644 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80c67970-a697-4de5-bc1e-d925bc67d7f5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/80c67970-a697-4de5-bc1e-d925bc67d7f5-metrics-certs\") pod \"80c67970-a697-4de5-bc1e-d925bc67d7f5\" (UID: \"80c67970-a697-4de5-bc1e-d925bc67d7f5\") " Nov 03 22:57:37.224820 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:37.224789 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80c67970-a697-4de5-bc1e-d925bc67d7f5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80c67970-a697-4de5-bc1e-d925bc67d7f5-openshift-service-ca\") pod \"80c67970-a697-4de5-bc1e-d925bc67d7f5\" (UID: \"80c67970-a697-4de5-bc1e-d925bc67d7f5\") " Nov 03 22:57:37.224938 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:37.224849 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80c67970-a697-4de5-bc1e-d925bc67d7f5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80c67970-a697-4de5-bc1e-d925bc67d7f5-maas-api-tls\") pod \"80c67970-a697-4de5-bc1e-d925bc67d7f5\" (UID: \"80c67970-a697-4de5-bc1e-d925bc67d7f5\") " Nov 03 22:57:37.225004 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:37.224938 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80c67970-a697-4de5-bc1e-d925bc67d7f5-kube-api-access-dmmxl\" (UniqueName: \"kubernetes.io/projected/80c67970-a697-4de5-bc1e-d925bc67d7f5-kube-api-access-dmmxl\") pod \"80c67970-a697-4de5-bc1e-d925bc67d7f5\" (UID: \"80c67970-a697-4de5-bc1e-d925bc67d7f5\") " Nov 03 22:57:37.225426 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:37.225217 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80c67970-a697-4de5-bc1e-d925bc67d7f5-openshift-service-ca" pod "80c67970-a697-4de5-bc1e-d925bc67d7f5" (UID: "80c67970-a697-4de5-bc1e-d925bc67d7f5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:57:37.225426 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:37.225406 5196 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80c67970-a697-4de5-bc1e-d925bc67d7f5-openshift-service-ca\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:57:37.227326 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:37.227302 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80c67970-a697-4de5-bc1e-d925bc67d7f5-metrics-certs" pod "80c67970-a697-4de5-bc1e-d925bc67d7f5" (UID: "80c67970-a697-4de5-bc1e-d925bc67d7f5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:57:37.227473 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:37.227456 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80c67970-a697-4de5-bc1e-d925bc67d7f5-maas-api-tls" pod "80c67970-a697-4de5-bc1e-d925bc67d7f5" (UID: "80c67970-a697-4de5-bc1e-d925bc67d7f5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:57:37.227541 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:37.227508 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80c67970-a697-4de5-bc1e-d925bc67d7f5-kube-api-access-dmmxl" pod "80c67970-a697-4de5-bc1e-d925bc67d7f5" (UID: "80c67970-a697-4de5-bc1e-d925bc67d7f5"). InnerVolumeSpecName "kube-api-access-dmmxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:57:37.336944 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:37.334909 5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80c67970-a697-4de5-bc1e-d925bc67d7f5-maas-api-tls\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:57:37.336944 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:37.334944 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmmxl\" (UniqueName: \"kubernetes.io/projected/80c67970-a697-4de5-bc1e-d925bc67d7f5-kube-api-access-dmmxl\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:57:37.336944 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:37.334961 5196 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80c67970-a697-4de5-bc1e-d925bc67d7f5-metrics-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:57:37.445101 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:37.445007 5196 generic.go:356] "Generic (PLEG): container finished" podID="80c67970-a697-4de5-bc1e-d925bc67d7f5" containerID="b74a00e279182b1693a5f5720705d2e86d6154381590565c3dbcbcf308e8e3e5" exitCode=0 Nov 03 22:57:37.445101 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:37.445061 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn" Nov 03 22:57:37.445101 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:37.445060 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn" event={"ID":"80c67970-a697-4de5-bc1e-d925bc67d7f5","Type":"ContainerDied","Data":"b74a00e279182b1693a5f5720705d2e86d6154381590565c3dbcbcf308e8e3e5"} Nov 03 22:57:37.445393 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:37.445122 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn" event={"ID":"80c67970-a697-4de5-bc1e-d925bc67d7f5","Type":"ContainerDied","Data":"04080e76f12e52b594433c93b2799e2fe247e62aaffb349ccc47662ae403c546"} Nov 03 22:57:37.445393 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:37.445145 5196 scope.go:122] "RemoveContainer" containerID="b74a00e279182b1693a5f5720705d2e86d6154381590565c3dbcbcf308e8e3e5" Nov 03 22:57:37.457235 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:37.457216 5196 scope.go:122] "RemoveContainer" containerID="b74a00e279182b1693a5f5720705d2e86d6154381590565c3dbcbcf308e8e3e5" Nov 03 22:57:37.457492 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:57:37.457472 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b74a00e279182b1693a5f5720705d2e86d6154381590565c3dbcbcf308e8e3e5\": container with ID starting with b74a00e279182b1693a5f5720705d2e86d6154381590565c3dbcbcf308e8e3e5 not found: ID does not exist" containerID="b74a00e279182b1693a5f5720705d2e86d6154381590565c3dbcbcf308e8e3e5" Nov 03 22:57:37.457588 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:37.457502 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b74a00e279182b1693a5f5720705d2e86d6154381590565c3dbcbcf308e8e3e5"} err="failed to get container status \"b74a00e279182b1693a5f5720705d2e86d6154381590565c3dbcbcf308e8e3e5\": rpc error: code = NotFound desc = could not find container \"b74a00e279182b1693a5f5720705d2e86d6154381590565c3dbcbcf308e8e3e5\": container with ID starting with b74a00e279182b1693a5f5720705d2e86d6154381590565c3dbcbcf308e8e3e5 not found: ID does not exist" Nov 03 22:57:37.614345 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:37.614282 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn"] Nov 03 22:57:37.618502 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:37.618473 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-de5715-86b65c979c-dzvkn"] Nov 03 22:57:38.521325 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:38.521294 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:38.789530 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:38.789442 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:38.815538 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:38.815509 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80c67970-a697-4de5-bc1e-d925bc67d7f5" path="/var/lib/kubelet/pods/80c67970-a697-4de5-bc1e-d925bc67d7f5/volumes" Nov 03 22:57:38.936324 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:38.936297 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:39.422766 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:39.422735 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:42.549681 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:42.549634 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:42.690385 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:42.690354 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:43.834012 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:43.833978 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:45.645085 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:45.645052 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:45.789724 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:45.789696 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:45.959554 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:45.959456 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:48.744111 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:48.744068 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:53.868819 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:53.868790 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:55.921895 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:57:55.921826 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:57:59.693006 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:59.692975 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:57:59.895184 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:57:59.895148 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:00.038680 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:00.038591 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:00.300164 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:00.300081 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:00.942994 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:00.942960 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:04.979169 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:04.979138 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:05.151677 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:05.151645 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:05.294808 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:05.294730 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:07.590189 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:07.590159 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:07.738032 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:07.738003 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:07.915591 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:07.915511 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:07.918521 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:07.918498 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2"] Nov 03 22:58:07.918733 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:07.918711 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" podUID="75a0d61a-c8e8-4925-941b-d729b4b21114" containerName="istio-proxy" containerID="cri-o://fef06f8ebf273399f4651503c28956673497a315df6bec21d2be11b4e043f3ec" gracePeriod=30 Nov 03 22:58:08.070225 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.070188 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v"] Nov 03 22:58:08.070704 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.070690 5196 cpu_manager.go:414] "containerMap: removing container" podUID="80c67970-a697-4de5-bc1e-d925bc67d7f5" containerName="maas-api" Nov 03 22:58:08.070748 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.070706 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80c67970-a697-4de5-bc1e-d925bc67d7f5" containerName="maas-api" Nov 03 22:58:08.070748 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.070718 5196 cpu_manager.go:414] "containerMap: removing container" podUID="f0ca4784-6fe2-4001-ac15-25675fc41a02" containerName="istio-proxy" Nov 03 22:58:08.070748 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.070723 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0ca4784-6fe2-4001-ac15-25675fc41a02" containerName="istio-proxy" Nov 03 22:58:08.070748 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.070745 5196 cpu_manager.go:414] "containerMap: removing container" podUID="0474bb77-105a-4009-9f7e-2e19a4068fc9" containerName="istio-proxy" Nov 03 22:58:08.070939 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.070750 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0474bb77-105a-4009-9f7e-2e19a4068fc9" containerName="istio-proxy" Nov 03 22:58:08.070939 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.070762 5196 cpu_manager.go:414] "containerMap: removing container" podUID="25d699f6-aaae-480e-9efb-45afb8297cce" containerName="istio-proxy" Nov 03 22:58:08.070939 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.070767 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25d699f6-aaae-480e-9efb-45afb8297cce" containerName="istio-proxy" Nov 03 22:58:08.070939 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.070776 5196 cpu_manager.go:414] "containerMap: removing container" podUID="0474bb77-105a-4009-9f7e-2e19a4068fc9" containerName="istio-proxy" Nov 03 22:58:08.070939 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.070782 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0474bb77-105a-4009-9f7e-2e19a4068fc9" containerName="istio-proxy" Nov 03 22:58:08.070939 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.070791 5196 cpu_manager.go:414] "containerMap: removing container" podUID="05bd48b9-ff9d-4511-9c48-9878b3ae40b3" containerName="istio-proxy" Nov 03 22:58:08.070939 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.070796 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05bd48b9-ff9d-4511-9c48-9878b3ae40b3" containerName="istio-proxy" Nov 03 22:58:08.070939 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.070804 5196 cpu_manager.go:414] "containerMap: removing container" podUID="bbe0a3aa-0540-40df-ba43-9196bf53b562" containerName="maas-api" Nov 03 22:58:08.070939 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.070808 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbe0a3aa-0540-40df-ba43-9196bf53b562" containerName="maas-api" Nov 03 22:58:08.070939 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.070818 5196 cpu_manager.go:414] "containerMap: removing container" podUID="ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4" containerName="maas-api" Nov 03 22:58:08.070939 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.070823 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4" containerName="maas-api" Nov 03 22:58:08.070939 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.070839 5196 cpu_manager.go:414] "containerMap: removing container" podUID="734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0" containerName="istio-proxy" Nov 03 22:58:08.070939 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.070843 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0" containerName="istio-proxy" Nov 03 22:58:08.070939 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.070939 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0ca4784-6fe2-4001-ac15-25675fc41a02" containerName="istio-proxy" Nov 03 22:58:08.071367 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.070948 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="05bd48b9-ff9d-4511-9c48-9878b3ae40b3" containerName="istio-proxy" Nov 03 22:58:08.071367 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.070954 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbe0a3aa-0540-40df-ba43-9196bf53b562" containerName="maas-api" Nov 03 22:58:08.071367 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.070966 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="734b0dd8-e9c1-4e23-b58d-a5dbf1ee5db0" containerName="istio-proxy" Nov 03 22:58:08.071367 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.070975 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="0474bb77-105a-4009-9f7e-2e19a4068fc9" containerName="istio-proxy" Nov 03 22:58:08.071367 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.070984 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="25d699f6-aaae-480e-9efb-45afb8297cce" containerName="istio-proxy" Nov 03 22:58:08.071367 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.070994 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee80b8b6-4f98-41da-b3d0-9f3c9121c0b4" containerName="maas-api" Nov 03 22:58:08.071367 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.071003 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="80c67970-a697-4de5-bc1e-d925bc67d7f5" containerName="maas-api" Nov 03 22:58:08.075583 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.075563 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 22:58:08.132116 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.132086 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-924fba-openshift-default-dockercfg-5t6h9\"" Nov 03 22:58:08.140927 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.140893 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-istio-podinfo\") pod \"e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v\" (UID: \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\") " pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 22:58:08.141180 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.141160 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-workload-certs\") pod \"e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v\" (UID: \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\") " pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 22:58:08.141340 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.141325 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-istiod-ca-cert\") pod \"e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v\" (UID: \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\") " pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 22:58:08.141500 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.141483 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nct7k\" (UniqueName: \"kubernetes.io/projected/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-kube-api-access-nct7k\") pod \"e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v\" (UID: \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\") " pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 22:58:08.141623 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.141608 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-credential-socket\") pod \"e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v\" (UID: \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\") " pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 22:58:08.141726 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.141711 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-istio-token\") pod \"e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v\" (UID: \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\") " pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 22:58:08.141843 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.141828 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-istio-envoy\") pod \"e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v\" (UID: \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\") " pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 22:58:08.142015 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.141998 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-istio-data\") pod \"e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v\" (UID: \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\") " pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 22:58:08.142165 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.142147 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-workload-socket\") pod \"e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v\" (UID: \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\") " pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 22:58:08.143746 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.143721 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v"] Nov 03 22:58:08.171571 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.171488 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:08.244110 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.244072 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-workload-socket\") pod \"e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v\" (UID: \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\") " pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 22:58:08.244327 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.244185 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-istio-podinfo\") pod \"e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v\" (UID: \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\") " pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 22:58:08.244327 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.244233 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-workload-certs\") pod \"e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v\" (UID: \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\") " pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 22:58:08.244327 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.244296 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-istiod-ca-cert\") pod \"e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v\" (UID: \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\") " pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 22:58:08.244513 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.244376 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nct7k\" (UniqueName: \"kubernetes.io/projected/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-kube-api-access-nct7k\") pod \"e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v\" (UID: \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\") " pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 22:58:08.244513 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.244418 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-credential-socket\") pod \"e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v\" (UID: \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\") " pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 22:58:08.244513 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.244445 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-istio-token\") pod \"e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v\" (UID: \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\") " pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 22:58:08.244513 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.244483 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-istio-envoy\") pod \"e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v\" (UID: \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\") " pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 22:58:08.244727 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.244527 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-istio-data\") pod \"e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v\" (UID: \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\") " pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 22:58:08.244727 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.244612 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-workload-socket\") pod \"e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v\" (UID: \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\") " pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 22:58:08.244984 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.244959 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-istio-data\") pod \"e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v\" (UID: \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\") " pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 22:58:08.245247 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.245225 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-workload-certs\") pod \"e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v\" (UID: \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\") " pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 22:58:08.245490 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.245468 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-credential-socket\") pod \"e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v\" (UID: \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\") " pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 22:58:08.245850 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.245819 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-istiod-ca-cert\") pod \"e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v\" (UID: \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\") " pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 22:58:08.254915 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.250659 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-istio-envoy\") pod \"e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v\" (UID: \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\") " pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 22:58:08.254915 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.252078 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-istio-podinfo\") pod \"e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v\" (UID: \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\") " pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 22:58:08.346899 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.330716 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-istio-token\") pod \"e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v\" (UID: \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\") " pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 22:58:08.346899 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.331794 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nct7k\" (UniqueName: \"kubernetes.io/projected/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-kube-api-access-nct7k\") pod \"e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v\" (UID: \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\") " pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 22:58:08.389327 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.389294 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 22:58:08.447551 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.447473 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:08.770622 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.770536 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:08.824183 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.824154 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v"] Nov 03 22:58:08.825818 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:58:08.825785 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcef4b051_45b0_4db8_8ab5_afcfe3b9149d.slice/crio-874af0e53973cd7adeb2bd912e133318560c9b0e8e661e9904e21053410981cb WatchSource:0}: Error finding container 874af0e53973cd7adeb2bd912e133318560c9b0e8e661e9904e21053410981cb: Status 404 returned error can't find the container with id 874af0e53973cd7adeb2bd912e133318560c9b0e8e661e9904e21053410981cb Nov 03 22:58:08.829384 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.829333 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:58:08.829495 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.829422 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:58:08.829495 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:08.829468 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:58:09.651538 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:09.651495 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" event={"ID":"cef4b051-45b0-4db8-8ab5-afcfe3b9149d","Type":"ContainerStarted","Data":"154f65d83db979208f117a3eda40f081f98a9631b46bee2a47df32867f5c6966"} Nov 03 22:58:09.651701 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:09.651563 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" event={"ID":"cef4b051-45b0-4db8-8ab5-afcfe3b9149d","Type":"ContainerStarted","Data":"874af0e53973cd7adeb2bd912e133318560c9b0e8e661e9904e21053410981cb"} Nov 03 22:58:09.773710 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:09.773659 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" podStartSLOduration=1.773643904 podStartE2EDuration="1.773643904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:58:09.772465082 +0000 UTC m=+2119.082024390" watchObservedRunningTime="2026-11-03 22:58:09.773643904 +0000 UTC m=+2119.083203197" Nov 03 22:58:10.072421 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:10.072324 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:10.351140 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:10.351109 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:10.392368 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:10.391142 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 22:58:10.416419 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:10.416374 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 22:58:10.658739 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:10.658643 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 22:58:10.659811 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:10.659788 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 22:58:11.077778 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:11.077687 5196 patch_prober.go:29] interesting pod/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:58:11.077778 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:11.077747 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" podUID="75a0d61a-c8e8-4925-941b-d729b4b21114" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:58:12.627034 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:12.627004 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:12.846468 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:12.846440 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:13.163878 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.163830 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:13.191104 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.191036 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:58:13.314995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.314966 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a0d61a-c8e8-4925-941b-d729b4b21114-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75a0d61a-c8e8-4925-941b-d729b4b21114-istio-envoy\") pod \"75a0d61a-c8e8-4925-941b-d729b4b21114\" (UID: \"75a0d61a-c8e8-4925-941b-d729b4b21114\") " Nov 03 22:58:13.315246 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.315014 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75a0d61a-c8e8-4925-941b-d729b4b21114-kube-api-access-tv6b2\" (UniqueName: \"kubernetes.io/projected/75a0d61a-c8e8-4925-941b-d729b4b21114-kube-api-access-tv6b2\") pod \"75a0d61a-c8e8-4925-941b-d729b4b21114\" (UID: \"75a0d61a-c8e8-4925-941b-d729b4b21114\") " Nov 03 22:58:13.315246 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.315071 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a0d61a-c8e8-4925-941b-d729b4b21114-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75a0d61a-c8e8-4925-941b-d729b4b21114-workload-certs\") pod \"75a0d61a-c8e8-4925-941b-d729b4b21114\" (UID: \"75a0d61a-c8e8-4925-941b-d729b4b21114\") " Nov 03 22:58:13.315246 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.315170 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a0d61a-c8e8-4925-941b-d729b4b21114-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75a0d61a-c8e8-4925-941b-d729b4b21114-istio-data\") pod \"75a0d61a-c8e8-4925-941b-d729b4b21114\" (UID: \"75a0d61a-c8e8-4925-941b-d729b4b21114\") " Nov 03 22:58:13.315446 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.315291 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75a0d61a-c8e8-4925-941b-d729b4b21114-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75a0d61a-c8e8-4925-941b-d729b4b21114-istio-podinfo\") pod \"75a0d61a-c8e8-4925-941b-d729b4b21114\" (UID: \"75a0d61a-c8e8-4925-941b-d729b4b21114\") " Nov 03 22:58:13.315446 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.315342 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75a0d61a-c8e8-4925-941b-d729b4b21114-istio-token\" (UniqueName: \"kubernetes.io/projected/75a0d61a-c8e8-4925-941b-d729b4b21114-istio-token\") pod \"75a0d61a-c8e8-4925-941b-d729b4b21114\" (UID: \"75a0d61a-c8e8-4925-941b-d729b4b21114\") " Nov 03 22:58:13.315446 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.315409 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a0d61a-c8e8-4925-941b-d729b4b21114-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a0d61a-c8e8-4925-941b-d729b4b21114-credential-socket\") pod \"75a0d61a-c8e8-4925-941b-d729b4b21114\" (UID: \"75a0d61a-c8e8-4925-941b-d729b4b21114\") " Nov 03 22:58:13.315621 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.315449 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a0d61a-c8e8-4925-941b-d729b4b21114-workload-certs" pod "75a0d61a-c8e8-4925-941b-d729b4b21114" (UID: "75a0d61a-c8e8-4925-941b-d729b4b21114"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:13.315621 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.315477 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75a0d61a-c8e8-4925-941b-d729b4b21114-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75a0d61a-c8e8-4925-941b-d729b4b21114-istiod-ca-cert\") pod \"75a0d61a-c8e8-4925-941b-d729b4b21114\" (UID: \"75a0d61a-c8e8-4925-941b-d729b4b21114\") " Nov 03 22:58:13.315621 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.315524 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a0d61a-c8e8-4925-941b-d729b4b21114-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a0d61a-c8e8-4925-941b-d729b4b21114-workload-socket\") pod \"75a0d61a-c8e8-4925-941b-d729b4b21114\" (UID: \"75a0d61a-c8e8-4925-941b-d729b4b21114\") " Nov 03 22:58:13.315621 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.315531 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a0d61a-c8e8-4925-941b-d729b4b21114-istio-data" pod "75a0d61a-c8e8-4925-941b-d729b4b21114" (UID: "75a0d61a-c8e8-4925-941b-d729b4b21114"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:13.315843 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.315755 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a0d61a-c8e8-4925-941b-d729b4b21114-credential-socket" pod "75a0d61a-c8e8-4925-941b-d729b4b21114" (UID: "75a0d61a-c8e8-4925-941b-d729b4b21114"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:13.316144 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.316075 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a0d61a-c8e8-4925-941b-d729b4b21114-workload-socket" pod "75a0d61a-c8e8-4925-941b-d729b4b21114" (UID: "75a0d61a-c8e8-4925-941b-d729b4b21114"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:13.316144 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.316115 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75a0d61a-c8e8-4925-941b-d729b4b21114-istiod-ca-cert" pod "75a0d61a-c8e8-4925-941b-d729b4b21114" (UID: "75a0d61a-c8e8-4925-941b-d729b4b21114"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:58:13.316310 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.316271 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75a0d61a-c8e8-4925-941b-d729b4b21114-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:13.316310 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.316291 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a0d61a-c8e8-4925-941b-d729b4b21114-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:13.316310 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.316305 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75a0d61a-c8e8-4925-941b-d729b4b21114-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:13.316465 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.316319 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75a0d61a-c8e8-4925-941b-d729b4b21114-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:13.317887 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.317829 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a0d61a-c8e8-4925-941b-d729b4b21114-kube-api-access-tv6b2" pod "75a0d61a-c8e8-4925-941b-d729b4b21114" (UID: "75a0d61a-c8e8-4925-941b-d729b4b21114"). InnerVolumeSpecName "kube-api-access-tv6b2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:13.318117 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.318093 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75a0d61a-c8e8-4925-941b-d729b4b21114-istio-podinfo" pod "75a0d61a-c8e8-4925-941b-d729b4b21114" (UID: "75a0d61a-c8e8-4925-941b-d729b4b21114"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:58:13.318200 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.318159 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a0d61a-c8e8-4925-941b-d729b4b21114-istio-envoy" pod "75a0d61a-c8e8-4925-941b-d729b4b21114" (UID: "75a0d61a-c8e8-4925-941b-d729b4b21114"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:13.318278 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.318261 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a0d61a-c8e8-4925-941b-d729b4b21114-istio-token" pod "75a0d61a-c8e8-4925-941b-d729b4b21114" (UID: "75a0d61a-c8e8-4925-941b-d729b4b21114"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:13.417375 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.417335 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv6b2\" (UniqueName: \"kubernetes.io/projected/75a0d61a-c8e8-4925-941b-d729b4b21114-kube-api-access-tv6b2\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:13.417375 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.417375 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75a0d61a-c8e8-4925-941b-d729b4b21114-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:13.417580 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.417391 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75a0d61a-c8e8-4925-941b-d729b4b21114-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:13.417580 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.417405 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75a0d61a-c8e8-4925-941b-d729b4b21114-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:13.417580 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.417419 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75a0d61a-c8e8-4925-941b-d729b4b21114-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:13.679529 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.679490 5196 generic.go:356] "Generic (PLEG): container finished" podID="75a0d61a-c8e8-4925-941b-d729b4b21114" containerID="fef06f8ebf273399f4651503c28956673497a315df6bec21d2be11b4e043f3ec" exitCode=0 Nov 03 22:58:13.679995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.679545 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" Nov 03 22:58:13.679995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.679569 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" event={"ID":"75a0d61a-c8e8-4925-941b-d729b4b21114","Type":"ContainerDied","Data":"fef06f8ebf273399f4651503c28956673497a315df6bec21d2be11b4e043f3ec"} Nov 03 22:58:13.679995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.679628 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2" event={"ID":"75a0d61a-c8e8-4925-941b-d729b4b21114","Type":"ContainerDied","Data":"c3b449549c5a8f0f844df63eb66f9fcb4e4432e000caffdc87f4f9b01fa38103"} Nov 03 22:58:13.679995 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.679653 5196 scope.go:122] "RemoveContainer" containerID="fef06f8ebf273399f4651503c28956673497a315df6bec21d2be11b4e043f3ec" Nov 03 22:58:13.690659 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.690641 5196 scope.go:122] "RemoveContainer" containerID="5cdece8ad478f12547a659c127254ce222eb9bd255aa13471ce7c1b26a104e84" Nov 03 22:58:13.700029 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.700011 5196 scope.go:122] "RemoveContainer" containerID="fef06f8ebf273399f4651503c28956673497a315df6bec21d2be11b4e043f3ec" Nov 03 22:58:13.700282 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:58:13.700262 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fef06f8ebf273399f4651503c28956673497a315df6bec21d2be11b4e043f3ec\": container with ID starting with fef06f8ebf273399f4651503c28956673497a315df6bec21d2be11b4e043f3ec not found: ID does not exist" containerID="fef06f8ebf273399f4651503c28956673497a315df6bec21d2be11b4e043f3ec" Nov 03 22:58:13.700361 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.700297 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fef06f8ebf273399f4651503c28956673497a315df6bec21d2be11b4e043f3ec"} err="failed to get container status \"fef06f8ebf273399f4651503c28956673497a315df6bec21d2be11b4e043f3ec\": rpc error: code = NotFound desc = could not find container \"fef06f8ebf273399f4651503c28956673497a315df6bec21d2be11b4e043f3ec\": container with ID starting with fef06f8ebf273399f4651503c28956673497a315df6bec21d2be11b4e043f3ec not found: ID does not exist" Nov 03 22:58:13.700361 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.700321 5196 scope.go:122] "RemoveContainer" containerID="5cdece8ad478f12547a659c127254ce222eb9bd255aa13471ce7c1b26a104e84" Nov 03 22:58:13.700573 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:58:13.700554 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cdece8ad478f12547a659c127254ce222eb9bd255aa13471ce7c1b26a104e84\": container with ID starting with 5cdece8ad478f12547a659c127254ce222eb9bd255aa13471ce7c1b26a104e84 not found: ID does not exist" containerID="5cdece8ad478f12547a659c127254ce222eb9bd255aa13471ce7c1b26a104e84" Nov 03 22:58:13.700612 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.700580 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cdece8ad478f12547a659c127254ce222eb9bd255aa13471ce7c1b26a104e84"} err="failed to get container status \"5cdece8ad478f12547a659c127254ce222eb9bd255aa13471ce7c1b26a104e84\": rpc error: code = NotFound desc = could not find container \"5cdece8ad478f12547a659c127254ce222eb9bd255aa13471ce7c1b26a104e84\": container with ID starting with 5cdece8ad478f12547a659c127254ce222eb9bd255aa13471ce7c1b26a104e84 not found: ID does not exist" Nov 03 22:58:13.844669 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.844629 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:13.853927 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.853904 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2"] Nov 03 22:58:13.856668 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:13.856646 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a91d2d-openshift-default-6b9b889877-szkd2"] Nov 03 22:58:14.815325 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:14.815282 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75a0d61a-c8e8-4925-941b-d729b4b21114" path="/var/lib/kubelet/pods/75a0d61a-c8e8-4925-941b-d729b4b21114/volumes" Nov 03 22:58:15.096632 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:15.096598 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn"] Nov 03 22:58:15.146766 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:15.146736 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6"] Nov 03 22:58:15.287151 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:15.287117 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:15.721445 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:15.721407 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:16.684372 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:16.684339 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:16.884999 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:16.884963 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:17.334635 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:17.334557 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:17.484608 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:17.484561 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:20.982659 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:20.982625 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:24.188195 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:24.188159 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:24.328978 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:24.328941 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:25.752301 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:25.752262 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5"] Nov 03 22:58:25.767332 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:25.754500 5196 cpu_manager.go:414] "containerMap: removing container" podUID="75a0d61a-c8e8-4925-941b-d729b4b21114" containerName="istio-proxy" Nov 03 22:58:25.767332 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:25.754521 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75a0d61a-c8e8-4925-941b-d729b4b21114" containerName="istio-proxy" Nov 03 22:58:25.767332 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:25.754542 5196 cpu_manager.go:414] "containerMap: removing container" podUID="75a0d61a-c8e8-4925-941b-d729b4b21114" containerName="istio-proxy" Nov 03 22:58:25.767332 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:25.754547 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75a0d61a-c8e8-4925-941b-d729b4b21114" containerName="istio-proxy" Nov 03 22:58:25.767332 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:25.754647 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="75a0d61a-c8e8-4925-941b-d729b4b21114" containerName="istio-proxy" Nov 03 22:58:25.767332 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:25.754663 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="0474bb77-105a-4009-9f7e-2e19a4068fc9" containerName="istio-proxy" Nov 03 22:58:25.767332 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:25.760466 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5" Nov 03 22:58:25.794507 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:25.794481 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/994606f4-8a09-4d39-8310-d25dc898c140-maas-api-tls\") pod \"maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5\" (UID: \"994606f4-8a09-4d39-8310-d25dc898c140\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5" Nov 03 22:58:25.794665 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:25.794524 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v75j4\" (UniqueName: \"kubernetes.io/projected/994606f4-8a09-4d39-8310-d25dc898c140-kube-api-access-v75j4\") pod \"maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5\" (UID: \"994606f4-8a09-4d39-8310-d25dc898c140\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5" Nov 03 22:58:25.794665 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:25.794594 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/994606f4-8a09-4d39-8310-d25dc898c140-metrics-certs\") pod \"maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5\" (UID: \"994606f4-8a09-4d39-8310-d25dc898c140\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5" Nov 03 22:58:25.794739 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:25.794661 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/994606f4-8a09-4d39-8310-d25dc898c140-openshift-service-ca\") pod \"maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5\" (UID: \"994606f4-8a09-4d39-8310-d25dc898c140\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5" Nov 03 22:58:25.814035 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:25.814003 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5"] Nov 03 22:58:25.860663 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:25.860629 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-efd39a-serving-cert\"" Nov 03 22:58:25.896041 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:25.896010 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/994606f4-8a09-4d39-8310-d25dc898c140-maas-api-tls\") pod \"maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5\" (UID: \"994606f4-8a09-4d39-8310-d25dc898c140\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5" Nov 03 22:58:25.896198 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:25.896072 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v75j4\" (UniqueName: \"kubernetes.io/projected/994606f4-8a09-4d39-8310-d25dc898c140-kube-api-access-v75j4\") pod \"maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5\" (UID: \"994606f4-8a09-4d39-8310-d25dc898c140\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5" Nov 03 22:58:25.896198 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:25.896115 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/994606f4-8a09-4d39-8310-d25dc898c140-metrics-certs\") pod \"maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5\" (UID: \"994606f4-8a09-4d39-8310-d25dc898c140\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5" Nov 03 22:58:25.896198 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:58:25.896139 5196 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-efd39a-serving-cert: secret "maas-api-e2e-inf-a-efd39a-serving-cert" not found Nov 03 22:58:25.896369 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:58:25.896215 5196 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/994606f4-8a09-4d39-8310-d25dc898c140-maas-api-tls podName:994606f4-8a09-4d39-8310-d25dc898c140 nodeName:}" failed. No retries permitted until 2026-11-03 22:58:26.396196382 +0000 UTC m=+2135.705755672 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/994606f4-8a09-4d39-8310-d25dc898c140-maas-api-tls") pod "maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5" (UID: "994606f4-8a09-4d39-8310-d25dc898c140") : secret "maas-api-e2e-inf-a-efd39a-serving-cert" not found Nov 03 22:58:25.896369 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:25.896147 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/994606f4-8a09-4d39-8310-d25dc898c140-openshift-service-ca\") pod \"maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5\" (UID: \"994606f4-8a09-4d39-8310-d25dc898c140\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5" Nov 03 22:58:25.896788 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:25.896765 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/994606f4-8a09-4d39-8310-d25dc898c140-openshift-service-ca\") pod \"maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5\" (UID: \"994606f4-8a09-4d39-8310-d25dc898c140\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5" Nov 03 22:58:25.899124 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:25.899100 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/994606f4-8a09-4d39-8310-d25dc898c140-metrics-certs\") pod \"maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5\" (UID: \"994606f4-8a09-4d39-8310-d25dc898c140\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5" Nov 03 22:58:25.928739 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:58:25.928690 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:58:25.958721 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:25.958692 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v75j4\" (UniqueName: \"kubernetes.io/projected/994606f4-8a09-4d39-8310-d25dc898c140-kube-api-access-v75j4\") pod \"maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5\" (UID: \"994606f4-8a09-4d39-8310-d25dc898c140\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5" Nov 03 22:58:25.961907 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:25.961879 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:26.120223 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:26.120192 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:26.264846 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:26.264819 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:26.403126 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:26.403045 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/994606f4-8a09-4d39-8310-d25dc898c140-maas-api-tls\") pod \"maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5\" (UID: \"994606f4-8a09-4d39-8310-d25dc898c140\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5" Nov 03 22:58:26.405497 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:26.405475 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/994606f4-8a09-4d39-8310-d25dc898c140-maas-api-tls\") pod \"maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5\" (UID: \"994606f4-8a09-4d39-8310-d25dc898c140\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5" Nov 03 22:58:26.686306 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:26.686227 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5" Nov 03 22:58:27.071144 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:27.071114 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5"] Nov 03 22:58:27.071699 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:58:27.071678 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod994606f4_8a09_4d39_8310_d25dc898c140.slice/crio-7cc1cb87cf0827a7c2d1ec508620404a72d62939328289d333e4914a336f9126 WatchSource:0}: Error finding container 7cc1cb87cf0827a7c2d1ec508620404a72d62939328289d333e4914a336f9126: Status 404 returned error can't find the container with id 7cc1cb87cf0827a7c2d1ec508620404a72d62939328289d333e4914a336f9126 Nov 03 22:58:27.799399 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:27.799363 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5" event={"ID":"994606f4-8a09-4d39-8310-d25dc898c140","Type":"ContainerStarted","Data":"7cc1cb87cf0827a7c2d1ec508620404a72d62939328289d333e4914a336f9126"} Nov 03 22:58:28.861427 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:28.861343 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5" event={"ID":"994606f4-8a09-4d39-8310-d25dc898c140","Type":"ContainerStarted","Data":"cefea8cf76843c208e38dadff2c819a38c1c0e045d95399f356f7106eebf7b9e"} Nov 03 22:58:28.862540 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:28.862496 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5" Nov 03 22:58:28.985399 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:28.985341 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5" podStartSLOduration=3.214860757 podStartE2EDuration="3.985326838s" totalImagesPullingTime="770.466081ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:58:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:58:28.981453245 +0000 UTC m=+2138.291012538" watchObservedRunningTime="2026-11-03 22:58:28.985326838 +0000 UTC m=+2138.294886130" Nov 03 22:58:30.119599 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.119553 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" podUID="9f2e3f52-0664-434f-919b-32736cbf0602" containerName="main" containerID="cri-o://14daded1822daa9e551f636f3236518944f047f327eca57cc49833e29b38b5e0" gracePeriod=45 Nov 03 22:58:30.166953 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.166900 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" podUID="82749c55-0fca-4a14-b6ba-7df122a6a407" containerName="main" containerID="cri-o://94d5702d4f2a5a0f8105bbcea39f754e07b84bbea99c7c24ad25c92f879b06d0" gracePeriod=45 Nov 03 22:58:30.306504 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.306174 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" podUID="9f2e3f52-0664-434f-919b-32736cbf0602" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": dial tcp 10.133.0.58:8000: connect: connection refused" Nov 03 22:58:30.461591 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.461569 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" Nov 03 22:58:30.464007 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.463987 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" Nov 03 22:58:30.563897 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.563871 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f2e3f52-0664-434f-919b-32736cbf0602-home\" (UniqueName: \"kubernetes.io/empty-dir/9f2e3f52-0664-434f-919b-32736cbf0602-home\") pod \"9f2e3f52-0664-434f-919b-32736cbf0602\" (UID: \"9f2e3f52-0664-434f-919b-32736cbf0602\") " Nov 03 22:58:30.564057 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.563913 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f2e3f52-0664-434f-919b-32736cbf0602-kube-api-access-xcpwb\" (UniqueName: \"kubernetes.io/projected/9f2e3f52-0664-434f-919b-32736cbf0602-kube-api-access-xcpwb\") pod \"9f2e3f52-0664-434f-919b-32736cbf0602\" (UID: \"9f2e3f52-0664-434f-919b-32736cbf0602\") " Nov 03 22:58:30.564057 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.564003 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f2e3f52-0664-434f-919b-32736cbf0602-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f2e3f52-0664-434f-919b-32736cbf0602-tmp-dir\") pod \"9f2e3f52-0664-434f-919b-32736cbf0602\" (UID: \"9f2e3f52-0664-434f-919b-32736cbf0602\") " Nov 03 22:58:30.564057 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.564035 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82749c55-0fca-4a14-b6ba-7df122a6a407-kube-api-access-5z5fq\" (UniqueName: \"kubernetes.io/projected/82749c55-0fca-4a14-b6ba-7df122a6a407-kube-api-access-5z5fq\") pod \"82749c55-0fca-4a14-b6ba-7df122a6a407\" (UID: \"82749c55-0fca-4a14-b6ba-7df122a6a407\") " Nov 03 22:58:30.564234 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.564060 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82749c55-0fca-4a14-b6ba-7df122a6a407-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82749c55-0fca-4a14-b6ba-7df122a6a407-model-cache\") pod \"82749c55-0fca-4a14-b6ba-7df122a6a407\" (UID: \"82749c55-0fca-4a14-b6ba-7df122a6a407\") " Nov 03 22:58:30.564234 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.564087 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2e3f52-0664-434f-919b-32736cbf0602-home" pod "9f2e3f52-0664-434f-919b-32736cbf0602" (UID: "9f2e3f52-0664-434f-919b-32736cbf0602"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:30.564234 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.564112 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f2e3f52-0664-434f-919b-32736cbf0602-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f2e3f52-0664-434f-919b-32736cbf0602-model-cache\") pod \"9f2e3f52-0664-434f-919b-32736cbf0602\" (UID: \"9f2e3f52-0664-434f-919b-32736cbf0602\") " Nov 03 22:58:30.564234 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.564180 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82749c55-0fca-4a14-b6ba-7df122a6a407-tls-certs\" (UniqueName: \"kubernetes.io/secret/82749c55-0fca-4a14-b6ba-7df122a6a407-tls-certs\") pod \"82749c55-0fca-4a14-b6ba-7df122a6a407\" (UID: \"82749c55-0fca-4a14-b6ba-7df122a6a407\") " Nov 03 22:58:30.564234 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.564225 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82749c55-0fca-4a14-b6ba-7df122a6a407-dshm\" (UniqueName: \"kubernetes.io/empty-dir/82749c55-0fca-4a14-b6ba-7df122a6a407-dshm\") pod \"82749c55-0fca-4a14-b6ba-7df122a6a407\" (UID: \"82749c55-0fca-4a14-b6ba-7df122a6a407\") " Nov 03 22:58:30.564497 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.564256 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f2e3f52-0664-434f-919b-32736cbf0602-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f2e3f52-0664-434f-919b-32736cbf0602-dshm\") pod \"9f2e3f52-0664-434f-919b-32736cbf0602\" (UID: \"9f2e3f52-0664-434f-919b-32736cbf0602\") " Nov 03 22:58:30.564497 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.564280 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2e3f52-0664-434f-919b-32736cbf0602-tmp-dir" pod "9f2e3f52-0664-434f-919b-32736cbf0602" (UID: "9f2e3f52-0664-434f-919b-32736cbf0602"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:30.564497 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.564287 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82749c55-0fca-4a14-b6ba-7df122a6a407-model-cache" pod "82749c55-0fca-4a14-b6ba-7df122a6a407" (UID: "82749c55-0fca-4a14-b6ba-7df122a6a407"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:30.564497 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.564388 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2e3f52-0664-434f-919b-32736cbf0602-model-cache" pod "9f2e3f52-0664-434f-919b-32736cbf0602" (UID: "9f2e3f52-0664-434f-919b-32736cbf0602"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:30.564497 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.564388 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f2e3f52-0664-434f-919b-32736cbf0602-tls-certs\" (UniqueName: \"kubernetes.io/secret/9f2e3f52-0664-434f-919b-32736cbf0602-tls-certs\") pod \"9f2e3f52-0664-434f-919b-32736cbf0602\" (UID: \"9f2e3f52-0664-434f-919b-32736cbf0602\") " Nov 03 22:58:30.564497 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.564463 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82749c55-0fca-4a14-b6ba-7df122a6a407-home\" (UniqueName: \"kubernetes.io/empty-dir/82749c55-0fca-4a14-b6ba-7df122a6a407-home\") pod \"82749c55-0fca-4a14-b6ba-7df122a6a407\" (UID: \"82749c55-0fca-4a14-b6ba-7df122a6a407\") " Nov 03 22:58:30.564981 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.564533 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82749c55-0fca-4a14-b6ba-7df122a6a407-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82749c55-0fca-4a14-b6ba-7df122a6a407-tmp-dir\") pod \"82749c55-0fca-4a14-b6ba-7df122a6a407\" (UID: \"82749c55-0fca-4a14-b6ba-7df122a6a407\") " Nov 03 22:58:30.565046 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.565008 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82749c55-0fca-4a14-b6ba-7df122a6a407-home" pod "82749c55-0fca-4a14-b6ba-7df122a6a407" (UID: "82749c55-0fca-4a14-b6ba-7df122a6a407"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:30.565256 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.565228 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82749c55-0fca-4a14-b6ba-7df122a6a407-tmp-dir" pod "82749c55-0fca-4a14-b6ba-7df122a6a407" (UID: "82749c55-0fca-4a14-b6ba-7df122a6a407"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:30.565379 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.565361 5196 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82749c55-0fca-4a14-b6ba-7df122a6a407-home\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:30.565447 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.565386 5196 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82749c55-0fca-4a14-b6ba-7df122a6a407-tmp-dir\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:30.565447 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.565400 5196 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f2e3f52-0664-434f-919b-32736cbf0602-home\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:30.565447 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.565415 5196 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f2e3f52-0664-434f-919b-32736cbf0602-tmp-dir\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:30.565447 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.565428 5196 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82749c55-0fca-4a14-b6ba-7df122a6a407-model-cache\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:30.565447 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.565441 5196 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f2e3f52-0664-434f-919b-32736cbf0602-model-cache\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:30.566949 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.566923 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f2e3f52-0664-434f-919b-32736cbf0602-tls-certs" pod "9f2e3f52-0664-434f-919b-32736cbf0602" (UID: "9f2e3f52-0664-434f-919b-32736cbf0602"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:58:30.567058 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.566964 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f2e3f52-0664-434f-919b-32736cbf0602-kube-api-access-xcpwb" pod "9f2e3f52-0664-434f-919b-32736cbf0602" (UID: "9f2e3f52-0664-434f-919b-32736cbf0602"). InnerVolumeSpecName "kube-api-access-xcpwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:30.567058 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.567019 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82749c55-0fca-4a14-b6ba-7df122a6a407-dshm" pod "82749c55-0fca-4a14-b6ba-7df122a6a407" (UID: "82749c55-0fca-4a14-b6ba-7df122a6a407"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:30.567058 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.567035 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2e3f52-0664-434f-919b-32736cbf0602-dshm" pod "9f2e3f52-0664-434f-919b-32736cbf0602" (UID: "9f2e3f52-0664-434f-919b-32736cbf0602"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:30.567165 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.567083 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82749c55-0fca-4a14-b6ba-7df122a6a407-kube-api-access-5z5fq" pod "82749c55-0fca-4a14-b6ba-7df122a6a407" (UID: "82749c55-0fca-4a14-b6ba-7df122a6a407"). InnerVolumeSpecName "kube-api-access-5z5fq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:30.567245 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.567232 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82749c55-0fca-4a14-b6ba-7df122a6a407-tls-certs" pod "82749c55-0fca-4a14-b6ba-7df122a6a407" (UID: "82749c55-0fca-4a14-b6ba-7df122a6a407"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:58:30.666566 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.666495 5196 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f2e3f52-0664-434f-919b-32736cbf0602-tls-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:30.666566 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.666521 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcpwb\" (UniqueName: \"kubernetes.io/projected/9f2e3f52-0664-434f-919b-32736cbf0602-kube-api-access-xcpwb\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:30.666566 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.666532 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z5fq\" (UniqueName: \"kubernetes.io/projected/82749c55-0fca-4a14-b6ba-7df122a6a407-kube-api-access-5z5fq\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:30.666566 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.666541 5196 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82749c55-0fca-4a14-b6ba-7df122a6a407-tls-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:30.666566 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.666551 5196 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82749c55-0fca-4a14-b6ba-7df122a6a407-dshm\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:30.666566 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.666559 5196 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f2e3f52-0664-434f-919b-32736cbf0602-dshm\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:30.878803 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.878759 5196 generic.go:356] "Generic (PLEG): container finished" podID="9f2e3f52-0664-434f-919b-32736cbf0602" containerID="14daded1822daa9e551f636f3236518944f047f327eca57cc49833e29b38b5e0" exitCode=0 Nov 03 22:58:30.879022 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.878850 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" Nov 03 22:58:30.879022 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.878928 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" event={"ID":"9f2e3f52-0664-434f-919b-32736cbf0602","Type":"ContainerDied","Data":"14daded1822daa9e551f636f3236518944f047f327eca57cc49833e29b38b5e0"} Nov 03 22:58:30.879022 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.878974 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn" event={"ID":"9f2e3f52-0664-434f-919b-32736cbf0602","Type":"ContainerDied","Data":"0fc50d0fedce4b471a31a37d16b46d4c015f22b4b128c840a94359f5c8a59a1a"} Nov 03 22:58:30.879022 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.879004 5196 scope.go:122] "RemoveContainer" containerID="14daded1822daa9e551f636f3236518944f047f327eca57cc49833e29b38b5e0" Nov 03 22:58:30.880631 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.880607 5196 generic.go:356] "Generic (PLEG): container finished" podID="82749c55-0fca-4a14-b6ba-7df122a6a407" containerID="94d5702d4f2a5a0f8105bbcea39f754e07b84bbea99c7c24ad25c92f879b06d0" exitCode=0 Nov 03 22:58:30.880756 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.880664 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" event={"ID":"82749c55-0fca-4a14-b6ba-7df122a6a407","Type":"ContainerDied","Data":"94d5702d4f2a5a0f8105bbcea39f754e07b84bbea99c7c24ad25c92f879b06d0"} Nov 03 22:58:30.880756 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.880689 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" Nov 03 22:58:30.880756 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.880701 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6" event={"ID":"82749c55-0fca-4a14-b6ba-7df122a6a407","Type":"ContainerDied","Data":"9e23e5d1ee8f209f4e07c3186fb2053e2e5296495a18e2980bc925885fe57783"} Nov 03 22:58:30.890583 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.890567 5196 scope.go:122] "RemoveContainer" containerID="14daded1822daa9e551f636f3236518944f047f327eca57cc49833e29b38b5e0" Nov 03 22:58:30.890848 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:58:30.890824 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14daded1822daa9e551f636f3236518944f047f327eca57cc49833e29b38b5e0\": container with ID starting with 14daded1822daa9e551f636f3236518944f047f327eca57cc49833e29b38b5e0 not found: ID does not exist" containerID="14daded1822daa9e551f636f3236518944f047f327eca57cc49833e29b38b5e0" Nov 03 22:58:30.890927 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.890872 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14daded1822daa9e551f636f3236518944f047f327eca57cc49833e29b38b5e0"} err="failed to get container status \"14daded1822daa9e551f636f3236518944f047f327eca57cc49833e29b38b5e0\": rpc error: code = NotFound desc = could not find container \"14daded1822daa9e551f636f3236518944f047f327eca57cc49833e29b38b5e0\": container with ID starting with 14daded1822daa9e551f636f3236518944f047f327eca57cc49833e29b38b5e0 not found: ID does not exist" Nov 03 22:58:30.890927 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.890888 5196 scope.go:122] "RemoveContainer" containerID="94d5702d4f2a5a0f8105bbcea39f754e07b84bbea99c7c24ad25c92f879b06d0" Nov 03 22:58:30.899645 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.899602 5196 scope.go:122] "RemoveContainer" containerID="94d5702d4f2a5a0f8105bbcea39f754e07b84bbea99c7c24ad25c92f879b06d0" Nov 03 22:58:30.899951 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:58:30.899931 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94d5702d4f2a5a0f8105bbcea39f754e07b84bbea99c7c24ad25c92f879b06d0\": container with ID starting with 94d5702d4f2a5a0f8105bbcea39f754e07b84bbea99c7c24ad25c92f879b06d0 not found: ID does not exist" containerID="94d5702d4f2a5a0f8105bbcea39f754e07b84bbea99c7c24ad25c92f879b06d0" Nov 03 22:58:30.900018 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:30.899960 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94d5702d4f2a5a0f8105bbcea39f754e07b84bbea99c7c24ad25c92f879b06d0"} err="failed to get container status \"94d5702d4f2a5a0f8105bbcea39f754e07b84bbea99c7c24ad25c92f879b06d0\": rpc error: code = NotFound desc = could not find container \"94d5702d4f2a5a0f8105bbcea39f754e07b84bbea99c7c24ad25c92f879b06d0\": container with ID starting with 94d5702d4f2a5a0f8105bbcea39f754e07b84bbea99c7c24ad25c92f879b06d0 not found: ID does not exist" Nov 03 22:58:31.000831 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:31.000739 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:31.084020 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:31.083983 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn"] Nov 03 22:58:31.089503 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:31.089475 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-cbd26c/distinct-w5-cbd26c-kserve-559694f4c6-tdkzn"] Nov 03 22:58:31.200564 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:31.200533 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6"] Nov 03 22:58:31.205096 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:31.205070 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-cbd26c/simulator-w5-cbd26c-kserve-6d46dbb7d8-txlw6"] Nov 03 22:58:32.866880 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:32.866300 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82749c55-0fca-4a14-b6ba-7df122a6a407" path="/var/lib/kubelet/pods/82749c55-0fca-4a14-b6ba-7df122a6a407/volumes" Nov 03 22:58:32.867427 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:32.867056 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f2e3f52-0664-434f-919b-32736cbf0602" path="/var/lib/kubelet/pods/9f2e3f52-0664-434f-919b-32736cbf0602/volumes" Nov 03 22:58:35.679230 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:35.679189 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:35.887189 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:35.887163 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5" Nov 03 22:58:38.688005 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:38.687970 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn"] Nov 03 22:58:38.688479 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:38.688202 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn" podUID="58d33b25-27e9-4d87-bb8c-78033c13eff3" containerName="discovery" containerID="cri-o://30ef2c70c1bb1536d1f2979e06f34c1837bc539030811b4daf84af28b2057818" gracePeriod=30 Nov 03 22:58:38.963337 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:38.963258 5196 generic.go:356] "Generic (PLEG): container finished" podID="58d33b25-27e9-4d87-bb8c-78033c13eff3" containerID="30ef2c70c1bb1536d1f2979e06f34c1837bc539030811b4daf84af28b2057818" exitCode=0 Nov 03 22:58:38.963337 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:38.963323 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn" event={"ID":"58d33b25-27e9-4d87-bb8c-78033c13eff3","Type":"ContainerDied","Data":"30ef2c70c1bb1536d1f2979e06f34c1837bc539030811b4daf84af28b2057818"} Nov 03 22:58:39.055418 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:39.055105 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn" Nov 03 22:58:39.223928 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:39.222447 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58d33b25-27e9-4d87-bb8c-78033c13eff3-cacerts\" (UniqueName: \"kubernetes.io/secret/58d33b25-27e9-4d87-bb8c-78033c13eff3-cacerts\") pod \"58d33b25-27e9-4d87-bb8c-78033c13eff3\" (UID: \"58d33b25-27e9-4d87-bb8c-78033c13eff3\") " Nov 03 22:58:39.223928 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:39.222497 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58d33b25-27e9-4d87-bb8c-78033c13eff3-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/58d33b25-27e9-4d87-bb8c-78033c13eff3-istio-csr-ca-configmap\") pod \"58d33b25-27e9-4d87-bb8c-78033c13eff3\" (UID: \"58d33b25-27e9-4d87-bb8c-78033c13eff3\") " Nov 03 22:58:39.223928 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:39.222644 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58d33b25-27e9-4d87-bb8c-78033c13eff3-kube-api-access-lrqhb\" (UniqueName: \"kubernetes.io/projected/58d33b25-27e9-4d87-bb8c-78033c13eff3-kube-api-access-lrqhb\") pod \"58d33b25-27e9-4d87-bb8c-78033c13eff3\" (UID: \"58d33b25-27e9-4d87-bb8c-78033c13eff3\") " Nov 03 22:58:39.223928 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:39.222699 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58d33b25-27e9-4d87-bb8c-78033c13eff3-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d33b25-27e9-4d87-bb8c-78033c13eff3-local-certs\") pod \"58d33b25-27e9-4d87-bb8c-78033c13eff3\" (UID: \"58d33b25-27e9-4d87-bb8c-78033c13eff3\") " Nov 03 22:58:39.223928 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:39.222804 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58d33b25-27e9-4d87-bb8c-78033c13eff3-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/58d33b25-27e9-4d87-bb8c-78033c13eff3-istio-kubeconfig\") pod \"58d33b25-27e9-4d87-bb8c-78033c13eff3\" (UID: \"58d33b25-27e9-4d87-bb8c-78033c13eff3\") " Nov 03 22:58:39.223928 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:39.222888 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58d33b25-27e9-4d87-bb8c-78033c13eff3-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/58d33b25-27e9-4d87-bb8c-78033c13eff3-istio-csr-dns-cert\") pod \"58d33b25-27e9-4d87-bb8c-78033c13eff3\" (UID: \"58d33b25-27e9-4d87-bb8c-78033c13eff3\") " Nov 03 22:58:39.223928 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:39.222945 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58d33b25-27e9-4d87-bb8c-78033c13eff3-istio-token\" (UniqueName: \"kubernetes.io/projected/58d33b25-27e9-4d87-bb8c-78033c13eff3-istio-token\") pod \"58d33b25-27e9-4d87-bb8c-78033c13eff3\" (UID: \"58d33b25-27e9-4d87-bb8c-78033c13eff3\") " Nov 03 22:58:39.233235 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:39.225205 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58d33b25-27e9-4d87-bb8c-78033c13eff3-istio-csr-ca-configmap" pod "58d33b25-27e9-4d87-bb8c-78033c13eff3" (UID: "58d33b25-27e9-4d87-bb8c-78033c13eff3"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:58:39.236814 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:39.236609 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58d33b25-27e9-4d87-bb8c-78033c13eff3-istio-kubeconfig" pod "58d33b25-27e9-4d87-bb8c-78033c13eff3" (UID: "58d33b25-27e9-4d87-bb8c-78033c13eff3"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:58:39.236814 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:39.236685 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58d33b25-27e9-4d87-bb8c-78033c13eff3-cacerts" pod "58d33b25-27e9-4d87-bb8c-78033c13eff3" (UID: "58d33b25-27e9-4d87-bb8c-78033c13eff3"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:58:39.241184 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:39.241157 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58d33b25-27e9-4d87-bb8c-78033c13eff3-istio-csr-dns-cert" pod "58d33b25-27e9-4d87-bb8c-78033c13eff3" (UID: "58d33b25-27e9-4d87-bb8c-78033c13eff3"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:58:39.241329 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:39.241311 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d33b25-27e9-4d87-bb8c-78033c13eff3-kube-api-access-lrqhb" pod "58d33b25-27e9-4d87-bb8c-78033c13eff3" (UID: "58d33b25-27e9-4d87-bb8c-78033c13eff3"). InnerVolumeSpecName "kube-api-access-lrqhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:39.241446 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:39.241420 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d33b25-27e9-4d87-bb8c-78033c13eff3-istio-token" pod "58d33b25-27e9-4d87-bb8c-78033c13eff3" (UID: "58d33b25-27e9-4d87-bb8c-78033c13eff3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:39.246908 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:39.243686 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d33b25-27e9-4d87-bb8c-78033c13eff3-local-certs" pod "58d33b25-27e9-4d87-bb8c-78033c13eff3" (UID: "58d33b25-27e9-4d87-bb8c-78033c13eff3"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:39.325882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:39.324938 5196 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/58d33b25-27e9-4d87-bb8c-78033c13eff3-cacerts\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:39.325882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:39.324978 5196 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/58d33b25-27e9-4d87-bb8c-78033c13eff3-istio-csr-ca-configmap\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:39.325882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:39.324997 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrqhb\" (UniqueName: \"kubernetes.io/projected/58d33b25-27e9-4d87-bb8c-78033c13eff3-kube-api-access-lrqhb\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:39.325882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:39.325017 5196 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d33b25-27e9-4d87-bb8c-78033c13eff3-local-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:39.325882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:39.325031 5196 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/58d33b25-27e9-4d87-bb8c-78033c13eff3-istio-kubeconfig\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:39.325882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:39.325045 5196 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/58d33b25-27e9-4d87-bb8c-78033c13eff3-istio-csr-dns-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:39.325882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:39.325059 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58d33b25-27e9-4d87-bb8c-78033c13eff3-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:39.975059 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:39.975020 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn" Nov 03 22:58:39.975494 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:39.975044 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn" event={"ID":"58d33b25-27e9-4d87-bb8c-78033c13eff3","Type":"ContainerDied","Data":"d27f0a20fd95f04acb67aac1a8379f6432566557cd60c09cd44b163c7df287bb"} Nov 03 22:58:39.975494 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:39.975125 5196 scope.go:122] "RemoveContainer" containerID="30ef2c70c1bb1536d1f2979e06f34c1837bc539030811b4daf84af28b2057818" Nov 03 22:58:40.217832 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:40.217751 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn"] Nov 03 22:58:40.266051 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:40.265972 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vsltn"] Nov 03 22:58:40.655012 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:40.654982 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:40.823536 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:40.823504 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58d33b25-27e9-4d87-bb8c-78033c13eff3" path="/var/lib/kubelet/pods/58d33b25-27e9-4d87-bb8c-78033c13eff3/volumes" Nov 03 22:58:40.917707 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:40.917618 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk"] Nov 03 22:58:40.917883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:40.917811 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" podUID="552d9e04-ea9c-4e58-b861-dd19b8def5ec" containerName="istio-proxy" containerID="cri-o://751410b70db17196ae0eaf2ae0e80045a36bfee238b7c6a80af9b021dacd7e4e" gracePeriod=30 Nov 03 22:58:41.123043 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:41.123004 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46"] Nov 03 22:58:41.123493 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:41.123241 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" podUID="c974ac3a-2c08-40df-a4a1-c62b3d3fa040" containerName="istio-proxy" containerID="cri-o://f972cb5d9f3b6cd667637ec463a0527751eec2f01cc0de27c88d6a9cb6e97993" gracePeriod=30 Nov 03 22:58:41.630137 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:41.630094 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh"] Nov 03 22:58:41.630340 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:41.630318 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" podUID="5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb" containerName="istio-proxy" containerID="cri-o://981bd9f52a26c18d870bac2da7b4736f76abc9a7b166395d84275f978d8812d5" gracePeriod=30 Nov 03 22:58:42.559336 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:42.559302 5196 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:58:42.559739 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:42.559365 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" podUID="c974ac3a-2c08-40df-a4a1-c62b3d3fa040" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:58:43.481145 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:43.481109 5196 patch_prober.go:29] interesting pod/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:58:43.481302 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:43.481163 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" podUID="552d9e04-ea9c-4e58-b861-dd19b8def5ec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:58:45.349237 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:45.349194 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:46.255460 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.255438 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" Nov 03 22:58:46.318397 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.318368 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/552d9e04-ea9c-4e58-b861-dd19b8def5ec-kube-api-access-579zq\" (UniqueName: \"kubernetes.io/projected/552d9e04-ea9c-4e58-b861-dd19b8def5ec-kube-api-access-579zq\") pod \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\" (UID: \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\") " Nov 03 22:58:46.318564 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.318512 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/552d9e04-ea9c-4e58-b861-dd19b8def5ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/552d9e04-ea9c-4e58-b861-dd19b8def5ec-istio-data\") pod \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\" (UID: \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\") " Nov 03 22:58:46.318634 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.318579 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/552d9e04-ea9c-4e58-b861-dd19b8def5ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/552d9e04-ea9c-4e58-b861-dd19b8def5ec-istio-podinfo\") pod \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\" (UID: \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\") " Nov 03 22:58:46.318634 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.318628 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/552d9e04-ea9c-4e58-b861-dd19b8def5ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/552d9e04-ea9c-4e58-b861-dd19b8def5ec-istiod-ca-cert\") pod \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\" (UID: \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\") " Nov 03 22:58:46.319030 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.318753 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/552d9e04-ea9c-4e58-b861-dd19b8def5ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/552d9e04-ea9c-4e58-b861-dd19b8def5ec-workload-certs\") pod \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\" (UID: \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\") " Nov 03 22:58:46.319030 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.318788 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/552d9e04-ea9c-4e58-b861-dd19b8def5ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/552d9e04-ea9c-4e58-b861-dd19b8def5ec-workload-socket\") pod \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\" (UID: \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\") " Nov 03 22:58:46.319030 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.318847 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/552d9e04-ea9c-4e58-b861-dd19b8def5ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/552d9e04-ea9c-4e58-b861-dd19b8def5ec-istio-envoy\") pod \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\" (UID: \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\") " Nov 03 22:58:46.319030 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.318923 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/552d9e04-ea9c-4e58-b861-dd19b8def5ec-istio-token\" (UniqueName: \"kubernetes.io/projected/552d9e04-ea9c-4e58-b861-dd19b8def5ec-istio-token\") pod \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\" (UID: \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\") " Nov 03 22:58:46.319030 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.318987 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/552d9e04-ea9c-4e58-b861-dd19b8def5ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/552d9e04-ea9c-4e58-b861-dd19b8def5ec-credential-socket\") pod \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\" (UID: \"552d9e04-ea9c-4e58-b861-dd19b8def5ec\") " Nov 03 22:58:46.319333 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.319219 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/552d9e04-ea9c-4e58-b861-dd19b8def5ec-workload-socket" pod "552d9e04-ea9c-4e58-b861-dd19b8def5ec" (UID: "552d9e04-ea9c-4e58-b861-dd19b8def5ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:46.319607 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.319433 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/552d9e04-ea9c-4e58-b861-dd19b8def5ec-istio-data" pod "552d9e04-ea9c-4e58-b861-dd19b8def5ec" (UID: "552d9e04-ea9c-4e58-b861-dd19b8def5ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:46.319607 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.319446 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/552d9e04-ea9c-4e58-b861-dd19b8def5ec-credential-socket" pod "552d9e04-ea9c-4e58-b861-dd19b8def5ec" (UID: "552d9e04-ea9c-4e58-b861-dd19b8def5ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:46.319758 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.319603 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/552d9e04-ea9c-4e58-b861-dd19b8def5ec-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.319758 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.319627 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/552d9e04-ea9c-4e58-b861-dd19b8def5ec-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.319758 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.319643 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/552d9e04-ea9c-4e58-b861-dd19b8def5ec-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.320050 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.320005 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/552d9e04-ea9c-4e58-b861-dd19b8def5ec-workload-certs" pod "552d9e04-ea9c-4e58-b861-dd19b8def5ec" (UID: "552d9e04-ea9c-4e58-b861-dd19b8def5ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:46.320324 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.320294 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/552d9e04-ea9c-4e58-b861-dd19b8def5ec-istiod-ca-cert" pod "552d9e04-ea9c-4e58-b861-dd19b8def5ec" (UID: "552d9e04-ea9c-4e58-b861-dd19b8def5ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:58:46.323079 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.323057 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/552d9e04-ea9c-4e58-b861-dd19b8def5ec-istio-envoy" pod "552d9e04-ea9c-4e58-b861-dd19b8def5ec" (UID: "552d9e04-ea9c-4e58-b861-dd19b8def5ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:46.323173 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.323088 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/552d9e04-ea9c-4e58-b861-dd19b8def5ec-kube-api-access-579zq" pod "552d9e04-ea9c-4e58-b861-dd19b8def5ec" (UID: "552d9e04-ea9c-4e58-b861-dd19b8def5ec"). InnerVolumeSpecName "kube-api-access-579zq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:46.323400 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.323381 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/552d9e04-ea9c-4e58-b861-dd19b8def5ec-istio-token" pod "552d9e04-ea9c-4e58-b861-dd19b8def5ec" (UID: "552d9e04-ea9c-4e58-b861-dd19b8def5ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:46.324328 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.324306 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/552d9e04-ea9c-4e58-b861-dd19b8def5ec-istio-podinfo" pod "552d9e04-ea9c-4e58-b861-dd19b8def5ec" (UID: "552d9e04-ea9c-4e58-b861-dd19b8def5ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:58:46.400901 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.400875 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" Nov 03 22:58:46.421353 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.421321 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/552d9e04-ea9c-4e58-b861-dd19b8def5ec-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.421353 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.421354 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/552d9e04-ea9c-4e58-b861-dd19b8def5ec-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.421546 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.421371 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/552d9e04-ea9c-4e58-b861-dd19b8def5ec-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.421546 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.421384 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/552d9e04-ea9c-4e58-b861-dd19b8def5ec-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.421546 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.421396 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/552d9e04-ea9c-4e58-b861-dd19b8def5ec-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.421546 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.421409 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-579zq\" (UniqueName: \"kubernetes.io/projected/552d9e04-ea9c-4e58-b861-dd19b8def5ec-kube-api-access-579zq\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.522663 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.522572 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-credential-socket\") pod \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\" (UID: \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\") " Nov 03 22:58:46.522663 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.522628 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-istio-envoy\") pod \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\" (UID: \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\") " Nov 03 22:58:46.522955 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.522670 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-workload-certs\") pod \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\" (UID: \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\") " Nov 03 22:58:46.522955 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.522767 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-istio-token\" (UniqueName: \"kubernetes.io/projected/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-istio-token\") pod \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\" (UID: \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\") " Nov 03 22:58:46.522955 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.522801 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-istio-podinfo\") pod \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\" (UID: \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\") " Nov 03 22:58:46.522955 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.522848 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-workload-socket\") pod \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\" (UID: \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\") " Nov 03 22:58:46.522955 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.522910 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-credential-socket" pod "c974ac3a-2c08-40df-a4a1-c62b3d3fa040" (UID: "c974ac3a-2c08-40df-a4a1-c62b3d3fa040"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:46.523296 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.522998 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-istio-data\") pod \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\" (UID: \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\") " Nov 03 22:58:46.523296 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.523046 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-kube-api-access-zwlk2\" (UniqueName: \"kubernetes.io/projected/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-kube-api-access-zwlk2\") pod \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\" (UID: \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\") " Nov 03 22:58:46.523296 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.523130 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-workload-certs" pod "c974ac3a-2c08-40df-a4a1-c62b3d3fa040" (UID: "c974ac3a-2c08-40df-a4a1-c62b3d3fa040"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:46.523296 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.523142 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-workload-socket" pod "c974ac3a-2c08-40df-a4a1-c62b3d3fa040" (UID: "c974ac3a-2c08-40df-a4a1-c62b3d3fa040"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:46.523296 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.523220 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-istiod-ca-cert\") pod \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\" (UID: \"c974ac3a-2c08-40df-a4a1-c62b3d3fa040\") " Nov 03 22:58:46.523296 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.523277 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-istio-data" pod "c974ac3a-2c08-40df-a4a1-c62b3d3fa040" (UID: "c974ac3a-2c08-40df-a4a1-c62b3d3fa040"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:46.523787 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.523760 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-istiod-ca-cert" pod "c974ac3a-2c08-40df-a4a1-c62b3d3fa040" (UID: "c974ac3a-2c08-40df-a4a1-c62b3d3fa040"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:58:46.523997 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.523978 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.523997 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.524001 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.523997 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.524019 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.523997 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.524035 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.523997 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.524052 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.525156 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.525134 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-istio-envoy" pod "c974ac3a-2c08-40df-a4a1-c62b3d3fa040" (UID: "c974ac3a-2c08-40df-a4a1-c62b3d3fa040"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:46.525156 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.525132 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-istio-podinfo" pod "c974ac3a-2c08-40df-a4a1-c62b3d3fa040" (UID: "c974ac3a-2c08-40df-a4a1-c62b3d3fa040"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:58:46.525156 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.525134 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-istio-token" pod "c974ac3a-2c08-40df-a4a1-c62b3d3fa040" (UID: "c974ac3a-2c08-40df-a4a1-c62b3d3fa040"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:46.525495 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.525479 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-kube-api-access-zwlk2" pod "c974ac3a-2c08-40df-a4a1-c62b3d3fa040" (UID: "c974ac3a-2c08-40df-a4a1-c62b3d3fa040"). InnerVolumeSpecName "kube-api-access-zwlk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:46.625514 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.625474 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwlk2\" (UniqueName: \"kubernetes.io/projected/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-kube-api-access-zwlk2\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.625514 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.625507 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.625724 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.625524 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.625724 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.625537 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c974ac3a-2c08-40df-a4a1-c62b3d3fa040-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.880812 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.880785 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" Nov 03 22:58:46.929968 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.929939 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-credential-socket\") pod \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\" (UID: \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\") " Nov 03 22:58:46.930142 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.929991 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-istio-token\" (UniqueName: \"kubernetes.io/projected/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-istio-token\") pod \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\" (UID: \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\") " Nov 03 22:58:46.930142 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.930027 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-istio-data\") pod \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\" (UID: \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\") " Nov 03 22:58:46.930142 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.930100 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-workload-certs\") pod \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\" (UID: \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\") " Nov 03 22:58:46.930304 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.930220 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-istio-envoy\") pod \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\" (UID: \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\") " Nov 03 22:58:46.930304 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.930241 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-credential-socket" pod "5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb" (UID: "5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:46.930416 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.930332 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-istio-podinfo\") pod \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\" (UID: \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\") " Nov 03 22:58:46.930416 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.930373 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-kube-api-access-scr6d\" (UniqueName: \"kubernetes.io/projected/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-kube-api-access-scr6d\") pod \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\" (UID: \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\") " Nov 03 22:58:46.930416 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.930381 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-istio-data" pod "5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb" (UID: "5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:46.930416 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.930399 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-workload-certs" pod "5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb" (UID: "5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:46.930609 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.930403 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-workload-socket\") pod \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\" (UID: \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\") " Nov 03 22:58:46.930609 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.930483 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-istiod-ca-cert\") pod \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\" (UID: \"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb\") " Nov 03 22:58:46.930752 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.930606 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-workload-socket" pod "5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb" (UID: "5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:46.931391 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.930846 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-istiod-ca-cert" pod "5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb" (UID: "5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:58:46.931391 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.931181 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.931391 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.931202 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.931391 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.931217 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.931391 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.931235 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.931391 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.931249 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:46.932615 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.932592 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-istio-envoy" pod "5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb" (UID: "5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:58:46.932714 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.932604 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-istio-token" pod "5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb" (UID: "5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:46.933023 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.933007 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-istio-podinfo" pod "5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb" (UID: "5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:58:46.933067 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:46.933040 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-kube-api-access-scr6d" pod "5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb" (UID: "5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb"). InnerVolumeSpecName "kube-api-access-scr6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:47.028258 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:47.028225 5196 generic.go:356] "Generic (PLEG): container finished" podID="5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb" containerID="981bd9f52a26c18d870bac2da7b4736f76abc9a7b166395d84275f978d8812d5" exitCode=0 Nov 03 22:58:47.028471 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:47.028284 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" Nov 03 22:58:47.028471 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:47.028314 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" event={"ID":"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb","Type":"ContainerDied","Data":"981bd9f52a26c18d870bac2da7b4736f76abc9a7b166395d84275f978d8812d5"} Nov 03 22:58:47.028471 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:47.028370 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh" event={"ID":"5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb","Type":"ContainerDied","Data":"e5db1f0dab1ace1d151471a628a5e0bc9f381a831a5a20473119adf7e5d143b2"} Nov 03 22:58:47.028471 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:47.028399 5196 scope.go:122] "RemoveContainer" containerID="981bd9f52a26c18d870bac2da7b4736f76abc9a7b166395d84275f978d8812d5" Nov 03 22:58:47.031095 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:47.031068 5196 generic.go:356] "Generic (PLEG): container finished" podID="552d9e04-ea9c-4e58-b861-dd19b8def5ec" containerID="751410b70db17196ae0eaf2ae0e80045a36bfee238b7c6a80af9b021dacd7e4e" exitCode=0 Nov 03 22:58:47.031229 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:47.031138 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" event={"ID":"552d9e04-ea9c-4e58-b861-dd19b8def5ec","Type":"ContainerDied","Data":"751410b70db17196ae0eaf2ae0e80045a36bfee238b7c6a80af9b021dacd7e4e"} Nov 03 22:58:47.031229 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:47.031177 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" event={"ID":"552d9e04-ea9c-4e58-b861-dd19b8def5ec","Type":"ContainerDied","Data":"e7ed88a204d9bb0a52833aa863d8c584310bed67e4807b55c9c3ff72ad54d73a"} Nov 03 22:58:47.031352 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:47.031239 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk" Nov 03 22:58:47.032522 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:47.032501 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:47.032699 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:47.032527 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:47.032699 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:47.032542 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:47.032699 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:47.032554 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scr6d\" (UniqueName: \"kubernetes.io/projected/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb-kube-api-access-scr6d\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:58:47.038909 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:47.038376 5196 generic.go:356] "Generic (PLEG): container finished" podID="c974ac3a-2c08-40df-a4a1-c62b3d3fa040" containerID="f972cb5d9f3b6cd667637ec463a0527751eec2f01cc0de27c88d6a9cb6e97993" exitCode=0 Nov 03 22:58:47.038909 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:47.038537 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" event={"ID":"c974ac3a-2c08-40df-a4a1-c62b3d3fa040","Type":"ContainerDied","Data":"f972cb5d9f3b6cd667637ec463a0527751eec2f01cc0de27c88d6a9cb6e97993"} Nov 03 22:58:47.038909 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:47.038581 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" event={"ID":"c974ac3a-2c08-40df-a4a1-c62b3d3fa040","Type":"ContainerDied","Data":"7ddc2bb8df66d4752d1072e32b6534eabbbceeb9b8893da6023435f58b63d9b5"} Nov 03 22:58:47.038909 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:47.038586 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46" Nov 03 22:58:47.043936 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:47.043884 5196 scope.go:122] "RemoveContainer" containerID="981bd9f52a26c18d870bac2da7b4736f76abc9a7b166395d84275f978d8812d5" Nov 03 22:58:47.044259 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:58:47.044233 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"981bd9f52a26c18d870bac2da7b4736f76abc9a7b166395d84275f978d8812d5\": container with ID starting with 981bd9f52a26c18d870bac2da7b4736f76abc9a7b166395d84275f978d8812d5 not found: ID does not exist" containerID="981bd9f52a26c18d870bac2da7b4736f76abc9a7b166395d84275f978d8812d5" Nov 03 22:58:47.044337 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:47.044273 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"981bd9f52a26c18d870bac2da7b4736f76abc9a7b166395d84275f978d8812d5"} err="failed to get container status \"981bd9f52a26c18d870bac2da7b4736f76abc9a7b166395d84275f978d8812d5\": rpc error: code = NotFound desc = could not find container \"981bd9f52a26c18d870bac2da7b4736f76abc9a7b166395d84275f978d8812d5\": container with ID starting with 981bd9f52a26c18d870bac2da7b4736f76abc9a7b166395d84275f978d8812d5 not found: ID does not exist" Nov 03 22:58:47.044337 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:47.044299 5196 scope.go:122] "RemoveContainer" containerID="751410b70db17196ae0eaf2ae0e80045a36bfee238b7c6a80af9b021dacd7e4e" Nov 03 22:58:47.055021 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:47.054996 5196 scope.go:122] "RemoveContainer" containerID="751410b70db17196ae0eaf2ae0e80045a36bfee238b7c6a80af9b021dacd7e4e" Nov 03 22:58:47.055363 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:58:47.055341 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"751410b70db17196ae0eaf2ae0e80045a36bfee238b7c6a80af9b021dacd7e4e\": container with ID starting with 751410b70db17196ae0eaf2ae0e80045a36bfee238b7c6a80af9b021dacd7e4e not found: ID does not exist" containerID="751410b70db17196ae0eaf2ae0e80045a36bfee238b7c6a80af9b021dacd7e4e" Nov 03 22:58:47.055455 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:47.055370 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"751410b70db17196ae0eaf2ae0e80045a36bfee238b7c6a80af9b021dacd7e4e"} err="failed to get container status \"751410b70db17196ae0eaf2ae0e80045a36bfee238b7c6a80af9b021dacd7e4e\": rpc error: code = NotFound desc = could not find container \"751410b70db17196ae0eaf2ae0e80045a36bfee238b7c6a80af9b021dacd7e4e\": container with ID starting with 751410b70db17196ae0eaf2ae0e80045a36bfee238b7c6a80af9b021dacd7e4e not found: ID does not exist" Nov 03 22:58:47.055455 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:47.055387 5196 scope.go:122] "RemoveContainer" containerID="f972cb5d9f3b6cd667637ec463a0527751eec2f01cc0de27c88d6a9cb6e97993" Nov 03 22:58:47.066737 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:47.066717 5196 scope.go:122] "RemoveContainer" containerID="f972cb5d9f3b6cd667637ec463a0527751eec2f01cc0de27c88d6a9cb6e97993" Nov 03 22:58:47.067083 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:58:47.067064 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f972cb5d9f3b6cd667637ec463a0527751eec2f01cc0de27c88d6a9cb6e97993\": container with ID starting with f972cb5d9f3b6cd667637ec463a0527751eec2f01cc0de27c88d6a9cb6e97993 not found: ID does not exist" containerID="f972cb5d9f3b6cd667637ec463a0527751eec2f01cc0de27c88d6a9cb6e97993" Nov 03 22:58:47.067184 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:47.067088 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f972cb5d9f3b6cd667637ec463a0527751eec2f01cc0de27c88d6a9cb6e97993"} err="failed to get container status \"f972cb5d9f3b6cd667637ec463a0527751eec2f01cc0de27c88d6a9cb6e97993\": rpc error: code = NotFound desc = could not find container \"f972cb5d9f3b6cd667637ec463a0527751eec2f01cc0de27c88d6a9cb6e97993\": container with ID starting with f972cb5d9f3b6cd667637ec463a0527751eec2f01cc0de27c88d6a9cb6e97993 not found: ID does not exist" Nov 03 22:58:47.202412 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:47.202385 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk"] Nov 03 22:58:47.206437 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:47.206411 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-2hmfk"] Nov 03 22:58:47.317706 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:47.317601 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46"] Nov 03 22:58:47.326238 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:47.326208 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-vvn46"] Nov 03 22:58:47.440087 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:47.440050 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh"] Nov 03 22:58:47.444181 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:47.444155 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-q95hh"] Nov 03 22:58:48.581162 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:48.581128 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:48.728384 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:48.728347 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:48.815231 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:48.815193 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="552d9e04-ea9c-4e58-b861-dd19b8def5ec" path="/var/lib/kubelet/pods/552d9e04-ea9c-4e58-b861-dd19b8def5ec/volumes" Nov 03 22:58:48.815748 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:48.815727 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb" path="/var/lib/kubelet/pods/5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb/volumes" Nov 03 22:58:48.816275 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:48.816247 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c974ac3a-2c08-40df-a4a1-c62b3d3fa040" path="/var/lib/kubelet/pods/c974ac3a-2c08-40df-a4a1-c62b3d3fa040/volumes" Nov 03 22:58:49.868000 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:49.867974 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr"] Nov 03 22:58:49.868452 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:49.868439 5196 cpu_manager.go:414] "containerMap: removing container" podUID="552d9e04-ea9c-4e58-b861-dd19b8def5ec" containerName="istio-proxy" Nov 03 22:58:49.868509 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:49.868452 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="552d9e04-ea9c-4e58-b861-dd19b8def5ec" containerName="istio-proxy" Nov 03 22:58:49.868509 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:49.868471 5196 cpu_manager.go:414] "containerMap: removing container" podUID="9f2e3f52-0664-434f-919b-32736cbf0602" containerName="main" Nov 03 22:58:49.868509 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:49.868477 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f2e3f52-0664-434f-919b-32736cbf0602" containerName="main" Nov 03 22:58:49.868509 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:49.868485 5196 cpu_manager.go:414] "containerMap: removing container" podUID="c974ac3a-2c08-40df-a4a1-c62b3d3fa040" containerName="istio-proxy" Nov 03 22:58:49.868509 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:49.868490 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c974ac3a-2c08-40df-a4a1-c62b3d3fa040" containerName="istio-proxy" Nov 03 22:58:49.868509 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:49.868506 5196 cpu_manager.go:414] "containerMap: removing container" podUID="58d33b25-27e9-4d87-bb8c-78033c13eff3" containerName="discovery" Nov 03 22:58:49.868509 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:49.868511 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58d33b25-27e9-4d87-bb8c-78033c13eff3" containerName="discovery" Nov 03 22:58:49.868720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:49.868525 5196 cpu_manager.go:414] "containerMap: removing container" podUID="5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb" containerName="istio-proxy" Nov 03 22:58:49.868720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:49.868530 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb" containerName="istio-proxy" Nov 03 22:58:49.868720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:49.868542 5196 cpu_manager.go:414] "containerMap: removing container" podUID="82749c55-0fca-4a14-b6ba-7df122a6a407" containerName="main" Nov 03 22:58:49.868720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:49.868547 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82749c55-0fca-4a14-b6ba-7df122a6a407" containerName="main" Nov 03 22:58:49.868720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:49.868631 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c75a6b1-9f57-4b0b-bafb-06e1dd7957bb" containerName="istio-proxy" Nov 03 22:58:49.868720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:49.868641 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="75a0d61a-c8e8-4925-941b-d729b4b21114" containerName="istio-proxy" Nov 03 22:58:49.868720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:49.868648 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f2e3f52-0664-434f-919b-32736cbf0602" containerName="main" Nov 03 22:58:49.868720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:49.868654 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="82749c55-0fca-4a14-b6ba-7df122a6a407" containerName="main" Nov 03 22:58:49.868720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:49.868662 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="58d33b25-27e9-4d87-bb8c-78033c13eff3" containerName="discovery" Nov 03 22:58:49.868720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:49.868668 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="c974ac3a-2c08-40df-a4a1-c62b3d3fa040" containerName="istio-proxy" Nov 03 22:58:49.868720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:49.868674 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="552d9e04-ea9c-4e58-b861-dd19b8def5ec" containerName="istio-proxy" Nov 03 22:58:49.873447 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:49.873428 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr" Nov 03 22:58:49.928774 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:49.928744 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-f8cd0c-serving-cert\"" Nov 03 22:58:49.932456 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:49.932427 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr"] Nov 03 22:58:49.963643 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:49.963609 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4042310-83e8-44e6-b8e4-73ecb70403b3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr\" (UID: \"d4042310-83e8-44e6-b8e4-73ecb70403b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr" Nov 03 22:58:49.963813 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:49.963653 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svkzw\" (UniqueName: \"kubernetes.io/projected/d4042310-83e8-44e6-b8e4-73ecb70403b3-kube-api-access-svkzw\") pod \"maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr\" (UID: \"d4042310-83e8-44e6-b8e4-73ecb70403b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr" Nov 03 22:58:49.963813 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:49.963778 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4042310-83e8-44e6-b8e4-73ecb70403b3-maas-api-tls\") pod \"maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr\" (UID: \"d4042310-83e8-44e6-b8e4-73ecb70403b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr" Nov 03 22:58:49.964060 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:49.963810 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4042310-83e8-44e6-b8e4-73ecb70403b3-metrics-certs\") pod \"maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr\" (UID: \"d4042310-83e8-44e6-b8e4-73ecb70403b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr" Nov 03 22:58:50.056836 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:50.056805 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:50.064799 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:50.064774 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4042310-83e8-44e6-b8e4-73ecb70403b3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr\" (UID: \"d4042310-83e8-44e6-b8e4-73ecb70403b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr" Nov 03 22:58:50.064931 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:50.064805 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svkzw\" (UniqueName: \"kubernetes.io/projected/d4042310-83e8-44e6-b8e4-73ecb70403b3-kube-api-access-svkzw\") pod \"maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr\" (UID: \"d4042310-83e8-44e6-b8e4-73ecb70403b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr" Nov 03 22:58:50.065003 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:50.064981 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4042310-83e8-44e6-b8e4-73ecb70403b3-maas-api-tls\") pod \"maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr\" (UID: \"d4042310-83e8-44e6-b8e4-73ecb70403b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr" Nov 03 22:58:50.065066 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:50.065033 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4042310-83e8-44e6-b8e4-73ecb70403b3-metrics-certs\") pod \"maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr\" (UID: \"d4042310-83e8-44e6-b8e4-73ecb70403b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr" Nov 03 22:58:50.065125 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:58:50.065114 5196 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-f8cd0c-serving-cert: secret "maas-api-e2e-inf-a-f8cd0c-serving-cert" not found Nov 03 22:58:50.065178 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:58:50.065170 5196 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4042310-83e8-44e6-b8e4-73ecb70403b3-maas-api-tls podName:d4042310-83e8-44e6-b8e4-73ecb70403b3 nodeName:}" failed. No retries permitted until 2026-11-03 22:58:50.565156739 +0000 UTC m=+2159.874716026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d4042310-83e8-44e6-b8e4-73ecb70403b3-maas-api-tls") pod "maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr" (UID: "d4042310-83e8-44e6-b8e4-73ecb70403b3") : secret "maas-api-e2e-inf-a-f8cd0c-serving-cert" not found Nov 03 22:58:50.065520 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:50.065498 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4042310-83e8-44e6-b8e4-73ecb70403b3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr\" (UID: \"d4042310-83e8-44e6-b8e4-73ecb70403b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr" Nov 03 22:58:50.067639 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:50.067616 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4042310-83e8-44e6-b8e4-73ecb70403b3-metrics-certs\") pod \"maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr\" (UID: \"d4042310-83e8-44e6-b8e4-73ecb70403b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr" Nov 03 22:58:50.129806 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:50.129716 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svkzw\" (UniqueName: \"kubernetes.io/projected/d4042310-83e8-44e6-b8e4-73ecb70403b3-kube-api-access-svkzw\") pod \"maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr\" (UID: \"d4042310-83e8-44e6-b8e4-73ecb70403b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr" Nov 03 22:58:50.211819 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:50.211785 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:50.392680 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:50.392594 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:50.570616 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:50.570575 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4042310-83e8-44e6-b8e4-73ecb70403b3-maas-api-tls\") pod \"maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr\" (UID: \"d4042310-83e8-44e6-b8e4-73ecb70403b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr" Nov 03 22:58:50.573148 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:50.573125 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4042310-83e8-44e6-b8e4-73ecb70403b3-maas-api-tls\") pod \"maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr\" (UID: \"d4042310-83e8-44e6-b8e4-73ecb70403b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr" Nov 03 22:58:50.786529 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:50.786447 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr" Nov 03 22:58:51.167033 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:51.166997 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr"] Nov 03 22:58:51.168691 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:58:51.168665 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4042310_83e8_44e6_b8e4_73ecb70403b3.slice/crio-c79b365a0a319b9f3afd64098ffcbe694a9d2e11b8c0c549f92b2a898ee7ebbd WatchSource:0}: Error finding container c79b365a0a319b9f3afd64098ffcbe694a9d2e11b8c0c549f92b2a898ee7ebbd: Status 404 returned error can't find the container with id c79b365a0a319b9f3afd64098ffcbe694a9d2e11b8c0c549f92b2a898ee7ebbd Nov 03 22:58:52.075233 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:52.075189 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr" event={"ID":"d4042310-83e8-44e6-b8e4-73ecb70403b3","Type":"ContainerStarted","Data":"c79b365a0a319b9f3afd64098ffcbe694a9d2e11b8c0c549f92b2a898ee7ebbd"} Nov 03 22:58:52.212399 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:52.212319 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:52.358601 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:52.358566 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:53.084184 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:53.084144 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr" event={"ID":"d4042310-83e8-44e6-b8e4-73ecb70403b3","Type":"ContainerStarted","Data":"f31f9fafc261f77015412837af9f54e9b660ca7379393f4953e1fc7bc6e96bd0"} Nov 03 22:58:53.084379 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:53.084257 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr" Nov 03 22:58:53.204144 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:53.204097 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:53.205116 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:53.205079 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr" podStartSLOduration=3.347267542 podStartE2EDuration="4.205066662s" totalImagesPullingTime="857.79912ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:58:53.203205658 +0000 UTC m=+2162.512764952" watchObservedRunningTime="2026-11-03 22:58:53.205066662 +0000 UTC m=+2162.514625955" Nov 03 22:58:54.583910 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:54.583840 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:55.819089 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:55.819051 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t"] Nov 03 22:58:55.824188 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:55.824158 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" Nov 03 22:58:55.922890 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:58:55.922817 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:58:55.931682 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:55.931656 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t"] Nov 03 22:58:55.936602 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:55.936573 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-istio-data\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t\" (UID: \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" Nov 03 22:58:55.936744 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:55.936607 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-istio-envoy\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t\" (UID: \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" Nov 03 22:58:55.936744 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:55.936630 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-credential-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t\" (UID: \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" Nov 03 22:58:55.936912 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:55.936749 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-workload-certs\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t\" (UID: \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" Nov 03 22:58:55.936912 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:55.936892 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-istiod-ca-cert\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t\" (UID: \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" Nov 03 22:58:55.937045 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:55.936913 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmbgb\" (UniqueName: \"kubernetes.io/projected/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-kube-api-access-gmbgb\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t\" (UID: \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" Nov 03 22:58:55.937045 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:55.936937 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-istio-podinfo\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t\" (UID: \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" Nov 03 22:58:55.937045 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:55.937006 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-istio-token\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t\" (UID: \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" Nov 03 22:58:55.937222 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:55.937146 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-workload-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t\" (UID: \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" Nov 03 22:58:56.038292 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:56.038257 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-istiod-ca-cert\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t\" (UID: \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" Nov 03 22:58:56.038292 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:56.038293 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmbgb\" (UniqueName: \"kubernetes.io/projected/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-kube-api-access-gmbgb\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t\" (UID: \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" Nov 03 22:58:56.038518 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:56.038315 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-istio-podinfo\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t\" (UID: \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" Nov 03 22:58:56.038518 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:56.038499 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-istio-token\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t\" (UID: \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" Nov 03 22:58:56.038617 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:56.038598 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-workload-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t\" (UID: \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" Nov 03 22:58:56.038723 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:56.038705 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-istio-data\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t\" (UID: \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" Nov 03 22:58:56.038807 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:56.038741 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-istio-envoy\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t\" (UID: \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" Nov 03 22:58:56.038807 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:56.038772 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-credential-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t\" (UID: \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" Nov 03 22:58:56.038950 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:56.038821 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-workload-certs\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t\" (UID: \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" Nov 03 22:58:56.039077 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:56.039018 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-workload-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t\" (UID: \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" Nov 03 22:58:56.039077 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:56.039044 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-istio-data\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t\" (UID: \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" Nov 03 22:58:56.039077 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:56.039067 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-istiod-ca-cert\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t\" (UID: \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" Nov 03 22:58:56.039373 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:56.039226 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-credential-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t\" (UID: \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" Nov 03 22:58:56.039373 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:56.039277 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-workload-certs\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t\" (UID: \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" Nov 03 22:58:56.041044 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:56.041020 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-istio-podinfo\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t\" (UID: \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" Nov 03 22:58:56.041181 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:56.041165 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-istio-envoy\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t\" (UID: \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" Nov 03 22:58:56.119895 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:56.117479 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-istio-token\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t\" (UID: \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" Nov 03 22:58:56.121197 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:56.121038 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmbgb\" (UniqueName: \"kubernetes.io/projected/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-kube-api-access-gmbgb\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t\" (UID: \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" Nov 03 22:58:56.139608 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:56.139572 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" Nov 03 22:58:56.162189 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:56.162153 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:56.436693 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:56.436615 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:56.580917 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:56.580891 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t"] Nov 03 22:58:56.582778 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:58:56.582748 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d0e7774_f0b0_427d_bbd4_7070d70b8fca.slice/crio-f150be4bc397750f4cb584e9996ad460db41d469f90bcd0d364909fbd9a173ad WatchSource:0}: Error finding container f150be4bc397750f4cb584e9996ad460db41d469f90bcd0d364909fbd9a173ad: Status 404 returned error can't find the container with id f150be4bc397750f4cb584e9996ad460db41d469f90bcd0d364909fbd9a173ad Nov 03 22:58:56.585826 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:56.585787 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:58:56.585948 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:56.585910 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:58:56.585948 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:56.585944 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:58:56.587980 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:56.587949 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:57.124151 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:57.124114 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:57.135124 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:57.135092 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" event={"ID":"9d0e7774-f0b0-427d-bbd4-7070d70b8fca","Type":"ContainerStarted","Data":"c52e75cd15a3b17b8ddcc2930732c1275843e2434da7cf7c47b4a8205e8ffc75"} Nov 03 22:58:57.135300 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:57.135142 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" event={"ID":"9d0e7774-f0b0-427d-bbd4-7070d70b8fca","Type":"ContainerStarted","Data":"f150be4bc397750f4cb584e9996ad460db41d469f90bcd0d364909fbd9a173ad"} Nov 03 22:58:57.142521 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:57.142493 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" Nov 03 22:58:57.147244 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:57.144830 5196 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 03 22:58:57.147244 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:57.144900 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" podUID="9d0e7774-f0b0-427d-bbd4-7070d70b8fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 03 22:58:57.257964 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:57.257914 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" podStartSLOduration=2.257897806 podStartE2EDuration="2.257897806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:58:57.256344386 +0000 UTC m=+2166.565903703" watchObservedRunningTime="2026-11-03 22:58:57.257897806 +0000 UTC m=+2166.567457099" Nov 03 22:58:58.140279 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:58.140243 5196 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 03 22:58:58.140692 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:58.140294 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" podUID="9d0e7774-f0b0-427d-bbd4-7070d70b8fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 03 22:58:59.117517 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:59.117485 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr" Nov 03 22:58:59.140792 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:59.140762 5196 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 03 22:58:59.141298 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:59.140818 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" podUID="9d0e7774-f0b0-427d-bbd4-7070d70b8fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 03 22:58:59.284275 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:59.284243 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:58:59.429568 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:58:59.429456 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:00.140642 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:00.140605 5196 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 03 22:59:00.140806 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:00.140662 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" podUID="9d0e7774-f0b0-427d-bbd4-7070d70b8fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 03 22:59:01.141079 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:01.141041 5196 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 03 22:59:01.141697 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:01.141666 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" podUID="9d0e7774-f0b0-427d-bbd4-7070d70b8fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 03 22:59:01.673521 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:01.673485 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:02.140644 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:02.140607 5196 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 03 22:59:02.140907 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:02.140680 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" podUID="9d0e7774-f0b0-427d-bbd4-7070d70b8fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 03 22:59:03.198669 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:03.198632 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" Nov 03 22:59:03.199850 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:03.199827 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" Nov 03 22:59:03.200711 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:03.200686 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" Nov 03 22:59:03.364279 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:03.364243 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:03.671392 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:03.671361 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:03.769130 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:03.769092 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj"] Nov 03 22:59:03.769615 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:03.769581 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" podUID="9fde3ea1-b20e-4828-b8e6-5ffd537294f6" containerName="istio-proxy" containerID="cri-o://7036bd1effaa4e0ac338ddb97f2e51a6bd0e6d9aacb1a4b14b59bcae9d216231" gracePeriod=30 Nov 03 22:59:03.829261 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:03.829185 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:04.632105 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:04.632033 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:05.238489 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:05.238448 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm"] Nov 03 22:59:05.242684 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:05.242666 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 22:59:05.305886 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:05.296921 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ea5c8f-openshift-default-dockercfg-qhhhz\"" Nov 03 22:59:05.359042 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:05.359005 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm"] Nov 03 22:59:05.369834 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:05.368617 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:05.391959 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:05.391811 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-istio-token\") pod \"e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm\" (UID: \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\") " pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 22:59:05.392165 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:05.392086 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-istio-envoy\") pod \"e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm\" (UID: \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\") " pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 22:59:05.392165 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:05.392133 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-istio-data\") pod \"e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm\" (UID: \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\") " pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 22:59:05.392314 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:05.392234 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-workload-socket\") pod \"e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm\" (UID: \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\") " pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 22:59:05.392436 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:05.392408 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-istiod-ca-cert\") pod \"e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm\" (UID: \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\") " pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 22:59:05.392579 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:05.392470 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-istio-podinfo\") pod \"e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm\" (UID: \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\") " pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 22:59:05.392579 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:05.392537 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-credential-socket\") pod \"e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm\" (UID: \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\") " pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 22:59:05.392700 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:05.392590 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzkzw\" (UniqueName: \"kubernetes.io/projected/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-kube-api-access-qzkzw\") pod \"e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm\" (UID: \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\") " pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 22:59:05.392700 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:05.392636 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-workload-certs\") pod \"e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm\" (UID: \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\") " pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 22:59:05.493937 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:05.493838 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-credential-socket\") pod \"e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm\" (UID: \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\") " pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 22:59:05.493937 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:05.493913 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzkzw\" (UniqueName: \"kubernetes.io/projected/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-kube-api-access-qzkzw\") pod \"e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm\" (UID: \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\") " pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 22:59:05.494172 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:05.493957 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-workload-certs\") pod \"e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm\" (UID: \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\") " pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 22:59:05.494172 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:05.494008 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-istio-token\") pod \"e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm\" (UID: \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\") " pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 22:59:05.494172 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:05.494057 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-istio-envoy\") pod \"e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm\" (UID: \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\") " pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 22:59:05.494172 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:05.494087 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-istio-data\") pod \"e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm\" (UID: \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\") " pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 22:59:05.494172 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:05.494115 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-workload-socket\") pod \"e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm\" (UID: \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\") " pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 22:59:05.494424 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:05.494273 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-credential-socket\") pod \"e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm\" (UID: \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\") " pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 22:59:05.494424 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:05.494285 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-istiod-ca-cert\") pod \"e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm\" (UID: \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\") " pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 22:59:05.494424 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:05.494362 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-istio-podinfo\") pod \"e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm\" (UID: \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\") " pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 22:59:05.494979 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:05.494955 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-istiod-ca-cert\") pod \"e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm\" (UID: \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\") " pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 22:59:05.495891 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:05.495403 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-workload-certs\") pod \"e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm\" (UID: \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\") " pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 22:59:05.496067 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:05.495539 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-istio-data\") pod \"e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm\" (UID: \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\") " pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 22:59:05.496067 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:05.495683 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-workload-socket\") pod \"e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm\" (UID: \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\") " pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 22:59:05.497461 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:05.497439 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-istio-podinfo\") pod \"e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm\" (UID: \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\") " pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 22:59:05.497974 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:05.497957 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-istio-envoy\") pod \"e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm\" (UID: \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\") " pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 22:59:05.556794 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:05.556764 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-istio-token\") pod \"e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm\" (UID: \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\") " pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 22:59:05.557047 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:05.557027 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzkzw\" (UniqueName: \"kubernetes.io/projected/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-kube-api-access-qzkzw\") pod \"e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm\" (UID: \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\") " pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 22:59:05.839081 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:05.839012 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:05.856542 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:05.856520 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 22:59:06.296728 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:06.296699 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm"] Nov 03 22:59:06.297535 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:59:06.297502 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc332f22b_f8b4_4b95_a2dc_bac68ef527d7.slice/crio-aa6bfc3d1af80f8b2cb8d1223fc92f3aacfc7af1e045706cff328a62954c77a2 WatchSource:0}: Error finding container aa6bfc3d1af80f8b2cb8d1223fc92f3aacfc7af1e045706cff328a62954c77a2: Status 404 returned error can't find the container with id aa6bfc3d1af80f8b2cb8d1223fc92f3aacfc7af1e045706cff328a62954c77a2 Nov 03 22:59:06.300200 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:06.300164 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:59:06.300308 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:06.300246 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:59:06.300308 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:06.300294 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:59:06.309137 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:06.309113 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w"] Nov 03 22:59:06.309297 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:06.309278 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w" podUID="9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7" containerName="maas-api" containerID="cri-o://cb7e033ae66f070148ca45f9182b60397ad0ec9e33d90a8cc406cae6df0e87a9" gracePeriod=30 Nov 03 22:59:06.406742 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:06.406710 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:06.632504 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:06.632476 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w" Nov 03 22:59:06.709040 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:06.709006 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7-metrics-certs\") pod \"9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7\" (UID: \"9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7\") " Nov 03 22:59:06.709237 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:06.709070 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7-maas-api-tls\") pod \"9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7\" (UID: \"9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7\") " Nov 03 22:59:06.709237 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:06.709140 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7-openshift-service-ca\") pod \"9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7\" (UID: \"9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7\") " Nov 03 22:59:06.709364 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:06.709259 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7-kube-api-access-qdvzh\" (UniqueName: \"kubernetes.io/projected/9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7-kube-api-access-qdvzh\") pod \"9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7\" (UID: \"9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7\") " Nov 03 22:59:06.709809 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:06.709700 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7-openshift-service-ca" pod "9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7" (UID: "9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:59:06.712156 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:06.712127 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7-maas-api-tls" pod "9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7" (UID: "9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:59:06.712507 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:06.712480 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7-metrics-certs" pod "9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7" (UID: "9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:59:06.712949 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:06.712924 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7-kube-api-access-qdvzh" pod "9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7" (UID: "9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7"). InnerVolumeSpecName "kube-api-access-qdvzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:59:06.815169 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:06.812543 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdvzh\" (UniqueName: \"kubernetes.io/projected/9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7-kube-api-access-qdvzh\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:06.815169 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:06.812582 5196 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7-metrics-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:06.815169 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:06.812599 5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7-maas-api-tls\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:06.815169 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:06.812611 5196 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7-openshift-service-ca\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:06.878931 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:06.873886 5196 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:59:06.878931 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:06.873980 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" podUID="9fde3ea1-b20e-4828-b8e6-5ffd537294f6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:59:07.238472 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:07.238386 5196 generic.go:356] "Generic (PLEG): container finished" podID="9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7" containerID="cb7e033ae66f070148ca45f9182b60397ad0ec9e33d90a8cc406cae6df0e87a9" exitCode=0 Nov 03 22:59:07.238472 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:07.238448 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w" Nov 03 22:59:07.238700 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:07.238468 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w" event={"ID":"9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7","Type":"ContainerDied","Data":"cb7e033ae66f070148ca45f9182b60397ad0ec9e33d90a8cc406cae6df0e87a9"} Nov 03 22:59:07.238700 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:07.238561 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w" event={"ID":"9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7","Type":"ContainerDied","Data":"a4ae9313252465c0cb71b20b0ddbd2e5b09973e11106b6d7266835500decbf15"} Nov 03 22:59:07.238700 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:07.238593 5196 scope.go:122] "RemoveContainer" containerID="cb7e033ae66f070148ca45f9182b60397ad0ec9e33d90a8cc406cae6df0e87a9" Nov 03 22:59:07.241055 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:07.241013 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" event={"ID":"c332f22b-f8b4-4b95-a2dc-bac68ef527d7","Type":"ContainerStarted","Data":"bf358276393c80b5a63eefe869e46ca83776e0e6652062f8c9b31f4967180499"} Nov 03 22:59:07.241196 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:07.241072 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" event={"ID":"c332f22b-f8b4-4b95-a2dc-bac68ef527d7","Type":"ContainerStarted","Data":"aa6bfc3d1af80f8b2cb8d1223fc92f3aacfc7af1e045706cff328a62954c77a2"} Nov 03 22:59:07.251029 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:07.251012 5196 scope.go:122] "RemoveContainer" containerID="cb7e033ae66f070148ca45f9182b60397ad0ec9e33d90a8cc406cae6df0e87a9" Nov 03 22:59:07.251368 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:59:07.251335 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb7e033ae66f070148ca45f9182b60397ad0ec9e33d90a8cc406cae6df0e87a9\": container with ID starting with cb7e033ae66f070148ca45f9182b60397ad0ec9e33d90a8cc406cae6df0e87a9 not found: ID does not exist" containerID="cb7e033ae66f070148ca45f9182b60397ad0ec9e33d90a8cc406cae6df0e87a9" Nov 03 22:59:07.251462 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:07.251379 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb7e033ae66f070148ca45f9182b60397ad0ec9e33d90a8cc406cae6df0e87a9"} err="failed to get container status \"cb7e033ae66f070148ca45f9182b60397ad0ec9e33d90a8cc406cae6df0e87a9\": rpc error: code = NotFound desc = could not find container \"cb7e033ae66f070148ca45f9182b60397ad0ec9e33d90a8cc406cae6df0e87a9\": container with ID starting with cb7e033ae66f070148ca45f9182b60397ad0ec9e33d90a8cc406cae6df0e87a9 not found: ID does not exist" Nov 03 22:59:07.422375 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:07.422326 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" podStartSLOduration=2.422311481 podStartE2EDuration="2.422311481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:59:07.418827013 +0000 UTC m=+2176.728386307" watchObservedRunningTime="2026-11-03 22:59:07.422311481 +0000 UTC m=+2176.731870831" Nov 03 22:59:07.534241 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:07.534145 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w"] Nov 03 22:59:07.553644 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:07.553608 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-aacab9-68cd458f74-dwf7w"] Nov 03 22:59:07.856766 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:07.856734 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 22:59:07.862917 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:07.862891 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 22:59:08.122232 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:08.122155 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:08.248334 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:08.248309 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 22:59:08.249193 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:08.249174 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 22:59:08.409278 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:08.409201 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:08.558439 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:08.558406 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:08.814527 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:08.814477 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:08.815009 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:08.814986 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7" path="/var/lib/kubelet/pods/9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7/volumes" Nov 03 22:59:09.050638 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.050613 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:09.066275 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.066256 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:59:09.140639 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.140612 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-workload-socket\") pod \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\" (UID: \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\") " Nov 03 22:59:09.141065 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.140654 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-workload-certs\") pod \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\" (UID: \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\") " Nov 03 22:59:09.141065 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.140688 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-credential-socket\") pod \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\" (UID: \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\") " Nov 03 22:59:09.141065 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.140729 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-istiod-ca-cert\") pod \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\" (UID: \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\") " Nov 03 22:59:09.141065 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.140761 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-kube-api-access-n4ml4\" (UniqueName: \"kubernetes.io/projected/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-kube-api-access-n4ml4\") pod \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\" (UID: \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\") " Nov 03 22:59:09.141065 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.140888 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-workload-socket" pod "9fde3ea1-b20e-4828-b8e6-5ffd537294f6" (UID: "9fde3ea1-b20e-4828-b8e6-5ffd537294f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:09.141065 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.140851 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-istio-data\") pod \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\" (UID: \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\") " Nov 03 22:59:09.141065 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.140969 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-workload-certs" pod "9fde3ea1-b20e-4828-b8e6-5ffd537294f6" (UID: "9fde3ea1-b20e-4828-b8e6-5ffd537294f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:09.141065 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.140997 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-istio-envoy\") pod \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\" (UID: \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\") " Nov 03 22:59:09.141728 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.141067 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-credential-socket" pod "9fde3ea1-b20e-4828-b8e6-5ffd537294f6" (UID: "9fde3ea1-b20e-4828-b8e6-5ffd537294f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:09.141728 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.141118 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-istio-podinfo\") pod \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\" (UID: \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\") " Nov 03 22:59:09.141728 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.141156 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-istio-token\" (UniqueName: \"kubernetes.io/projected/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-istio-token\") pod \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\" (UID: \"9fde3ea1-b20e-4828-b8e6-5ffd537294f6\") " Nov 03 22:59:09.141728 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.141162 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-istiod-ca-cert" pod "9fde3ea1-b20e-4828-b8e6-5ffd537294f6" (UID: "9fde3ea1-b20e-4828-b8e6-5ffd537294f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:59:09.141728 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.141161 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-istio-data" pod "9fde3ea1-b20e-4828-b8e6-5ffd537294f6" (UID: "9fde3ea1-b20e-4828-b8e6-5ffd537294f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:09.141728 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.141527 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:09.141728 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.141539 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:09.141728 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.141549 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:09.141728 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.141558 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:09.141728 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.141573 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:09.143343 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.143316 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-istio-podinfo" pod "9fde3ea1-b20e-4828-b8e6-5ffd537294f6" (UID: "9fde3ea1-b20e-4828-b8e6-5ffd537294f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:59:09.143343 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.143322 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-kube-api-access-n4ml4" pod "9fde3ea1-b20e-4828-b8e6-5ffd537294f6" (UID: "9fde3ea1-b20e-4828-b8e6-5ffd537294f6"). InnerVolumeSpecName "kube-api-access-n4ml4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:59:09.143759 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.143736 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-istio-envoy" pod "9fde3ea1-b20e-4828-b8e6-5ffd537294f6" (UID: "9fde3ea1-b20e-4828-b8e6-5ffd537294f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:09.143759 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.143753 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-istio-token" pod "9fde3ea1-b20e-4828-b8e6-5ffd537294f6" (UID: "9fde3ea1-b20e-4828-b8e6-5ffd537294f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:59:09.243306 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.243273 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:09.243306 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.243306 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:09.243493 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.243318 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:09.243493 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.243328 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4ml4\" (UniqueName: \"kubernetes.io/projected/9fde3ea1-b20e-4828-b8e6-5ffd537294f6-kube-api-access-n4ml4\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:09.256460 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.256433 5196 generic.go:356] "Generic (PLEG): container finished" podID="9fde3ea1-b20e-4828-b8e6-5ffd537294f6" containerID="7036bd1effaa4e0ac338ddb97f2e51a6bd0e6d9aacb1a4b14b59bcae9d216231" exitCode=0 Nov 03 22:59:09.256590 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.256489 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" Nov 03 22:59:09.256590 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.256521 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" event={"ID":"9fde3ea1-b20e-4828-b8e6-5ffd537294f6","Type":"ContainerDied","Data":"7036bd1effaa4e0ac338ddb97f2e51a6bd0e6d9aacb1a4b14b59bcae9d216231"} Nov 03 22:59:09.256590 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.256569 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj" event={"ID":"9fde3ea1-b20e-4828-b8e6-5ffd537294f6","Type":"ContainerDied","Data":"d554ffa146fd65630cefc6114f513f1129168bd4d116150149307b55d9b0f0e5"} Nov 03 22:59:09.256695 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.256590 5196 scope.go:122] "RemoveContainer" containerID="7036bd1effaa4e0ac338ddb97f2e51a6bd0e6d9aacb1a4b14b59bcae9d216231" Nov 03 22:59:09.267656 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.267640 5196 scope.go:122] "RemoveContainer" containerID="a48ffae424ef174332893917bac4904aad4fbab46d0da2576e210e54c8c3352f" Nov 03 22:59:09.277668 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.277650 5196 scope.go:122] "RemoveContainer" containerID="7036bd1effaa4e0ac338ddb97f2e51a6bd0e6d9aacb1a4b14b59bcae9d216231" Nov 03 22:59:09.277968 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:59:09.277949 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7036bd1effaa4e0ac338ddb97f2e51a6bd0e6d9aacb1a4b14b59bcae9d216231\": container with ID starting with 7036bd1effaa4e0ac338ddb97f2e51a6bd0e6d9aacb1a4b14b59bcae9d216231 not found: ID does not exist" containerID="7036bd1effaa4e0ac338ddb97f2e51a6bd0e6d9aacb1a4b14b59bcae9d216231" Nov 03 22:59:09.278035 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.277977 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7036bd1effaa4e0ac338ddb97f2e51a6bd0e6d9aacb1a4b14b59bcae9d216231"} err="failed to get container status \"7036bd1effaa4e0ac338ddb97f2e51a6bd0e6d9aacb1a4b14b59bcae9d216231\": rpc error: code = NotFound desc = could not find container \"7036bd1effaa4e0ac338ddb97f2e51a6bd0e6d9aacb1a4b14b59bcae9d216231\": container with ID starting with 7036bd1effaa4e0ac338ddb97f2e51a6bd0e6d9aacb1a4b14b59bcae9d216231 not found: ID does not exist" Nov 03 22:59:09.278035 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.277995 5196 scope.go:122] "RemoveContainer" containerID="a48ffae424ef174332893917bac4904aad4fbab46d0da2576e210e54c8c3352f" Nov 03 22:59:09.278264 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:59:09.278247 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a48ffae424ef174332893917bac4904aad4fbab46d0da2576e210e54c8c3352f\": container with ID starting with a48ffae424ef174332893917bac4904aad4fbab46d0da2576e210e54c8c3352f not found: ID does not exist" containerID="a48ffae424ef174332893917bac4904aad4fbab46d0da2576e210e54c8c3352f" Nov 03 22:59:09.278304 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.278271 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a48ffae424ef174332893917bac4904aad4fbab46d0da2576e210e54c8c3352f"} err="failed to get container status \"a48ffae424ef174332893917bac4904aad4fbab46d0da2576e210e54c8c3352f\": rpc error: code = NotFound desc = could not find container \"a48ffae424ef174332893917bac4904aad4fbab46d0da2576e210e54c8c3352f\": container with ID starting with a48ffae424ef174332893917bac4904aad4fbab46d0da2576e210e54c8c3352f not found: ID does not exist" Nov 03 22:59:09.428479 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.428452 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj"] Nov 03 22:59:09.430981 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:09.430957 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0406a8-openshift-default-689d85c855-l64hj"] Nov 03 22:59:10.224532 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:10.224503 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:10.390890 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:10.390822 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:10.543840 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:10.543761 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:10.823736 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:10.823651 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fde3ea1-b20e-4828-b8e6-5ffd537294f6" path="/var/lib/kubelet/pods/9fde3ea1-b20e-4828-b8e6-5ffd537294f6/volumes" Nov 03 22:59:10.959015 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:10.958984 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9"] Nov 03 22:59:10.959495 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:10.959481 5196 cpu_manager.go:414] "containerMap: removing container" podUID="9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7" containerName="maas-api" Nov 03 22:59:10.959536 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:10.959497 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7" containerName="maas-api" Nov 03 22:59:10.959536 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:10.959510 5196 cpu_manager.go:414] "containerMap: removing container" podUID="9fde3ea1-b20e-4828-b8e6-5ffd537294f6" containerName="istio-proxy" Nov 03 22:59:10.959536 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:10.959516 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fde3ea1-b20e-4828-b8e6-5ffd537294f6" containerName="istio-proxy" Nov 03 22:59:10.959536 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:10.959525 5196 cpu_manager.go:414] "containerMap: removing container" podUID="9fde3ea1-b20e-4828-b8e6-5ffd537294f6" containerName="istio-proxy" Nov 03 22:59:10.959536 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:10.959530 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fde3ea1-b20e-4828-b8e6-5ffd537294f6" containerName="istio-proxy" Nov 03 22:59:10.959688 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:10.959629 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fde3ea1-b20e-4828-b8e6-5ffd537294f6" containerName="istio-proxy" Nov 03 22:59:10.959688 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:10.959638 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fde3ea1-b20e-4828-b8e6-5ffd537294f6" containerName="istio-proxy" Nov 03 22:59:10.959688 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:10.959648 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e0c1f9c-4b12-42d2-bc9f-8caaffacb2b7" containerName="maas-api" Nov 03 22:59:10.964405 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:10.964385 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" Nov 03 22:59:11.063554 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:11.063520 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4deb89b6-4d18-4423-a47e-6ab4c477145d-istio-podinfo\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9\" (UID: \"4deb89b6-4d18-4423-a47e-6ab4c477145d\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" Nov 03 22:59:11.063554 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:11.063560 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4deb89b6-4d18-4423-a47e-6ab4c477145d-istio-data\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9\" (UID: \"4deb89b6-4d18-4423-a47e-6ab4c477145d\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" Nov 03 22:59:11.063836 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:11.063597 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4deb89b6-4d18-4423-a47e-6ab4c477145d-istio-token\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9\" (UID: \"4deb89b6-4d18-4423-a47e-6ab4c477145d\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" Nov 03 22:59:11.063836 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:11.063670 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4deb89b6-4d18-4423-a47e-6ab4c477145d-istio-envoy\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9\" (UID: \"4deb89b6-4d18-4423-a47e-6ab4c477145d\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" Nov 03 22:59:11.063836 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:11.063704 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4deb89b6-4d18-4423-a47e-6ab4c477145d-credential-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9\" (UID: \"4deb89b6-4d18-4423-a47e-6ab4c477145d\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" Nov 03 22:59:11.063836 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:11.063809 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4deb89b6-4d18-4423-a47e-6ab4c477145d-istiod-ca-cert\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9\" (UID: \"4deb89b6-4d18-4423-a47e-6ab4c477145d\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" Nov 03 22:59:11.064087 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:11.063967 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwbjg\" (UniqueName: \"kubernetes.io/projected/4deb89b6-4d18-4423-a47e-6ab4c477145d-kube-api-access-dwbjg\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9\" (UID: \"4deb89b6-4d18-4423-a47e-6ab4c477145d\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" Nov 03 22:59:11.064149 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:11.064088 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4deb89b6-4d18-4423-a47e-6ab4c477145d-workload-certs\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9\" (UID: \"4deb89b6-4d18-4423-a47e-6ab4c477145d\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" Nov 03 22:59:11.064149 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:11.064119 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4deb89b6-4d18-4423-a47e-6ab4c477145d-workload-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9\" (UID: \"4deb89b6-4d18-4423-a47e-6ab4c477145d\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" Nov 03 22:59:11.165218 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:11.165181 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4deb89b6-4d18-4423-a47e-6ab4c477145d-workload-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9\" (UID: \"4deb89b6-4d18-4423-a47e-6ab4c477145d\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" Nov 03 22:59:11.165407 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:11.165257 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4deb89b6-4d18-4423-a47e-6ab4c477145d-istio-podinfo\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9\" (UID: \"4deb89b6-4d18-4423-a47e-6ab4c477145d\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" Nov 03 22:59:11.165407 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:11.165276 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4deb89b6-4d18-4423-a47e-6ab4c477145d-istio-data\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9\" (UID: \"4deb89b6-4d18-4423-a47e-6ab4c477145d\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" Nov 03 22:59:11.165407 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:11.165302 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4deb89b6-4d18-4423-a47e-6ab4c477145d-istio-token\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9\" (UID: \"4deb89b6-4d18-4423-a47e-6ab4c477145d\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" Nov 03 22:59:11.165407 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:11.165320 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4deb89b6-4d18-4423-a47e-6ab4c477145d-istio-envoy\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9\" (UID: \"4deb89b6-4d18-4423-a47e-6ab4c477145d\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" Nov 03 22:59:11.165407 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:11.165350 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4deb89b6-4d18-4423-a47e-6ab4c477145d-credential-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9\" (UID: \"4deb89b6-4d18-4423-a47e-6ab4c477145d\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" Nov 03 22:59:11.165696 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:11.165412 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4deb89b6-4d18-4423-a47e-6ab4c477145d-istiod-ca-cert\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9\" (UID: \"4deb89b6-4d18-4423-a47e-6ab4c477145d\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" Nov 03 22:59:11.165696 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:11.165490 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwbjg\" (UniqueName: \"kubernetes.io/projected/4deb89b6-4d18-4423-a47e-6ab4c477145d-kube-api-access-dwbjg\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9\" (UID: \"4deb89b6-4d18-4423-a47e-6ab4c477145d\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" Nov 03 22:59:11.165696 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:11.165585 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4deb89b6-4d18-4423-a47e-6ab4c477145d-workload-certs\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9\" (UID: \"4deb89b6-4d18-4423-a47e-6ab4c477145d\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" Nov 03 22:59:11.165881 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:11.165699 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4deb89b6-4d18-4423-a47e-6ab4c477145d-istio-data\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9\" (UID: \"4deb89b6-4d18-4423-a47e-6ab4c477145d\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" Nov 03 22:59:11.165881 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:11.165728 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4deb89b6-4d18-4423-a47e-6ab4c477145d-workload-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9\" (UID: \"4deb89b6-4d18-4423-a47e-6ab4c477145d\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" Nov 03 22:59:11.165881 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:11.165789 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4deb89b6-4d18-4423-a47e-6ab4c477145d-credential-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9\" (UID: \"4deb89b6-4d18-4423-a47e-6ab4c477145d\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" Nov 03 22:59:11.166056 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:11.166015 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4deb89b6-4d18-4423-a47e-6ab4c477145d-workload-certs\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9\" (UID: \"4deb89b6-4d18-4423-a47e-6ab4c477145d\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" Nov 03 22:59:11.166230 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:11.166210 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4deb89b6-4d18-4423-a47e-6ab4c477145d-istiod-ca-cert\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9\" (UID: \"4deb89b6-4d18-4423-a47e-6ab4c477145d\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" Nov 03 22:59:11.168225 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:11.168201 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4deb89b6-4d18-4423-a47e-6ab4c477145d-istio-envoy\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9\" (UID: \"4deb89b6-4d18-4423-a47e-6ab4c477145d\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" Nov 03 22:59:11.168329 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:11.168251 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4deb89b6-4d18-4423-a47e-6ab4c477145d-istio-podinfo\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9\" (UID: \"4deb89b6-4d18-4423-a47e-6ab4c477145d\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" Nov 03 22:59:11.179689 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:11.179657 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9"] Nov 03 22:59:11.228135 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:11.228103 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4deb89b6-4d18-4423-a47e-6ab4c477145d-istio-token\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9\" (UID: \"4deb89b6-4d18-4423-a47e-6ab4c477145d\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" Nov 03 22:59:11.228485 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:11.228214 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwbjg\" (UniqueName: \"kubernetes.io/projected/4deb89b6-4d18-4423-a47e-6ab4c477145d-kube-api-access-dwbjg\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9\" (UID: \"4deb89b6-4d18-4423-a47e-6ab4c477145d\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" Nov 03 22:59:11.277221 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:11.277196 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" Nov 03 22:59:11.750806 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:11.750776 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9"] Nov 03 22:59:11.751695 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:59:11.751669 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4deb89b6_4d18_4423_a47e_6ab4c477145d.slice/crio-892087a140e27cff7f952d1e36d565cd8e09d514e7d92ca7d8b77da8c0b4b8d7 WatchSource:0}: Error finding container 892087a140e27cff7f952d1e36d565cd8e09d514e7d92ca7d8b77da8c0b4b8d7: Status 404 returned error can't find the container with id 892087a140e27cff7f952d1e36d565cd8e09d514e7d92ca7d8b77da8c0b4b8d7 Nov 03 22:59:11.754488 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:11.754451 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:59:11.754589 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:11.754530 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:59:11.754589 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:11.754560 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:59:12.279363 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:12.279329 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" event={"ID":"4deb89b6-4d18-4423-a47e-6ab4c477145d","Type":"ContainerStarted","Data":"660620dd08ad7d07fe7b886864e14355268aaaed7f0c36db17beb5fdc45280c3"} Nov 03 22:59:12.279792 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:12.279383 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" event={"ID":"4deb89b6-4d18-4423-a47e-6ab4c477145d","Type":"ContainerStarted","Data":"892087a140e27cff7f952d1e36d565cd8e09d514e7d92ca7d8b77da8c0b4b8d7"} Nov 03 22:59:13.277797 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:13.277763 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" Nov 03 22:59:13.279326 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:13.279300 5196 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 03 22:59:13.279450 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:13.279349 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" podUID="4deb89b6-4d18-4423-a47e-6ab4c477145d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 03 22:59:13.688782 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:13.688748 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:14.249366 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:14.249336 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:14.277793 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:14.277762 5196 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 03 22:59:14.277981 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:14.277814 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" podUID="4deb89b6-4d18-4423-a47e-6ab4c477145d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 03 22:59:15.014643 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:15.014596 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:15.224416 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:15.223234 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:15.277702 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:15.277625 5196 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 03 22:59:15.277847 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:15.277696 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" podUID="4deb89b6-4d18-4423-a47e-6ab4c477145d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 03 22:59:15.784906 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:15.784814 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" podStartSLOduration=5.784793223 podStartE2EDuration="5.784793223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:59:12.418226864 +0000 UTC m=+2181.727786168" watchObservedRunningTime="2026-11-03 22:59:15.784793223 +0000 UTC m=+2185.094352519" Nov 03 22:59:15.786536 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:15.786512 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ea5c8f-7db7d5b55f-qnbc8"] Nov 03 22:59:15.824010 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:15.823966 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ea5c8f-7db7d5b55f-qnbc8" Nov 03 22:59:15.848222 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:15.848197 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ea5c8f-7db7d5b55f-qnbc8"] Nov 03 22:59:15.852007 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:15.851979 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8gk9\" (UniqueName: \"kubernetes.io/projected/286e0af7-6c70-47b8-924c-5f584a4ba2f8-kube-api-access-t8gk9\") pod \"payload-pre-processing-e2e-api-a-ea5c8f-7db7d5b55f-qnbc8\" (UID: \"286e0af7-6c70-47b8-924c-5f584a4ba2f8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ea5c8f-7db7d5b55f-qnbc8" Nov 03 22:59:15.852202 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:15.852174 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/286e0af7-6c70-47b8-924c-5f584a4ba2f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ea5c8f-7db7d5b55f-qnbc8\" (UID: \"286e0af7-6c70-47b8-924c-5f584a4ba2f8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ea5c8f-7db7d5b55f-qnbc8" Nov 03 22:59:15.909890 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:15.902078 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ea5c8f\"" Nov 03 22:59:15.930764 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:15.930734 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ea5c8f-dockercfg-gzxdz\"" Nov 03 22:59:15.953407 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:15.953314 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/286e0af7-6c70-47b8-924c-5f584a4ba2f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ea5c8f-7db7d5b55f-qnbc8\" (UID: \"286e0af7-6c70-47b8-924c-5f584a4ba2f8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ea5c8f-7db7d5b55f-qnbc8" Nov 03 22:59:15.953624 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:15.953572 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8gk9\" (UniqueName: \"kubernetes.io/projected/286e0af7-6c70-47b8-924c-5f584a4ba2f8-kube-api-access-t8gk9\") pod \"payload-pre-processing-e2e-api-a-ea5c8f-7db7d5b55f-qnbc8\" (UID: \"286e0af7-6c70-47b8-924c-5f584a4ba2f8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ea5c8f-7db7d5b55f-qnbc8" Nov 03 22:59:15.954118 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:15.954099 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/286e0af7-6c70-47b8-924c-5f584a4ba2f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ea5c8f-7db7d5b55f-qnbc8\" (UID: \"286e0af7-6c70-47b8-924c-5f584a4ba2f8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ea5c8f-7db7d5b55f-qnbc8" Nov 03 22:59:16.017738 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:16.017707 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8gk9\" (UniqueName: \"kubernetes.io/projected/286e0af7-6c70-47b8-924c-5f584a4ba2f8-kube-api-access-t8gk9\") pod \"payload-pre-processing-e2e-api-a-ea5c8f-7db7d5b55f-qnbc8\" (UID: \"286e0af7-6c70-47b8-924c-5f584a4ba2f8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ea5c8f-7db7d5b55f-qnbc8" Nov 03 22:59:16.140753 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:16.140723 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ea5c8f-7db7d5b55f-qnbc8" Nov 03 22:59:16.278414 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:16.278377 5196 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 03 22:59:16.278553 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:16.278441 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" podUID="4deb89b6-4d18-4423-a47e-6ab4c477145d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 03 22:59:16.316932 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:16.315493 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:16.462828 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:16.462737 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:16.544597 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:16.544567 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ea5c8f-7db7d5b55f-qnbc8"] Nov 03 22:59:16.545968 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:59:16.545941 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod286e0af7_6c70_47b8_924c_5f584a4ba2f8.slice/crio-27d0fc94ca9cac4d78c843b5360cf3de87590082b9c3f9157a61ab8688722c8a WatchSource:0}: Error finding container 27d0fc94ca9cac4d78c843b5360cf3de87590082b9c3f9157a61ab8688722c8a: Status 404 returned error can't find the container with id 27d0fc94ca9cac4d78c843b5360cf3de87590082b9c3f9157a61ab8688722c8a Nov 03 22:59:16.659557 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:16.659525 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2"] Nov 03 22:59:16.659786 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:16.659762 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" podUID="9a97695c-47ae-48f9-ad2e-f97625e48234" containerName="istio-proxy" containerID="cri-o://3ad56f90432d3b07d261befb78877326ffefe902a254d307a6334578f5f76bfa" gracePeriod=30 Nov 03 22:59:16.688768 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:16.688741 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:17.278495 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:17.278458 5196 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 03 22:59:17.278905 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:17.278512 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" podUID="4deb89b6-4d18-4423-a47e-6ab4c477145d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 03 22:59:17.344187 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:17.344156 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ea5c8f-7db7d5b55f-qnbc8" event={"ID":"286e0af7-6c70-47b8-924c-5f584a4ba2f8","Type":"ContainerStarted","Data":"27d0fc94ca9cac4d78c843b5360cf3de87590082b9c3f9157a61ab8688722c8a"} Nov 03 22:59:17.517878 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:17.517798 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:18.033425 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:18.033385 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz"] Nov 03 22:59:18.037539 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:18.037513 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 22:59:18.126882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:18.093371 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-1048fc-openshift-default-dockercfg-chxfw\"" Nov 03 22:59:18.160946 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:18.156672 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:18.195059 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:18.188652 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d375a3f-2f84-4495-a683-42a98477ff52-istio-podinfo\") pod \"e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz\" (UID: \"0d375a3f-2f84-4495-a683-42a98477ff52\") " pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 22:59:18.195059 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:18.188729 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d375a3f-2f84-4495-a683-42a98477ff52-workload-socket\") pod \"e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz\" (UID: \"0d375a3f-2f84-4495-a683-42a98477ff52\") " pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 22:59:18.195059 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:18.188882 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwg2l\" (UniqueName: \"kubernetes.io/projected/0d375a3f-2f84-4495-a683-42a98477ff52-kube-api-access-qwg2l\") pod \"e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz\" (UID: \"0d375a3f-2f84-4495-a683-42a98477ff52\") " pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 22:59:18.195059 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:18.188913 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d375a3f-2f84-4495-a683-42a98477ff52-istio-data\") pod \"e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz\" (UID: \"0d375a3f-2f84-4495-a683-42a98477ff52\") " pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 22:59:18.195059 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:18.188940 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d375a3f-2f84-4495-a683-42a98477ff52-istiod-ca-cert\") pod \"e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz\" (UID: \"0d375a3f-2f84-4495-a683-42a98477ff52\") " pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 22:59:18.195059 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:18.188984 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d375a3f-2f84-4495-a683-42a98477ff52-workload-certs\") pod \"e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz\" (UID: \"0d375a3f-2f84-4495-a683-42a98477ff52\") " pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 22:59:18.195059 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:18.189064 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d375a3f-2f84-4495-a683-42a98477ff52-istio-envoy\") pod \"e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz\" (UID: \"0d375a3f-2f84-4495-a683-42a98477ff52\") " pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 22:59:18.195059 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:18.189140 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d375a3f-2f84-4495-a683-42a98477ff52-istio-token\") pod \"e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz\" (UID: \"0d375a3f-2f84-4495-a683-42a98477ff52\") " pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 22:59:18.195059 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:18.189182 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d375a3f-2f84-4495-a683-42a98477ff52-credential-socket\") pod \"e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz\" (UID: \"0d375a3f-2f84-4495-a683-42a98477ff52\") " pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 22:59:18.239191 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:18.239145 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz"] Nov 03 22:59:18.295039 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:18.294727 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d375a3f-2f84-4495-a683-42a98477ff52-istio-token\") pod \"e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz\" (UID: \"0d375a3f-2f84-4495-a683-42a98477ff52\") " pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 22:59:18.295039 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:18.294790 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d375a3f-2f84-4495-a683-42a98477ff52-credential-socket\") pod \"e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz\" (UID: \"0d375a3f-2f84-4495-a683-42a98477ff52\") " pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 22:59:18.295039 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:18.294843 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d375a3f-2f84-4495-a683-42a98477ff52-istio-podinfo\") pod \"e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz\" (UID: \"0d375a3f-2f84-4495-a683-42a98477ff52\") " pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 22:59:18.295039 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:18.294925 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d375a3f-2f84-4495-a683-42a98477ff52-workload-socket\") pod \"e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz\" (UID: \"0d375a3f-2f84-4495-a683-42a98477ff52\") " pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 22:59:18.295678 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:18.295050 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwg2l\" (UniqueName: \"kubernetes.io/projected/0d375a3f-2f84-4495-a683-42a98477ff52-kube-api-access-qwg2l\") pod \"e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz\" (UID: \"0d375a3f-2f84-4495-a683-42a98477ff52\") " pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 22:59:18.295678 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:18.295075 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d375a3f-2f84-4495-a683-42a98477ff52-istio-data\") pod \"e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz\" (UID: \"0d375a3f-2f84-4495-a683-42a98477ff52\") " pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 22:59:18.295678 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:18.295105 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d375a3f-2f84-4495-a683-42a98477ff52-istiod-ca-cert\") pod \"e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz\" (UID: \"0d375a3f-2f84-4495-a683-42a98477ff52\") " pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 22:59:18.295678 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:18.295147 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d375a3f-2f84-4495-a683-42a98477ff52-workload-certs\") pod \"e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz\" (UID: \"0d375a3f-2f84-4495-a683-42a98477ff52\") " pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 22:59:18.295678 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:18.295232 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d375a3f-2f84-4495-a683-42a98477ff52-istio-envoy\") pod \"e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz\" (UID: \"0d375a3f-2f84-4495-a683-42a98477ff52\") " pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 22:59:18.300288 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:18.299063 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d375a3f-2f84-4495-a683-42a98477ff52-istiod-ca-cert\") pod \"e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz\" (UID: \"0d375a3f-2f84-4495-a683-42a98477ff52\") " pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 22:59:18.300997 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:18.300712 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d375a3f-2f84-4495-a683-42a98477ff52-workload-certs\") pod \"e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz\" (UID: \"0d375a3f-2f84-4495-a683-42a98477ff52\") " pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 22:59:18.300997 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:18.300952 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d375a3f-2f84-4495-a683-42a98477ff52-istio-data\") pod \"e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz\" (UID: \"0d375a3f-2f84-4495-a683-42a98477ff52\") " pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 22:59:18.301484 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:18.301439 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d375a3f-2f84-4495-a683-42a98477ff52-workload-socket\") pod \"e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz\" (UID: \"0d375a3f-2f84-4495-a683-42a98477ff52\") " pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 22:59:18.301890 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:18.301851 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d375a3f-2f84-4495-a683-42a98477ff52-credential-socket\") pod \"e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz\" (UID: \"0d375a3f-2f84-4495-a683-42a98477ff52\") " pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 22:59:18.302290 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:18.302268 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d375a3f-2f84-4495-a683-42a98477ff52-istio-envoy\") pod \"e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz\" (UID: \"0d375a3f-2f84-4495-a683-42a98477ff52\") " pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 22:59:18.307926 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:18.307875 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d375a3f-2f84-4495-a683-42a98477ff52-istio-podinfo\") pod \"e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz\" (UID: \"0d375a3f-2f84-4495-a683-42a98477ff52\") " pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 22:59:18.373262 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:18.364398 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d375a3f-2f84-4495-a683-42a98477ff52-istio-token\") pod \"e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz\" (UID: \"0d375a3f-2f84-4495-a683-42a98477ff52\") " pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 22:59:18.373262 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:18.373176 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwg2l\" (UniqueName: \"kubernetes.io/projected/0d375a3f-2f84-4495-a683-42a98477ff52-kube-api-access-qwg2l\") pod \"e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz\" (UID: \"0d375a3f-2f84-4495-a683-42a98477ff52\") " pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 22:59:18.527488 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:18.527419 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" Nov 03 22:59:18.528501 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:18.528468 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" Nov 03 22:59:18.529448 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:18.529428 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" Nov 03 22:59:18.623485 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:18.623446 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:18.657098 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:18.657063 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 22:59:18.941999 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:18.941899 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:19.127095 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:19.127061 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz"] Nov 03 22:59:19.131515 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:59:19.131487 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d375a3f_2f84_4495_a683_42a98477ff52.slice/crio-3414d0eee6b63461b1168ca371619e6dbf37cee6b4d7e8f92e46a06f08233698 WatchSource:0}: Error finding container 3414d0eee6b63461b1168ca371619e6dbf37cee6b4d7e8f92e46a06f08233698: Status 404 returned error can't find the container with id 3414d0eee6b63461b1168ca371619e6dbf37cee6b4d7e8f92e46a06f08233698 Nov 03 22:59:19.135458 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:19.135417 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:59:19.135569 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:19.135515 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:59:19.135611 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:19.135566 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:59:19.366994 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:19.366924 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" event={"ID":"0d375a3f-2f84-4495-a683-42a98477ff52","Type":"ContainerStarted","Data":"162270799d34e633401238397fd27799cf07fc10e603e751e08527bd0178b50a"} Nov 03 22:59:19.366994 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:19.366989 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" event={"ID":"0d375a3f-2f84-4495-a683-42a98477ff52","Type":"ContainerStarted","Data":"3414d0eee6b63461b1168ca371619e6dbf37cee6b4d7e8f92e46a06f08233698"} Nov 03 22:59:19.491328 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:19.491214 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" podStartSLOduration=2.49119961 podStartE2EDuration="2.49119961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:59:19.488389396 +0000 UTC m=+2188.797948735" watchObservedRunningTime="2026-11-03 22:59:19.49119961 +0000 UTC m=+2188.800758903" Nov 03 22:59:19.657716 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:19.657674 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 22:59:20.374506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:20.374468 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ea5c8f-7db7d5b55f-qnbc8" event={"ID":"286e0af7-6c70-47b8-924c-5f584a4ba2f8","Type":"ContainerStarted","Data":"bca8602399801d660bfd7ee4d4b6359b9db494795fa74b77a76b65a4350cacb5"} Nov 03 22:59:20.495025 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:20.494967 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ea5c8f-7db7d5b55f-qnbc8" podStartSLOduration=2.006043273 podStartE2EDuration="5.494953322s" totalImagesPullingTime="3.488910049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:59:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:59:20.492022149 +0000 UTC m=+2189.801581465" watchObservedRunningTime="2026-11-03 22:59:20.494953322 +0000 UTC m=+2189.804512615" Nov 03 22:59:20.663518 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:20.663437 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 22:59:21.383269 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:21.383238 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ea5c8f-7db7d5b55f-qnbc8" Nov 03 22:59:21.383731 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:21.383292 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 22:59:21.384261 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:21.384245 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 22:59:21.941390 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:21.941332 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:59:22.044715 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.044675 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a97695c-47ae-48f9-ad2e-f97625e48234-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a97695c-47ae-48f9-ad2e-f97625e48234-credential-socket\") pod \"9a97695c-47ae-48f9-ad2e-f97625e48234\" (UID: \"9a97695c-47ae-48f9-ad2e-f97625e48234\") " Nov 03 22:59:22.044907 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.044771 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a97695c-47ae-48f9-ad2e-f97625e48234-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a97695c-47ae-48f9-ad2e-f97625e48234-istio-podinfo\") pod \"9a97695c-47ae-48f9-ad2e-f97625e48234\" (UID: \"9a97695c-47ae-48f9-ad2e-f97625e48234\") " Nov 03 22:59:22.044907 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.044819 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a97695c-47ae-48f9-ad2e-f97625e48234-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a97695c-47ae-48f9-ad2e-f97625e48234-istiod-ca-cert\") pod \"9a97695c-47ae-48f9-ad2e-f97625e48234\" (UID: \"9a97695c-47ae-48f9-ad2e-f97625e48234\") " Nov 03 22:59:22.044907 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.044893 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a97695c-47ae-48f9-ad2e-f97625e48234-kube-api-access-cmxhl\" (UniqueName: \"kubernetes.io/projected/9a97695c-47ae-48f9-ad2e-f97625e48234-kube-api-access-cmxhl\") pod \"9a97695c-47ae-48f9-ad2e-f97625e48234\" (UID: \"9a97695c-47ae-48f9-ad2e-f97625e48234\") " Nov 03 22:59:22.045091 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.044974 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a97695c-47ae-48f9-ad2e-f97625e48234-istio-token\" (UniqueName: \"kubernetes.io/projected/9a97695c-47ae-48f9-ad2e-f97625e48234-istio-token\") pod \"9a97695c-47ae-48f9-ad2e-f97625e48234\" (UID: \"9a97695c-47ae-48f9-ad2e-f97625e48234\") " Nov 03 22:59:22.045091 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.045003 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a97695c-47ae-48f9-ad2e-f97625e48234-credential-socket" pod "9a97695c-47ae-48f9-ad2e-f97625e48234" (UID: "9a97695c-47ae-48f9-ad2e-f97625e48234"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:22.045091 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.045050 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a97695c-47ae-48f9-ad2e-f97625e48234-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a97695c-47ae-48f9-ad2e-f97625e48234-workload-socket\") pod \"9a97695c-47ae-48f9-ad2e-f97625e48234\" (UID: \"9a97695c-47ae-48f9-ad2e-f97625e48234\") " Nov 03 22:59:22.045256 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.045097 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a97695c-47ae-48f9-ad2e-f97625e48234-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a97695c-47ae-48f9-ad2e-f97625e48234-istio-envoy\") pod \"9a97695c-47ae-48f9-ad2e-f97625e48234\" (UID: \"9a97695c-47ae-48f9-ad2e-f97625e48234\") " Nov 03 22:59:22.045256 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.045208 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a97695c-47ae-48f9-ad2e-f97625e48234-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a97695c-47ae-48f9-ad2e-f97625e48234-istio-data\") pod \"9a97695c-47ae-48f9-ad2e-f97625e48234\" (UID: \"9a97695c-47ae-48f9-ad2e-f97625e48234\") " Nov 03 22:59:22.045256 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.045251 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a97695c-47ae-48f9-ad2e-f97625e48234-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a97695c-47ae-48f9-ad2e-f97625e48234-workload-certs\") pod \"9a97695c-47ae-48f9-ad2e-f97625e48234\" (UID: \"9a97695c-47ae-48f9-ad2e-f97625e48234\") " Nov 03 22:59:22.045410 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.045262 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a97695c-47ae-48f9-ad2e-f97625e48234-istiod-ca-cert" pod "9a97695c-47ae-48f9-ad2e-f97625e48234" (UID: "9a97695c-47ae-48f9-ad2e-f97625e48234"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:59:22.045463 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.045415 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a97695c-47ae-48f9-ad2e-f97625e48234-workload-socket" pod "9a97695c-47ae-48f9-ad2e-f97625e48234" (UID: "9a97695c-47ae-48f9-ad2e-f97625e48234"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:22.045589 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.045567 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a97695c-47ae-48f9-ad2e-f97625e48234-istio-data" pod "9a97695c-47ae-48f9-ad2e-f97625e48234" (UID: "9a97695c-47ae-48f9-ad2e-f97625e48234"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:22.045720 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.045699 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a97695c-47ae-48f9-ad2e-f97625e48234-workload-certs" pod "9a97695c-47ae-48f9-ad2e-f97625e48234" (UID: "9a97695c-47ae-48f9-ad2e-f97625e48234"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:22.045918 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.045901 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a97695c-47ae-48f9-ad2e-f97625e48234-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:22.046004 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.045928 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a97695c-47ae-48f9-ad2e-f97625e48234-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:22.046004 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.045944 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a97695c-47ae-48f9-ad2e-f97625e48234-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:22.046004 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.045958 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a97695c-47ae-48f9-ad2e-f97625e48234-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:22.046004 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.045986 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a97695c-47ae-48f9-ad2e-f97625e48234-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:22.047941 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.047911 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a97695c-47ae-48f9-ad2e-f97625e48234-istio-podinfo" pod "9a97695c-47ae-48f9-ad2e-f97625e48234" (UID: "9a97695c-47ae-48f9-ad2e-f97625e48234"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:59:22.048122 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.048103 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a97695c-47ae-48f9-ad2e-f97625e48234-istio-token" pod "9a97695c-47ae-48f9-ad2e-f97625e48234" (UID: "9a97695c-47ae-48f9-ad2e-f97625e48234"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:59:22.048199 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.048184 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a97695c-47ae-48f9-ad2e-f97625e48234-kube-api-access-cmxhl" pod "9a97695c-47ae-48f9-ad2e-f97625e48234" (UID: "9a97695c-47ae-48f9-ad2e-f97625e48234"). InnerVolumeSpecName "kube-api-access-cmxhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:59:22.048525 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.048511 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a97695c-47ae-48f9-ad2e-f97625e48234-istio-envoy" pod "9a97695c-47ae-48f9-ad2e-f97625e48234" (UID: "9a97695c-47ae-48f9-ad2e-f97625e48234"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:22.147000 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.146964 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a97695c-47ae-48f9-ad2e-f97625e48234-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:22.147000 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.147001 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmxhl\" (UniqueName: \"kubernetes.io/projected/9a97695c-47ae-48f9-ad2e-f97625e48234-kube-api-access-cmxhl\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:22.147220 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.147017 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a97695c-47ae-48f9-ad2e-f97625e48234-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:22.147220 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.147031 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a97695c-47ae-48f9-ad2e-f97625e48234-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:22.353733 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.353693 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:22.392405 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.392370 5196 generic.go:356] "Generic (PLEG): container finished" podID="9a97695c-47ae-48f9-ad2e-f97625e48234" containerID="3ad56f90432d3b07d261befb78877326ffefe902a254d307a6334578f5f76bfa" exitCode=0 Nov 03 22:59:22.392947 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.392426 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" Nov 03 22:59:22.392947 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.392452 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" event={"ID":"9a97695c-47ae-48f9-ad2e-f97625e48234","Type":"ContainerDied","Data":"3ad56f90432d3b07d261befb78877326ffefe902a254d307a6334578f5f76bfa"} Nov 03 22:59:22.392947 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.392549 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2" event={"ID":"9a97695c-47ae-48f9-ad2e-f97625e48234","Type":"ContainerDied","Data":"4a7df835939df77635801bd9626b502a0846d49dc4efa83e27ec78c83fee5355"} Nov 03 22:59:22.392947 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.392583 5196 scope.go:122] "RemoveContainer" containerID="3ad56f90432d3b07d261befb78877326ffefe902a254d307a6334578f5f76bfa" Nov 03 22:59:22.403319 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.403300 5196 scope.go:122] "RemoveContainer" containerID="6b274616ced7135811db4386afb9aa23676ae1e48aa028c422227d91e16644f8" Nov 03 22:59:22.414952 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.414928 5196 scope.go:122] "RemoveContainer" containerID="3ad56f90432d3b07d261befb78877326ffefe902a254d307a6334578f5f76bfa" Nov 03 22:59:22.415256 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:59:22.415236 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ad56f90432d3b07d261befb78877326ffefe902a254d307a6334578f5f76bfa\": container with ID starting with 3ad56f90432d3b07d261befb78877326ffefe902a254d307a6334578f5f76bfa not found: ID does not exist" containerID="3ad56f90432d3b07d261befb78877326ffefe902a254d307a6334578f5f76bfa" Nov 03 22:59:22.415305 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.415267 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ad56f90432d3b07d261befb78877326ffefe902a254d307a6334578f5f76bfa"} err="failed to get container status \"3ad56f90432d3b07d261befb78877326ffefe902a254d307a6334578f5f76bfa\": rpc error: code = NotFound desc = could not find container \"3ad56f90432d3b07d261befb78877326ffefe902a254d307a6334578f5f76bfa\": container with ID starting with 3ad56f90432d3b07d261befb78877326ffefe902a254d307a6334578f5f76bfa not found: ID does not exist" Nov 03 22:59:22.415305 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.415285 5196 scope.go:122] "RemoveContainer" containerID="6b274616ced7135811db4386afb9aa23676ae1e48aa028c422227d91e16644f8" Nov 03 22:59:22.415546 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:59:22.415529 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b274616ced7135811db4386afb9aa23676ae1e48aa028c422227d91e16644f8\": container with ID starting with 6b274616ced7135811db4386afb9aa23676ae1e48aa028c422227d91e16644f8 not found: ID does not exist" containerID="6b274616ced7135811db4386afb9aa23676ae1e48aa028c422227d91e16644f8" Nov 03 22:59:22.415605 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.415556 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b274616ced7135811db4386afb9aa23676ae1e48aa028c422227d91e16644f8"} err="failed to get container status \"6b274616ced7135811db4386afb9aa23676ae1e48aa028c422227d91e16644f8\": rpc error: code = NotFound desc = could not find container \"6b274616ced7135811db4386afb9aa23676ae1e48aa028c422227d91e16644f8\": container with ID starting with 6b274616ced7135811db4386afb9aa23676ae1e48aa028c422227d91e16644f8 not found: ID does not exist" Nov 03 22:59:22.520058 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.520025 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:22.561154 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.561122 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2"] Nov 03 22:59:22.611190 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.611119 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-aacab9-openshift-default-7c7c84964c-lc9v2"] Nov 03 22:59:22.666161 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.666131 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:22.816474 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:22.816440 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a97695c-47ae-48f9-ad2e-f97625e48234" path="/var/lib/kubelet/pods/9a97695c-47ae-48f9-ad2e-f97625e48234/volumes" Nov 03 22:59:23.999374 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:23.999345 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:25.919349 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:59:25.919314 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:59:26.069764 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.069729 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6"] Nov 03 22:59:26.070306 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.070292 5196 cpu_manager.go:414] "containerMap: removing container" podUID="9a97695c-47ae-48f9-ad2e-f97625e48234" containerName="istio-proxy" Nov 03 22:59:26.070352 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.070309 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a97695c-47ae-48f9-ad2e-f97625e48234" containerName="istio-proxy" Nov 03 22:59:26.070352 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.070323 5196 cpu_manager.go:414] "containerMap: removing container" podUID="9a97695c-47ae-48f9-ad2e-f97625e48234" containerName="istio-proxy" Nov 03 22:59:26.070352 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.070330 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a97695c-47ae-48f9-ad2e-f97625e48234" containerName="istio-proxy" Nov 03 22:59:26.070453 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.070444 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a97695c-47ae-48f9-ad2e-f97625e48234" containerName="istio-proxy" Nov 03 22:59:26.070488 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.070453 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a97695c-47ae-48f9-ad2e-f97625e48234" containerName="istio-proxy" Nov 03 22:59:26.075457 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.075440 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" Nov 03 22:59:26.185307 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.185233 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6"] Nov 03 22:59:26.206077 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.205247 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6166d64-362f-4ada-b2fd-ecf68bc023a7-credential-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6\" (UID: \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" Nov 03 22:59:26.206077 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.205318 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6166d64-362f-4ada-b2fd-ecf68bc023a7-workload-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6\" (UID: \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" Nov 03 22:59:26.206077 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.205352 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6166d64-362f-4ada-b2fd-ecf68bc023a7-istio-token\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6\" (UID: \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" Nov 03 22:59:26.206077 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.205415 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6166d64-362f-4ada-b2fd-ecf68bc023a7-istio-podinfo\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6\" (UID: \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" Nov 03 22:59:26.206077 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.205452 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6166d64-362f-4ada-b2fd-ecf68bc023a7-istio-envoy\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6\" (UID: \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" Nov 03 22:59:26.206077 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.205497 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t29bc\" (UniqueName: \"kubernetes.io/projected/a6166d64-362f-4ada-b2fd-ecf68bc023a7-kube-api-access-t29bc\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6\" (UID: \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" Nov 03 22:59:26.206077 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.205580 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6166d64-362f-4ada-b2fd-ecf68bc023a7-istiod-ca-cert\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6\" (UID: \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" Nov 03 22:59:26.206077 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.205686 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6166d64-362f-4ada-b2fd-ecf68bc023a7-workload-certs\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6\" (UID: \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" Nov 03 22:59:26.206077 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.205757 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6166d64-362f-4ada-b2fd-ecf68bc023a7-istio-data\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6\" (UID: \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" Nov 03 22:59:26.306965 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.306930 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6166d64-362f-4ada-b2fd-ecf68bc023a7-istio-podinfo\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6\" (UID: \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" Nov 03 22:59:26.307128 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.306975 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6166d64-362f-4ada-b2fd-ecf68bc023a7-istio-envoy\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6\" (UID: \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" Nov 03 22:59:26.307128 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.307009 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t29bc\" (UniqueName: \"kubernetes.io/projected/a6166d64-362f-4ada-b2fd-ecf68bc023a7-kube-api-access-t29bc\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6\" (UID: \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" Nov 03 22:59:26.307240 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.307158 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6166d64-362f-4ada-b2fd-ecf68bc023a7-istiod-ca-cert\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6\" (UID: \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" Nov 03 22:59:26.307298 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.307280 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6166d64-362f-4ada-b2fd-ecf68bc023a7-workload-certs\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6\" (UID: \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" Nov 03 22:59:26.307383 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.307368 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6166d64-362f-4ada-b2fd-ecf68bc023a7-istio-data\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6\" (UID: \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" Nov 03 22:59:26.307449 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.307414 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6166d64-362f-4ada-b2fd-ecf68bc023a7-credential-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6\" (UID: \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" Nov 03 22:59:26.307506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.307474 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6166d64-362f-4ada-b2fd-ecf68bc023a7-workload-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6\" (UID: \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" Nov 03 22:59:26.307506 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.307500 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6166d64-362f-4ada-b2fd-ecf68bc023a7-istio-token\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6\" (UID: \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" Nov 03 22:59:26.307662 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.307636 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6166d64-362f-4ada-b2fd-ecf68bc023a7-workload-certs\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6\" (UID: \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" Nov 03 22:59:26.307719 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.307700 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6166d64-362f-4ada-b2fd-ecf68bc023a7-istio-data\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6\" (UID: \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" Nov 03 22:59:26.307910 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.307892 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6166d64-362f-4ada-b2fd-ecf68bc023a7-istiod-ca-cert\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6\" (UID: \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" Nov 03 22:59:26.308106 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.308082 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6166d64-362f-4ada-b2fd-ecf68bc023a7-workload-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6\" (UID: \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" Nov 03 22:59:26.308369 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.308348 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6166d64-362f-4ada-b2fd-ecf68bc023a7-credential-socket\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6\" (UID: \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" Nov 03 22:59:26.309530 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.309513 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6166d64-362f-4ada-b2fd-ecf68bc023a7-istio-envoy\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6\" (UID: \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" Nov 03 22:59:26.309699 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.309678 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6166d64-362f-4ada-b2fd-ecf68bc023a7-istio-podinfo\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6\" (UID: \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" Nov 03 22:59:26.369082 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.369054 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6166d64-362f-4ada-b2fd-ecf68bc023a7-istio-token\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6\" (UID: \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" Nov 03 22:59:26.369229 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.369135 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t29bc\" (UniqueName: \"kubernetes.io/projected/a6166d64-362f-4ada-b2fd-ecf68bc023a7-kube-api-access-t29bc\") pod \"e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6\" (UID: \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\") " pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" Nov 03 22:59:26.389685 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.389663 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" Nov 03 22:59:26.822607 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.822579 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6"] Nov 03 22:59:26.824051 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:59:26.824023 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6166d64_362f_4ada_b2fd_ecf68bc023a7.slice/crio-53ce57e00f416bbb1133ab97e71b2cadccdbdb17891a6e6ada81943834a18ee3 WatchSource:0}: Error finding container 53ce57e00f416bbb1133ab97e71b2cadccdbdb17891a6e6ada81943834a18ee3: Status 404 returned error can't find the container with id 53ce57e00f416bbb1133ab97e71b2cadccdbdb17891a6e6ada81943834a18ee3 Nov 03 22:59:26.826697 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.826645 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:59:26.826811 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.826731 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:59:26.826811 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:26.826768 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:59:27.503750 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:27.503210 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" event={"ID":"a6166d64-362f-4ada-b2fd-ecf68bc023a7","Type":"ContainerStarted","Data":"fd25979b4993272a6e065da57b2cd211a2f592f762ef7a886a246f93820166cc"} Nov 03 22:59:27.503750 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:27.503277 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" event={"ID":"a6166d64-362f-4ada-b2fd-ecf68bc023a7","Type":"ContainerStarted","Data":"53ce57e00f416bbb1133ab97e71b2cadccdbdb17891a6e6ada81943834a18ee3"} Nov 03 22:59:27.685548 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:27.685485 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" podStartSLOduration=1.68546503 podStartE2EDuration="1.68546503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:59:27.683438655 +0000 UTC m=+2196.992997949" watchObservedRunningTime="2026-11-03 22:59:27.68546503 +0000 UTC m=+2196.995024322" Nov 03 22:59:27.983409 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:27.983378 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6977-ece77b"] Nov 03 22:59:27.987409 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:27.987385 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6977-ece77b" Nov 03 22:59:28.042175 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:28.042143 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jgmv2\"" Nov 03 22:59:28.045194 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:28.045166 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6977-ece77b"] Nov 03 22:59:28.143671 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:28.143636 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pdm8\" (UniqueName: \"kubernetes.io/projected/57e4da61-0878-4436-9676-ebc18fae9986-kube-api-access-2pdm8\") pod \"test-curl-6977-ece77b\" (UID: \"57e4da61-0878-4436-9676-ebc18fae9986\") " pod="opendatahub/test-curl-6977-ece77b" Nov 03 22:59:28.245123 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:28.245037 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pdm8\" (UniqueName: \"kubernetes.io/projected/57e4da61-0878-4436-9676-ebc18fae9986-kube-api-access-2pdm8\") pod \"test-curl-6977-ece77b\" (UID: \"57e4da61-0878-4436-9676-ebc18fae9986\") " pod="opendatahub/test-curl-6977-ece77b" Nov 03 22:59:28.319789 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:28.319748 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pdm8\" (UniqueName: \"kubernetes.io/projected/57e4da61-0878-4436-9676-ebc18fae9986-kube-api-access-2pdm8\") pod \"test-curl-6977-ece77b\" (UID: \"57e4da61-0878-4436-9676-ebc18fae9986\") " pod="opendatahub/test-curl-6977-ece77b" Nov 03 22:59:28.325812 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:28.325771 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6977-ece77b" Nov 03 22:59:28.395970 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:28.395925 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" Nov 03 22:59:28.423579 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:28.423551 5196 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 22:59:28.423796 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:28.423757 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" podUID="a6166d64-362f-4ada-b2fd-ecf68bc023a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 22:59:28.731974 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:28.731934 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6977-ece77b"] Nov 03 22:59:28.733758 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:59:28.733731 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57e4da61_0878_4436_9676_ebc18fae9986.slice/crio-567c16bcb62afcf88889e18b09df1977334bdb93fe293c72196b4aeed449fc47 WatchSource:0}: Error finding container 567c16bcb62afcf88889e18b09df1977334bdb93fe293c72196b4aeed449fc47: Status 404 returned error can't find the container with id 567c16bcb62afcf88889e18b09df1977334bdb93fe293c72196b4aeed449fc47 Nov 03 22:59:28.975791 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:28.975761 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp"] Nov 03 22:59:28.979821 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:28.979803 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp" Nov 03 22:59:29.036241 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:29.036210 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-924fba-serving-cert\"" Nov 03 22:59:29.040284 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:29.040257 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp"] Nov 03 22:59:29.059222 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:29.059195 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tljwg\" (UniqueName: \"kubernetes.io/projected/47d17627-fece-4154-a00b-bfa9838574ca-kube-api-access-tljwg\") pod \"maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp\" (UID: \"47d17627-fece-4154-a00b-bfa9838574ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp" Nov 03 22:59:29.059360 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:29.059242 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47d17627-fece-4154-a00b-bfa9838574ca-openshift-service-ca\") pod \"maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp\" (UID: \"47d17627-fece-4154-a00b-bfa9838574ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp" Nov 03 22:59:29.059360 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:29.059327 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47d17627-fece-4154-a00b-bfa9838574ca-metrics-certs\") pod \"maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp\" (UID: \"47d17627-fece-4154-a00b-bfa9838574ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp" Nov 03 22:59:29.059464 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:29.059449 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47d17627-fece-4154-a00b-bfa9838574ca-maas-api-tls\") pod \"maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp\" (UID: \"47d17627-fece-4154-a00b-bfa9838574ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp" Nov 03 22:59:29.160343 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:29.160308 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tljwg\" (UniqueName: \"kubernetes.io/projected/47d17627-fece-4154-a00b-bfa9838574ca-kube-api-access-tljwg\") pod \"maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp\" (UID: \"47d17627-fece-4154-a00b-bfa9838574ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp" Nov 03 22:59:29.160343 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:29.160352 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47d17627-fece-4154-a00b-bfa9838574ca-openshift-service-ca\") pod \"maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp\" (UID: \"47d17627-fece-4154-a00b-bfa9838574ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp" Nov 03 22:59:29.160620 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:29.160391 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47d17627-fece-4154-a00b-bfa9838574ca-metrics-certs\") pod \"maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp\" (UID: \"47d17627-fece-4154-a00b-bfa9838574ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp" Nov 03 22:59:29.160620 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:29.160443 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47d17627-fece-4154-a00b-bfa9838574ca-maas-api-tls\") pod \"maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp\" (UID: \"47d17627-fece-4154-a00b-bfa9838574ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp" Nov 03 22:59:29.160746 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:59:29.160619 5196 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-924fba-serving-cert: secret "maas-api-e2e-inf-a-924fba-serving-cert" not found Nov 03 22:59:29.160746 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:59:29.160706 5196 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47d17627-fece-4154-a00b-bfa9838574ca-maas-api-tls podName:47d17627-fece-4154-a00b-bfa9838574ca nodeName:}" failed. No retries permitted until 2026-11-03 22:59:29.660687732 +0000 UTC m=+2198.970247003 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/47d17627-fece-4154-a00b-bfa9838574ca-maas-api-tls") pod "maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp" (UID: "47d17627-fece-4154-a00b-bfa9838574ca") : secret "maas-api-e2e-inf-a-924fba-serving-cert" not found Nov 03 22:59:29.161105 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:29.161084 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47d17627-fece-4154-a00b-bfa9838574ca-openshift-service-ca\") pod \"maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp\" (UID: \"47d17627-fece-4154-a00b-bfa9838574ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp" Nov 03 22:59:29.163065 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:29.163039 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47d17627-fece-4154-a00b-bfa9838574ca-metrics-certs\") pod \"maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp\" (UID: \"47d17627-fece-4154-a00b-bfa9838574ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp" Nov 03 22:59:29.169229 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:29.169200 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:29.223084 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:29.223058 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tljwg\" (UniqueName: \"kubernetes.io/projected/47d17627-fece-4154-a00b-bfa9838574ca-kube-api-access-tljwg\") pod \"maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp\" (UID: \"47d17627-fece-4154-a00b-bfa9838574ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp" Nov 03 22:59:29.394881 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:29.390634 5196 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 22:59:29.394881 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:29.390692 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" podUID="a6166d64-362f-4ada-b2fd-ecf68bc023a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 22:59:29.394881 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:29.392843 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:29.546929 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:29.546545 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:29.551115 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:29.549602 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6977-ece77b" event={"ID":"57e4da61-0878-4436-9676-ebc18fae9986","Type":"ContainerStarted","Data":"08f875438df53a52607639c68cd908e63b4e6a4320276f0050dfae58ea678806"} Nov 03 22:59:29.551115 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:29.549652 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6977-ece77b" event={"ID":"57e4da61-0878-4436-9676-ebc18fae9986","Type":"ContainerStarted","Data":"567c16bcb62afcf88889e18b09df1977334bdb93fe293c72196b4aeed449fc47"} Nov 03 22:59:29.673311 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:29.673205 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6977-ece77b" podStartSLOduration=2.673188642 podStartE2EDuration="2.673188642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:59:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:59:29.672911492 +0000 UTC m=+2198.982470806" watchObservedRunningTime="2026-11-03 22:59:29.673188642 +0000 UTC m=+2198.982747937" Nov 03 22:59:29.677183 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:29.677144 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47d17627-fece-4154-a00b-bfa9838574ca-maas-api-tls\") pod \"maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp\" (UID: \"47d17627-fece-4154-a00b-bfa9838574ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp" Nov 03 22:59:29.679944 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:29.679918 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47d17627-fece-4154-a00b-bfa9838574ca-maas-api-tls\") pod \"maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp\" (UID: \"47d17627-fece-4154-a00b-bfa9838574ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp" Nov 03 22:59:29.910882 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:29.908049 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp" Nov 03 22:59:30.346405 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:30.346379 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp"] Nov 03 22:59:30.348048 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:59:30.348016 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d17627_fece_4154_a00b_bfa9838574ca.slice/crio-0a36e521c5ce43d364922234e9ad8b31e37f699e0d573a38f5cab8c8ee330365 WatchSource:0}: Error finding container 0a36e521c5ce43d364922234e9ad8b31e37f699e0d573a38f5cab8c8ee330365: Status 404 returned error can't find the container with id 0a36e521c5ce43d364922234e9ad8b31e37f699e0d573a38f5cab8c8ee330365 Nov 03 22:59:30.411880 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:30.411239 5196 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 22:59:30.411880 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:30.411296 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" podUID="a6166d64-362f-4ada-b2fd-ecf68bc023a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 22:59:30.557164 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:30.557129 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp" event={"ID":"47d17627-fece-4154-a00b-bfa9838574ca","Type":"ContainerStarted","Data":"0a36e521c5ce43d364922234e9ad8b31e37f699e0d573a38f5cab8c8ee330365"} Nov 03 22:59:31.237274 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:31.237244 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6977-ece77b"] Nov 03 22:59:31.259592 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:31.259559 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6977-ece77b"] Nov 03 22:59:31.395230 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:31.395200 5196 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 22:59:31.395490 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:31.395462 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" podUID="a6166d64-362f-4ada-b2fd-ecf68bc023a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 22:59:31.566388 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:31.566297 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp" event={"ID":"47d17627-fece-4154-a00b-bfa9838574ca","Type":"ContainerStarted","Data":"58baef1b4d5b44c0810712da07bdb568f8f9f2b63a5be603c0a04c8e5fab8d61"} Nov 03 22:59:31.566388 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:31.566344 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6977-ece77b" podUID="57e4da61-0878-4436-9676-ebc18fae9986" containerName="test-curl-6977-ece77b" containerID="cri-o://08f875438df53a52607639c68cd908e63b4e6a4320276f0050dfae58ea678806" gracePeriod=2 Nov 03 22:59:31.685695 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:31.685646 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp" podStartSLOduration=2.85587812 podStartE2EDuration="3.685628338s" totalImagesPullingTime="829.750218ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:59:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:59:31.685362409 +0000 UTC m=+2200.994921702" watchObservedRunningTime="2026-11-03 22:59:31.685628338 +0000 UTC m=+2200.995187636" Nov 03 22:59:31.737245 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:59:31.737207 5196 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6977-ece77b\" is forbidden: User \"system:node:ip-10-0-143-132.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-132.us-west-1.compute.internal' and this object" podUID="57e4da61-0878-4436-9676-ebc18fae9986" pod="opendatahub/test-curl-6977-ece77b" Nov 03 22:59:31.789466 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:59:31.789418 5196 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6977-ece77b\" is forbidden: User \"system:node:ip-10-0-143-132.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-132.us-west-1.compute.internal' and this object" podUID="57e4da61-0878-4436-9676-ebc18fae9986" pod="opendatahub/test-curl-6977-ece77b" Nov 03 22:59:32.416883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:32.414332 5196 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 22:59:32.416883 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:32.414391 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" podUID="a6166d64-362f-4ada-b2fd-ecf68bc023a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 22:59:32.418924 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:32.418895 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ea5c8f-7db7d5b55f-qnbc8" Nov 03 22:59:32.471231 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:59:32.471194 5196 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6977-ece77b\" is forbidden: User \"system:node:ip-10-0-143-132.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-132.us-west-1.compute.internal' and this object" podUID="57e4da61-0878-4436-9676-ebc18fae9986" pod="opendatahub/test-curl-6977-ece77b" Nov 03 22:59:32.572306 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:32.572268 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp" Nov 03 22:59:32.977297 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:32.977261 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:33.124581 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:33.124542 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:33.440712 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:33.440682 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" Nov 03 22:59:33.441313 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:33.441283 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" Nov 03 22:59:33.443041 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:33.443019 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" Nov 03 22:59:33.849711 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:33.849685 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6977-ece77b" Nov 03 22:59:33.991244 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:33.991213 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57e4da61-0878-4436-9676-ebc18fae9986-kube-api-access-2pdm8\" (UniqueName: \"kubernetes.io/projected/57e4da61-0878-4436-9676-ebc18fae9986-kube-api-access-2pdm8\") pod \"57e4da61-0878-4436-9676-ebc18fae9986\" (UID: \"57e4da61-0878-4436-9676-ebc18fae9986\") " Nov 03 22:59:33.993573 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:33.993532 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57e4da61-0878-4436-9676-ebc18fae9986-kube-api-access-2pdm8" pod "57e4da61-0878-4436-9676-ebc18fae9986" (UID: "57e4da61-0878-4436-9676-ebc18fae9986"). InnerVolumeSpecName "kube-api-access-2pdm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:59:34.092816 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:34.092783 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pdm8\" (UniqueName: \"kubernetes.io/projected/57e4da61-0878-4436-9676-ebc18fae9986-kube-api-access-2pdm8\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 22:59:34.222812 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:34.222780 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:34.422353 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:34.422323 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:34.574573 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:34.574541 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:34.587607 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:34.587579 5196 generic.go:356] "Generic (PLEG): container finished" podID="57e4da61-0878-4436-9676-ebc18fae9986" containerID="08f875438df53a52607639c68cd908e63b4e6a4320276f0050dfae58ea678806" exitCode=137 Nov 03 22:59:34.587726 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:34.587625 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6977-ece77b" Nov 03 22:59:34.587726 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:34.587651 5196 scope.go:122] "RemoveContainer" containerID="08f875438df53a52607639c68cd908e63b4e6a4320276f0050dfae58ea678806" Nov 03 22:59:34.597600 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:34.597582 5196 scope.go:122] "RemoveContainer" containerID="08f875438df53a52607639c68cd908e63b4e6a4320276f0050dfae58ea678806" Nov 03 22:59:34.597889 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:59:34.597849 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08f875438df53a52607639c68cd908e63b4e6a4320276f0050dfae58ea678806\": container with ID starting with 08f875438df53a52607639c68cd908e63b4e6a4320276f0050dfae58ea678806 not found: ID does not exist" containerID="08f875438df53a52607639c68cd908e63b4e6a4320276f0050dfae58ea678806" Nov 03 22:59:34.597950 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:34.597902 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08f875438df53a52607639c68cd908e63b4e6a4320276f0050dfae58ea678806"} err="failed to get container status \"08f875438df53a52607639c68cd908e63b4e6a4320276f0050dfae58ea678806\": rpc error: code = NotFound desc = could not find container \"08f875438df53a52607639c68cd908e63b4e6a4320276f0050dfae58ea678806\": container with ID starting with 08f875438df53a52607639c68cd908e63b4e6a4320276f0050dfae58ea678806 not found: ID does not exist" Nov 03 22:59:34.648072 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:59:34.648041 5196 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6977-ece77b\" is forbidden: User \"system:node:ip-10-0-143-132.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-132.us-west-1.compute.internal' and this object" podUID="57e4da61-0878-4436-9676-ebc18fae9986" pod="opendatahub/test-curl-6977-ece77b" Nov 03 22:59:34.831923 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:34.830081 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57e4da61-0878-4436-9676-ebc18fae9986" path="/var/lib/kubelet/pods/57e4da61-0878-4436-9676-ebc18fae9986/volumes" Nov 03 22:59:38.585820 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:38.585791 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp" Nov 03 22:59:39.947250 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:39.947221 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:43.909571 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:43.909533 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:44.054339 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:44.054309 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:45.200040 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:45.200014 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:50.352579 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:50.352545 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:52.365913 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:52.365880 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:52.836556 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:52.836483 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:53.349389 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:53.349360 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4"] Nov 03 22:59:53.349915 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:53.349900 5196 cpu_manager.go:414] "containerMap: removing container" podUID="57e4da61-0878-4436-9676-ebc18fae9986" containerName="test-curl-6977-ece77b" Nov 03 22:59:53.349915 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:53.349915 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57e4da61-0878-4436-9676-ebc18fae9986" containerName="test-curl-6977-ece77b" Nov 03 22:59:53.350035 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:53.350010 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="57e4da61-0878-4436-9676-ebc18fae9986" containerName="test-curl-6977-ece77b" Nov 03 22:59:53.354871 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:53.354828 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 22:59:53.432973 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:53.432940 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59200876-e179-4170-832d-d85cba111281-workload-certs\") pod \"e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4\" (UID: \"59200876-e179-4170-832d-d85cba111281\") " pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 22:59:53.433390 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:53.433127 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59200876-e179-4170-832d-d85cba111281-workload-socket\") pod \"e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4\" (UID: \"59200876-e179-4170-832d-d85cba111281\") " pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 22:59:53.433390 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:53.433193 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzgxs\" (UniqueName: \"kubernetes.io/projected/59200876-e179-4170-832d-d85cba111281-kube-api-access-tzgxs\") pod \"e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4\" (UID: \"59200876-e179-4170-832d-d85cba111281\") " pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 22:59:53.433390 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:53.433291 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59200876-e179-4170-832d-d85cba111281-istio-data\") pod \"e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4\" (UID: \"59200876-e179-4170-832d-d85cba111281\") " pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 22:59:53.433390 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:53.433341 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59200876-e179-4170-832d-d85cba111281-istio-token\") pod \"e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4\" (UID: \"59200876-e179-4170-832d-d85cba111281\") " pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 22:59:53.433543 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:53.433491 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59200876-e179-4170-832d-d85cba111281-istio-podinfo\") pod \"e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4\" (UID: \"59200876-e179-4170-832d-d85cba111281\") " pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 22:59:53.433585 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:53.433541 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59200876-e179-4170-832d-d85cba111281-istio-envoy\") pod \"e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4\" (UID: \"59200876-e179-4170-832d-d85cba111281\") " pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 22:59:53.433656 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:53.433638 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59200876-e179-4170-832d-d85cba111281-credential-socket\") pod \"e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4\" (UID: \"59200876-e179-4170-832d-d85cba111281\") " pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 22:59:53.433709 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:53.433677 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59200876-e179-4170-832d-d85cba111281-istiod-ca-cert\") pod \"e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4\" (UID: \"59200876-e179-4170-832d-d85cba111281\") " pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 22:59:53.447987 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:53.447958 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e66d59-openshift-default-dockercfg-scwl5\"" Nov 03 22:59:53.448971 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:53.448935 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:53.464432 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:53.464409 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4"] Nov 03 22:59:53.563885 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:53.541606 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59200876-e179-4170-832d-d85cba111281-workload-certs\") pod \"e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4\" (UID: \"59200876-e179-4170-832d-d85cba111281\") " pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 22:59:53.589376 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:53.556489 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59200876-e179-4170-832d-d85cba111281-workload-certs\") pod \"e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4\" (UID: \"59200876-e179-4170-832d-d85cba111281\") " pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 22:59:53.598085 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:53.598047 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59200876-e179-4170-832d-d85cba111281-workload-socket\") pod \"e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4\" (UID: \"59200876-e179-4170-832d-d85cba111281\") " pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 22:59:53.620968 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:53.618245 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzgxs\" (UniqueName: \"kubernetes.io/projected/59200876-e179-4170-832d-d85cba111281-kube-api-access-tzgxs\") pod \"e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4\" (UID: \"59200876-e179-4170-832d-d85cba111281\") " pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 22:59:53.620968 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:53.618322 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59200876-e179-4170-832d-d85cba111281-istio-data\") pod \"e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4\" (UID: \"59200876-e179-4170-832d-d85cba111281\") " pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 22:59:53.620968 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:53.618376 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59200876-e179-4170-832d-d85cba111281-istio-token\") pod \"e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4\" (UID: \"59200876-e179-4170-832d-d85cba111281\") " pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 22:59:53.620968 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:53.618495 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59200876-e179-4170-832d-d85cba111281-istio-podinfo\") pod \"e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4\" (UID: \"59200876-e179-4170-832d-d85cba111281\") " pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 22:59:53.620968 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:53.618539 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59200876-e179-4170-832d-d85cba111281-istio-envoy\") pod \"e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4\" (UID: \"59200876-e179-4170-832d-d85cba111281\") " pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 22:59:53.620968 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:53.618592 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59200876-e179-4170-832d-d85cba111281-credential-socket\") pod \"e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4\" (UID: \"59200876-e179-4170-832d-d85cba111281\") " pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 22:59:53.620968 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:53.618625 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59200876-e179-4170-832d-d85cba111281-istiod-ca-cert\") pod \"e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4\" (UID: \"59200876-e179-4170-832d-d85cba111281\") " pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 22:59:53.620968 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:53.619608 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59200876-e179-4170-832d-d85cba111281-istiod-ca-cert\") pod \"e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4\" (UID: \"59200876-e179-4170-832d-d85cba111281\") " pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 22:59:53.620968 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:53.598630 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59200876-e179-4170-832d-d85cba111281-workload-socket\") pod \"e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4\" (UID: \"59200876-e179-4170-832d-d85cba111281\") " pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 22:59:53.620968 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:53.620274 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59200876-e179-4170-832d-d85cba111281-istio-data\") pod \"e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4\" (UID: \"59200876-e179-4170-832d-d85cba111281\") " pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 22:59:53.623266 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:53.623245 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59200876-e179-4170-832d-d85cba111281-istio-podinfo\") pod \"e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4\" (UID: \"59200876-e179-4170-832d-d85cba111281\") " pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 22:59:53.625620 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:53.625595 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59200876-e179-4170-832d-d85cba111281-istio-envoy\") pod \"e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4\" (UID: \"59200876-e179-4170-832d-d85cba111281\") " pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 22:59:53.626061 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:53.626041 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59200876-e179-4170-832d-d85cba111281-credential-socket\") pod \"e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4\" (UID: \"59200876-e179-4170-832d-d85cba111281\") " pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 22:59:53.681982 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:53.681950 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59200876-e179-4170-832d-d85cba111281-istio-token\") pod \"e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4\" (UID: \"59200876-e179-4170-832d-d85cba111281\") " pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 22:59:53.682145 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:53.682080 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzgxs\" (UniqueName: \"kubernetes.io/projected/59200876-e179-4170-832d-d85cba111281-kube-api-access-tzgxs\") pod \"e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4\" (UID: \"59200876-e179-4170-832d-d85cba111281\") " pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 22:59:53.780767 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:53.780731 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:53.970328 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:53.970249 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 22:59:54.412219 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:54.412195 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4"] Nov 03 22:59:54.413153 ip-10-0-143-132 kubenswrapper[5196]: W1103 22:59:54.413125 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59200876_e179_4170_832d_d85cba111281.slice/crio-55baed265dc0d69df9d7fc5e107098d99b00682b7571d2d3ee44dd3ee4ba2aa7 WatchSource:0}: Error finding container 55baed265dc0d69df9d7fc5e107098d99b00682b7571d2d3ee44dd3ee4ba2aa7: Status 404 returned error can't find the container with id 55baed265dc0d69df9d7fc5e107098d99b00682b7571d2d3ee44dd3ee4ba2aa7 Nov 03 22:59:54.416443 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:54.416408 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:59:54.416520 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:54.416477 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:59:54.416520 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:54.416506 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 22:59:54.728751 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:54.728671 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" event={"ID":"59200876-e179-4170-832d-d85cba111281","Type":"ContainerStarted","Data":"159fb8a18f6a38943292cbe931ada2d53535627cd6be0c70cda2d95bb733c204"} Nov 03 22:59:54.728751 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:54.728726 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" event={"ID":"59200876-e179-4170-832d-d85cba111281","Type":"ContainerStarted","Data":"55baed265dc0d69df9d7fc5e107098d99b00682b7571d2d3ee44dd3ee4ba2aa7"} Nov 03 22:59:54.854907 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:54.854798 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" podStartSLOduration=1.854779833 podStartE2EDuration="1.854779833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:59:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:59:54.850469282 +0000 UTC m=+2224.160028576" watchObservedRunningTime="2026-11-03 22:59:54.854779833 +0000 UTC m=+2224.164339128" Nov 03 22:59:54.974058 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:54.973968 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 22:59:55.598045 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:55.598010 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:55.919105 ip-10-0-143-132 kubenswrapper[5196]: E1103 22:59:55.919026 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 22:59:55.976786 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:55.976756 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 22:59:56.741008 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:56.740966 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 22:59:56.742058 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:56.742035 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 22:59:57.993440 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:57.993408 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:58.155053 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:58.155021 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 22:59:58.312702 ip-10-0-143-132 kubenswrapper[5196]: I1103 22:59:58.312611 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:00.828238 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:00.828207 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:03.030568 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:03.030532 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:03.178612 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:03.178582 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:06.025439 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:06.025401 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:09.169430 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:09.169375 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:10.826695 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:10.826638 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:14.589063 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:14.589031 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:14.959942 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:14.959663 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42"] Nov 03 23:00:14.960253 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:14.959944 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42" podUID="3a86e97c-52cb-4ec2-9ff2-250940649d94" containerName="maas-api" containerID="cri-o://114e29f9265d8fef6116b09bb1ef50357e3a2f7e9953368cfef6e16318b46d34" gracePeriod=30 Nov 03 23:00:15.004892 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:15.003727 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:15.526599 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:15.526570 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42" Nov 03 23:00:15.616883 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:15.604465 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a86e97c-52cb-4ec2-9ff2-250940649d94-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a86e97c-52cb-4ec2-9ff2-250940649d94-maas-api-tls\") pod \"3a86e97c-52cb-4ec2-9ff2-250940649d94\" (UID: \"3a86e97c-52cb-4ec2-9ff2-250940649d94\") " Nov 03 23:00:15.616883 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:15.604565 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a86e97c-52cb-4ec2-9ff2-250940649d94-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a86e97c-52cb-4ec2-9ff2-250940649d94-metrics-certs\") pod \"3a86e97c-52cb-4ec2-9ff2-250940649d94\" (UID: \"3a86e97c-52cb-4ec2-9ff2-250940649d94\") " Nov 03 23:00:15.616883 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:15.604724 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a86e97c-52cb-4ec2-9ff2-250940649d94-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a86e97c-52cb-4ec2-9ff2-250940649d94-openshift-service-ca\") pod \"3a86e97c-52cb-4ec2-9ff2-250940649d94\" (UID: \"3a86e97c-52cb-4ec2-9ff2-250940649d94\") " Nov 03 23:00:15.616883 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:15.604876 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a86e97c-52cb-4ec2-9ff2-250940649d94-kube-api-access-8zlxh\" (UniqueName: \"kubernetes.io/projected/3a86e97c-52cb-4ec2-9ff2-250940649d94-kube-api-access-8zlxh\") pod \"3a86e97c-52cb-4ec2-9ff2-250940649d94\" (UID: \"3a86e97c-52cb-4ec2-9ff2-250940649d94\") " Nov 03 23:00:15.619238 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:15.618592 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a86e97c-52cb-4ec2-9ff2-250940649d94-kube-api-access-8zlxh" pod "3a86e97c-52cb-4ec2-9ff2-250940649d94" (UID: "3a86e97c-52cb-4ec2-9ff2-250940649d94"). InnerVolumeSpecName "kube-api-access-8zlxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:15.619238 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:15.618637 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a86e97c-52cb-4ec2-9ff2-250940649d94-openshift-service-ca" pod "3a86e97c-52cb-4ec2-9ff2-250940649d94" (UID: "3a86e97c-52cb-4ec2-9ff2-250940649d94"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:15.632304 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:15.632267 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a86e97c-52cb-4ec2-9ff2-250940649d94-maas-api-tls" pod "3a86e97c-52cb-4ec2-9ff2-250940649d94" (UID: "3a86e97c-52cb-4ec2-9ff2-250940649d94"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:00:15.632439 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:15.632388 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a86e97c-52cb-4ec2-9ff2-250940649d94-metrics-certs" pod "3a86e97c-52cb-4ec2-9ff2-250940649d94" (UID: "3a86e97c-52cb-4ec2-9ff2-250940649d94"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:00:15.715882 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:15.713874 5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a86e97c-52cb-4ec2-9ff2-250940649d94-maas-api-tls\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:00:15.715882 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:15.714952 5196 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a86e97c-52cb-4ec2-9ff2-250940649d94-metrics-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:00:15.715882 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:15.714977 5196 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a86e97c-52cb-4ec2-9ff2-250940649d94-openshift-service-ca\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:00:15.715882 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:15.714993 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zlxh\" (UniqueName: \"kubernetes.io/projected/3a86e97c-52cb-4ec2-9ff2-250940649d94-kube-api-access-8zlxh\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:00:15.873645 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:15.873551 5196 generic.go:356] "Generic (PLEG): container finished" podID="3a86e97c-52cb-4ec2-9ff2-250940649d94" containerID="114e29f9265d8fef6116b09bb1ef50357e3a2f7e9953368cfef6e16318b46d34" exitCode=0 Nov 03 23:00:15.874086 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:15.874060 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42" event={"ID":"3a86e97c-52cb-4ec2-9ff2-250940649d94","Type":"ContainerDied","Data":"114e29f9265d8fef6116b09bb1ef50357e3a2f7e9953368cfef6e16318b46d34"} Nov 03 23:00:15.874258 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:15.874242 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42" event={"ID":"3a86e97c-52cb-4ec2-9ff2-250940649d94","Type":"ContainerDied","Data":"6d7d89843d296387f216a9f4b7c3c94273f0fd378060b70c4c497a44dfd44a51"} Nov 03 23:00:15.874377 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:15.874365 5196 scope.go:122] "RemoveContainer" containerID="114e29f9265d8fef6116b09bb1ef50357e3a2f7e9953368cfef6e16318b46d34" Nov 03 23:00:15.874632 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:15.874616 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42" Nov 03 23:00:15.949244 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:15.947094 5196 scope.go:122] "RemoveContainer" containerID="114e29f9265d8fef6116b09bb1ef50357e3a2f7e9953368cfef6e16318b46d34" Nov 03 23:00:15.949244 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:00:15.947500 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"114e29f9265d8fef6116b09bb1ef50357e3a2f7e9953368cfef6e16318b46d34\": container with ID starting with 114e29f9265d8fef6116b09bb1ef50357e3a2f7e9953368cfef6e16318b46d34 not found: ID does not exist" containerID="114e29f9265d8fef6116b09bb1ef50357e3a2f7e9953368cfef6e16318b46d34" Nov 03 23:00:15.949244 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:15.947538 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"114e29f9265d8fef6116b09bb1ef50357e3a2f7e9953368cfef6e16318b46d34"} err="failed to get container status \"114e29f9265d8fef6116b09bb1ef50357e3a2f7e9953368cfef6e16318b46d34\": rpc error: code = NotFound desc = could not find container \"114e29f9265d8fef6116b09bb1ef50357e3a2f7e9953368cfef6e16318b46d34\": container with ID starting with 114e29f9265d8fef6116b09bb1ef50357e3a2f7e9953368cfef6e16318b46d34 not found: ID does not exist" Nov 03 23:00:16.060612 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:16.060574 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42"] Nov 03 23:00:16.066719 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:16.066695 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42"] Nov 03 23:00:16.823182 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:16.823148 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a86e97c-52cb-4ec2-9ff2-250940649d94" path="/var/lib/kubelet/pods/3a86e97c-52cb-4ec2-9ff2-250940649d94/volumes" Nov 03 23:00:17.036219 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:17.036189 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:17.199370 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:17.199338 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:17.531331 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:17.531244 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:18.141326 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:18.141286 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:18.404938 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:18.400948 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-cbd26c-7b485579c6-62n42" podUID="3a86e97c-52cb-4ec2-9ff2-250940649d94" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.49:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:00:19.698083 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:19.698001 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:20.188219 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:20.188175 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:20.725095 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:20.725057 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn"] Nov 03 23:00:20.725937 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:20.725917 5196 cpu_manager.go:414] "containerMap: removing container" podUID="3a86e97c-52cb-4ec2-9ff2-250940649d94" containerName="maas-api" Nov 03 23:00:20.726011 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:20.725941 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a86e97c-52cb-4ec2-9ff2-250940649d94" containerName="maas-api" Nov 03 23:00:20.726129 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:20.726115 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a86e97c-52cb-4ec2-9ff2-250940649d94" containerName="maas-api" Nov 03 23:00:20.731452 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:20.731430 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:00:20.794190 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:20.792545 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-dcf5a5-openshift-default-dockercfg-tsr4x\"" Nov 03 23:00:20.859882 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:20.805633 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b847dde-7257-44e0-b25c-9ebe4cdad697-workload-socket\") pod \"e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn\" (UID: \"4b847dde-7257-44e0-b25c-9ebe4cdad697\") " pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:00:20.859882 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:20.805683 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b847dde-7257-44e0-b25c-9ebe4cdad697-istio-token\") pod \"e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn\" (UID: \"4b847dde-7257-44e0-b25c-9ebe4cdad697\") " pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:00:20.859882 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:20.805791 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b847dde-7257-44e0-b25c-9ebe4cdad697-workload-certs\") pod \"e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn\" (UID: \"4b847dde-7257-44e0-b25c-9ebe4cdad697\") " pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:00:20.859882 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:20.805845 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b847dde-7257-44e0-b25c-9ebe4cdad697-credential-socket\") pod \"e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn\" (UID: \"4b847dde-7257-44e0-b25c-9ebe4cdad697\") " pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:00:20.861355 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:20.860710 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b847dde-7257-44e0-b25c-9ebe4cdad697-istio-data\") pod \"e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn\" (UID: \"4b847dde-7257-44e0-b25c-9ebe4cdad697\") " pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:00:20.861355 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:20.860818 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq8kl\" (UniqueName: \"kubernetes.io/projected/4b847dde-7257-44e0-b25c-9ebe4cdad697-kube-api-access-rq8kl\") pod \"e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn\" (UID: \"4b847dde-7257-44e0-b25c-9ebe4cdad697\") " pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:00:20.861355 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:20.836342 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:20.861355 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:20.861047 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b847dde-7257-44e0-b25c-9ebe4cdad697-istio-podinfo\") pod \"e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn\" (UID: \"4b847dde-7257-44e0-b25c-9ebe4cdad697\") " pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:00:20.861355 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:20.861085 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b847dde-7257-44e0-b25c-9ebe4cdad697-istiod-ca-cert\") pod \"e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn\" (UID: \"4b847dde-7257-44e0-b25c-9ebe4cdad697\") " pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:00:20.861355 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:20.861171 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b847dde-7257-44e0-b25c-9ebe4cdad697-istio-envoy\") pod \"e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn\" (UID: \"4b847dde-7257-44e0-b25c-9ebe4cdad697\") " pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:00:20.963081 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:20.963040 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b847dde-7257-44e0-b25c-9ebe4cdad697-credential-socket\") pod \"e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn\" (UID: \"4b847dde-7257-44e0-b25c-9ebe4cdad697\") " pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:00:20.964961 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:20.964927 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b847dde-7257-44e0-b25c-9ebe4cdad697-istio-data\") pod \"e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn\" (UID: \"4b847dde-7257-44e0-b25c-9ebe4cdad697\") " pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:00:20.965101 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:20.965037 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq8kl\" (UniqueName: \"kubernetes.io/projected/4b847dde-7257-44e0-b25c-9ebe4cdad697-kube-api-access-rq8kl\") pod \"e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn\" (UID: \"4b847dde-7257-44e0-b25c-9ebe4cdad697\") " pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:00:20.965101 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:20.965083 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b847dde-7257-44e0-b25c-9ebe4cdad697-istio-podinfo\") pod \"e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn\" (UID: \"4b847dde-7257-44e0-b25c-9ebe4cdad697\") " pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:00:20.965222 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:20.965115 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b847dde-7257-44e0-b25c-9ebe4cdad697-istiod-ca-cert\") pod \"e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn\" (UID: \"4b847dde-7257-44e0-b25c-9ebe4cdad697\") " pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:00:20.965222 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:20.965189 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b847dde-7257-44e0-b25c-9ebe4cdad697-istio-envoy\") pod \"e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn\" (UID: \"4b847dde-7257-44e0-b25c-9ebe4cdad697\") " pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:00:20.965347 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:20.965243 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b847dde-7257-44e0-b25c-9ebe4cdad697-workload-socket\") pod \"e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn\" (UID: \"4b847dde-7257-44e0-b25c-9ebe4cdad697\") " pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:00:20.965347 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:20.965272 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b847dde-7257-44e0-b25c-9ebe4cdad697-istio-token\") pod \"e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn\" (UID: \"4b847dde-7257-44e0-b25c-9ebe4cdad697\") " pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:00:20.965453 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:20.965383 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b847dde-7257-44e0-b25c-9ebe4cdad697-workload-certs\") pod \"e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn\" (UID: \"4b847dde-7257-44e0-b25c-9ebe4cdad697\") " pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:00:20.965921 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:20.965894 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b847dde-7257-44e0-b25c-9ebe4cdad697-workload-certs\") pod \"e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn\" (UID: \"4b847dde-7257-44e0-b25c-9ebe4cdad697\") " pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:00:20.966410 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:20.966173 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b847dde-7257-44e0-b25c-9ebe4cdad697-workload-socket\") pod \"e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn\" (UID: \"4b847dde-7257-44e0-b25c-9ebe4cdad697\") " pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:00:20.966410 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:20.966346 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b847dde-7257-44e0-b25c-9ebe4cdad697-istio-data\") pod \"e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn\" (UID: \"4b847dde-7257-44e0-b25c-9ebe4cdad697\") " pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:00:20.966700 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:20.966625 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b847dde-7257-44e0-b25c-9ebe4cdad697-credential-socket\") pod \"e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn\" (UID: \"4b847dde-7257-44e0-b25c-9ebe4cdad697\") " pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:00:20.967323 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:20.967296 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b847dde-7257-44e0-b25c-9ebe4cdad697-istiod-ca-cert\") pod \"e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn\" (UID: \"4b847dde-7257-44e0-b25c-9ebe4cdad697\") " pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:00:20.967651 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:20.967624 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn"] Nov 03 23:00:20.972911 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:20.971235 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b847dde-7257-44e0-b25c-9ebe4cdad697-istio-podinfo\") pod \"e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn\" (UID: \"4b847dde-7257-44e0-b25c-9ebe4cdad697\") " pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:00:20.973343 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:20.973316 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b847dde-7257-44e0-b25c-9ebe4cdad697-istio-envoy\") pod \"e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn\" (UID: \"4b847dde-7257-44e0-b25c-9ebe4cdad697\") " pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:00:21.035681 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:21.035585 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b847dde-7257-44e0-b25c-9ebe4cdad697-istio-token\") pod \"e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn\" (UID: \"4b847dde-7257-44e0-b25c-9ebe4cdad697\") " pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:00:21.035932 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:21.035735 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq8kl\" (UniqueName: \"kubernetes.io/projected/4b847dde-7257-44e0-b25c-9ebe4cdad697-kube-api-access-rq8kl\") pod \"e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn\" (UID: \"4b847dde-7257-44e0-b25c-9ebe4cdad697\") " pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:00:21.077764 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:21.077724 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:00:21.284246 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:21.284201 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:21.529851 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:21.529661 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn"] Nov 03 23:00:21.530031 ip-10-0-143-132 kubenswrapper[5196]: W1103 23:00:21.529929 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b847dde_7257_44e0_b25c_9ebe4cdad697.slice/crio-8b1f5a79eb4f31269c5c2895d8168abd9dcfcd4666ee61e0fb4f8aacefece256 WatchSource:0}: Error finding container 8b1f5a79eb4f31269c5c2895d8168abd9dcfcd4666ee61e0fb4f8aacefece256: Status 404 returned error can't find the container with id 8b1f5a79eb4f31269c5c2895d8168abd9dcfcd4666ee61e0fb4f8aacefece256 Nov 03 23:00:21.533698 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:21.533655 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:00:21.533809 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:21.533741 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:00:21.533809 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:21.533783 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:00:21.976451 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:21.976406 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" event={"ID":"4b847dde-7257-44e0-b25c-9ebe4cdad697","Type":"ContainerStarted","Data":"031184b6b8125e3f772103763af015ba5f2dff52af37f785d5fbdac5b41e5c00"} Nov 03 23:00:21.976980 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:21.976484 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" event={"ID":"4b847dde-7257-44e0-b25c-9ebe4cdad697","Type":"ContainerStarted","Data":"8b1f5a79eb4f31269c5c2895d8168abd9dcfcd4666ee61e0fb4f8aacefece256"} Nov 03 23:00:22.078245 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:22.078200 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:00:22.080610 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:22.080579 5196 patch_prober.go:29] interesting pod/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 23:00:22.080756 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:22.080641 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" podUID="4b847dde-7257-44e0-b25c-9ebe4cdad697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 23:00:22.102342 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:22.099426 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" podStartSLOduration=2.099412377 podStartE2EDuration="2.099412377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:22.09907553 +0000 UTC m=+2251.408634822" watchObservedRunningTime="2026-11-03 23:00:22.099412377 +0000 UTC m=+2251.408971669" Nov 03 23:00:22.385355 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:22.385318 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:23.084113 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:23.083986 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:00:23.991876 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:23.991829 5196 generic.go:356] "Generic (PLEG): container finished" podID="cef4b051-45b0-4db8-8ab5-afcfe3b9149d" containerID="154f65d83db979208f117a3eda40f081f98a9631b46bee2a47df32867f5c6966" exitCode=0 Nov 03 23:00:23.992115 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:23.991918 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" event={"ID":"cef4b051-45b0-4db8-8ab5-afcfe3b9149d","Type":"ContainerDied","Data":"154f65d83db979208f117a3eda40f081f98a9631b46bee2a47df32867f5c6966"} Nov 03 23:00:23.992267 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:23.992234 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:00:23.992682 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:23.992561 5196 scope.go:122] "RemoveContainer" containerID="154f65d83db979208f117a3eda40f081f98a9631b46bee2a47df32867f5c6966" Nov 03 23:00:23.993415 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:23.993179 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:00:23.994167 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:23.994141 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:00:23.994219 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:23.994209 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:00:23.994255 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:23.994241 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:00:24.916039 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:24.916003 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:25.021137 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:25.021094 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" event={"ID":"cef4b051-45b0-4db8-8ab5-afcfe3b9149d","Type":"ContainerStarted","Data":"0e4e30b30aad4a7021f4e93782143e182169bab3591086fdf720c750a1efacfd"} Nov 03 23:00:25.120540 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:25.120493 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:25.274725 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:25.274636 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:25.391322 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:25.391290 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 23:00:25.394880 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:25.394054 5196 patch_prober.go:29] interesting pod/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 03 23:00:25.394880 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:25.394128 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" podUID="cef4b051-45b0-4db8-8ab5-afcfe3b9149d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 03 23:00:25.659121 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:25.659090 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 23:00:25.939610 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:00:25.939530 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:00:26.415036 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:26.414994 5196 patch_prober.go:29] interesting pod/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 03 23:00:26.415261 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:26.415066 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" podUID="cef4b051-45b0-4db8-8ab5-afcfe3b9149d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 03 23:00:27.354023 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:27.353992 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr"] Nov 03 23:00:27.357544 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:27.357521 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr" Nov 03 23:00:27.389668 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:27.389637 5196 patch_prober.go:29] interesting pod/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 03 23:00:27.389803 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:27.389681 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" podUID="cef4b051-45b0-4db8-8ab5-afcfe3b9149d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 03 23:00:27.415235 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:27.415173 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ea5c8f-serving-cert\"" Nov 03 23:00:27.463465 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:27.463432 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e42253b-644f-4b19-9520-1224e7d1746d-maas-api-tls\") pod \"maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr\" (UID: \"8e42253b-644f-4b19-9520-1224e7d1746d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr" Nov 03 23:00:27.463620 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:27.463562 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxgqp\" (UniqueName: \"kubernetes.io/projected/8e42253b-644f-4b19-9520-1224e7d1746d-kube-api-access-vxgqp\") pod \"maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr\" (UID: \"8e42253b-644f-4b19-9520-1224e7d1746d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr" Nov 03 23:00:27.463620 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:27.463594 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e42253b-644f-4b19-9520-1224e7d1746d-metrics-certs\") pod \"maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr\" (UID: \"8e42253b-644f-4b19-9520-1224e7d1746d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr" Nov 03 23:00:27.463692 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:27.463641 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e42253b-644f-4b19-9520-1224e7d1746d-openshift-service-ca\") pod \"maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr\" (UID: \"8e42253b-644f-4b19-9520-1224e7d1746d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr" Nov 03 23:00:27.469155 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:27.469132 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr"] Nov 03 23:00:27.533319 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:27.533288 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:27.564795 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:27.564763 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxgqp\" (UniqueName: \"kubernetes.io/projected/8e42253b-644f-4b19-9520-1224e7d1746d-kube-api-access-vxgqp\") pod \"maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr\" (UID: \"8e42253b-644f-4b19-9520-1224e7d1746d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr" Nov 03 23:00:27.564948 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:27.564801 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e42253b-644f-4b19-9520-1224e7d1746d-metrics-certs\") pod \"maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr\" (UID: \"8e42253b-644f-4b19-9520-1224e7d1746d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr" Nov 03 23:00:27.564948 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:27.564868 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e42253b-644f-4b19-9520-1224e7d1746d-openshift-service-ca\") pod \"maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr\" (UID: \"8e42253b-644f-4b19-9520-1224e7d1746d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr" Nov 03 23:00:27.565021 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:27.564994 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e42253b-644f-4b19-9520-1224e7d1746d-maas-api-tls\") pod \"maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr\" (UID: \"8e42253b-644f-4b19-9520-1224e7d1746d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr" Nov 03 23:00:27.565156 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:00:27.565135 5196 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-ea5c8f-serving-cert: secret "maas-api-e2e-api-a-ea5c8f-serving-cert" not found Nov 03 23:00:27.565220 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:00:27.565213 5196 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e42253b-644f-4b19-9520-1224e7d1746d-maas-api-tls podName:8e42253b-644f-4b19-9520-1224e7d1746d nodeName:}" failed. No retries permitted until 2026-11-03 23:00:28.065191655 +0000 UTC m=+2257.374750928 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8e42253b-644f-4b19-9520-1224e7d1746d-maas-api-tls") pod "maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr" (UID: "8e42253b-644f-4b19-9520-1224e7d1746d") : secret "maas-api-e2e-api-a-ea5c8f-serving-cert" not found Nov 03 23:00:27.565597 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:27.565579 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e42253b-644f-4b19-9520-1224e7d1746d-openshift-service-ca\") pod \"maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr\" (UID: \"8e42253b-644f-4b19-9520-1224e7d1746d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr" Nov 03 23:00:27.567510 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:27.567490 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e42253b-644f-4b19-9520-1224e7d1746d-metrics-certs\") pod \"maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr\" (UID: \"8e42253b-644f-4b19-9520-1224e7d1746d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr" Nov 03 23:00:27.641405 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:27.641333 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxgqp\" (UniqueName: \"kubernetes.io/projected/8e42253b-644f-4b19-9520-1224e7d1746d-kube-api-access-vxgqp\") pod \"maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr\" (UID: \"8e42253b-644f-4b19-9520-1224e7d1746d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr" Nov 03 23:00:27.723158 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:27.723124 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:27.881978 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:27.881949 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:28.071448 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:28.071362 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e42253b-644f-4b19-9520-1224e7d1746d-maas-api-tls\") pod \"maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr\" (UID: \"8e42253b-644f-4b19-9520-1224e7d1746d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr" Nov 03 23:00:28.074345 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:28.074311 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e42253b-644f-4b19-9520-1224e7d1746d-maas-api-tls\") pod \"maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr\" (UID: \"8e42253b-644f-4b19-9520-1224e7d1746d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr" Nov 03 23:00:28.269998 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:28.269959 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr" Nov 03 23:00:28.412891 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:28.389928 5196 patch_prober.go:29] interesting pod/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 03 23:00:28.412891 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:28.389994 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" podUID="cef4b051-45b0-4db8-8ab5-afcfe3b9149d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 03 23:00:28.681234 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:28.681173 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr"] Nov 03 23:00:28.684550 ip-10-0-143-132 kubenswrapper[5196]: W1103 23:00:28.684499 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e42253b_644f_4b19_9520_1224e7d1746d.slice/crio-40f29e5e4d6d4f5afbf7b08298638ce2d01a0b4152db3674cd7ed476e58074d3 WatchSource:0}: Error finding container 40f29e5e4d6d4f5afbf7b08298638ce2d01a0b4152db3674cd7ed476e58074d3: Status 404 returned error can't find the container with id 40f29e5e4d6d4f5afbf7b08298638ce2d01a0b4152db3674cd7ed476e58074d3 Nov 03 23:00:28.686916 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:28.686896 5196 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 23:00:29.045904 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:29.045776 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr" event={"ID":"8e42253b-644f-4b19-9520-1224e7d1746d","Type":"ContainerStarted","Data":"40f29e5e4d6d4f5afbf7b08298638ce2d01a0b4152db3674cd7ed476e58074d3"} Nov 03 23:00:29.415333 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:29.415291 5196 patch_prober.go:29] interesting pod/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 03 23:00:29.415835 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:29.415357 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" podUID="cef4b051-45b0-4db8-8ab5-afcfe3b9149d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 03 23:00:30.053665 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:30.053627 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr" event={"ID":"8e42253b-644f-4b19-9520-1224e7d1746d","Type":"ContainerStarted","Data":"b5b8619a973adca607afdf1f0ce3ccf67721e6eda1b2a41d772aa8407339fe29"} Nov 03 23:00:30.053845 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:30.053758 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr" Nov 03 23:00:30.173712 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:30.173484 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr" podStartSLOduration=2.315440745 podStartE2EDuration="3.173470558s" totalImagesPullingTime="858.029813ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:30.173096292 +0000 UTC m=+2259.482655586" watchObservedRunningTime="2026-11-03 23:00:30.173470558 +0000 UTC m=+2259.483029851" Nov 03 23:00:30.389946 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:30.389913 5196 patch_prober.go:29] interesting pod/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 03 23:00:30.390157 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:30.389970 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" podUID="cef4b051-45b0-4db8-8ab5-afcfe3b9149d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 03 23:00:31.290242 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:31.290210 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:31.408927 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:31.405269 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 23:00:31.410970 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:31.410461 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 23:00:31.478225 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:31.475526 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:33.025166 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:33.025129 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:36.065576 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:36.065551 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr" Nov 03 23:00:37.854548 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:37.854518 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:41.984683 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:41.984633 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:43.562239 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:43.562204 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:43.709443 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:43.709411 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:43.856242 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:43.856207 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:44.111380 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:44.111306 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:44.464081 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:44.463995 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:48.997389 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:48.997338 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:49.510188 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:49.510146 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:53.919219 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:53.919184 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:55.227010 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:55.226975 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:55.919259 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:00:55.919228 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:00:56.712607 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:56.712574 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr"] Nov 03 23:00:56.713080 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:56.712789 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" podUID="dcc0cca0-e8aa-49c0-a502-25badef479aa" containerName="istio-proxy" containerID="cri-o://2d8e84c9f92d492a9a24402f80d9a79e32721350f3b1e272e933542ee06eb32c" gracePeriod=30 Nov 03 23:00:58.740180 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:58.740150 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:00:59.340693 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:59.340660 5196 generic.go:356] "Generic (PLEG): container finished" podID="c332f22b-f8b4-4b95-a2dc-bac68ef527d7" containerID="bf358276393c80b5a63eefe869e46ca83776e0e6652062f8c9b31f4967180499" exitCode=0 Nov 03 23:00:59.340693 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:59.340698 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" event={"ID":"c332f22b-f8b4-4b95-a2dc-bac68ef527d7","Type":"ContainerDied","Data":"bf358276393c80b5a63eefe869e46ca83776e0e6652062f8c9b31f4967180499"} Nov 03 23:00:59.341152 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:59.341134 5196 scope.go:122] "RemoveContainer" containerID="bf358276393c80b5a63eefe869e46ca83776e0e6652062f8c9b31f4967180499" Nov 03 23:00:59.342897 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:59.342829 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:00:59.342972 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:59.342942 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:00:59.343010 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:00:59.342980 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:01:00.349646 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:00.349611 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" event={"ID":"c332f22b-f8b4-4b95-a2dc-bac68ef527d7","Type":"ContainerStarted","Data":"8a056dc2ca33477d802a1ccc1ac8d74e8b417b361bbfffee81d9b79a5a2c54ef"} Nov 03 23:01:00.857360 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:00.857331 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 23:01:00.857538 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:00.857488 5196 patch_prober.go:29] interesting pod/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 03 23:01:00.857585 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:00.857531 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" podUID="c332f22b-f8b4-4b95-a2dc-bac68ef527d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 03 23:01:01.545543 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:01.545510 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:01.763447 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:01.763414 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:01.801933 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:01.801323 5196 patch_prober.go:29] interesting pod/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 23:01:01.801933 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:01.801414 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" podUID="dcc0cca0-e8aa-49c0-a502-25badef479aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 23:01:01.857473 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:01.857326 5196 patch_prober.go:29] interesting pod/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 03 23:01:01.857473 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:01.857407 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" podUID="c332f22b-f8b4-4b95-a2dc-bac68ef527d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 03 23:01:02.019296 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.019155 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:02.030675 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.030645 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" Nov 03 23:01:02.038706 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.038672 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcc0cca0-e8aa-49c0-a502-25badef479aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcc0cca0-e8aa-49c0-a502-25badef479aa-istio-podinfo\") pod \"dcc0cca0-e8aa-49c0-a502-25badef479aa\" (UID: \"dcc0cca0-e8aa-49c0-a502-25badef479aa\") " Nov 03 23:01:02.038873 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.038763 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcc0cca0-e8aa-49c0-a502-25badef479aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcc0cca0-e8aa-49c0-a502-25badef479aa-istiod-ca-cert\") pod \"dcc0cca0-e8aa-49c0-a502-25badef479aa\" (UID: \"dcc0cca0-e8aa-49c0-a502-25badef479aa\") " Nov 03 23:01:02.038955 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.038926 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcc0cca0-e8aa-49c0-a502-25badef479aa-istio-token\" (UniqueName: \"kubernetes.io/projected/dcc0cca0-e8aa-49c0-a502-25badef479aa-istio-token\") pod \"dcc0cca0-e8aa-49c0-a502-25badef479aa\" (UID: \"dcc0cca0-e8aa-49c0-a502-25badef479aa\") " Nov 03 23:01:02.039006 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.038987 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcc0cca0-e8aa-49c0-a502-25badef479aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcc0cca0-e8aa-49c0-a502-25badef479aa-istio-data\") pod \"dcc0cca0-e8aa-49c0-a502-25badef479aa\" (UID: \"dcc0cca0-e8aa-49c0-a502-25badef479aa\") " Nov 03 23:01:02.039065 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.039012 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcc0cca0-e8aa-49c0-a502-25badef479aa-kube-api-access-hgtlz\" (UniqueName: \"kubernetes.io/projected/dcc0cca0-e8aa-49c0-a502-25badef479aa-kube-api-access-hgtlz\") pod \"dcc0cca0-e8aa-49c0-a502-25badef479aa\" (UID: \"dcc0cca0-e8aa-49c0-a502-25badef479aa\") " Nov 03 23:01:02.039203 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.039181 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcc0cca0-e8aa-49c0-a502-25badef479aa-istiod-ca-cert" pod "dcc0cca0-e8aa-49c0-a502-25badef479aa" (UID: "dcc0cca0-e8aa-49c0-a502-25badef479aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:01:02.039279 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.039258 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc0cca0-e8aa-49c0-a502-25badef479aa-istio-data" pod "dcc0cca0-e8aa-49c0-a502-25badef479aa" (UID: "dcc0cca0-e8aa-49c0-a502-25badef479aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:02.039323 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.039208 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcc0cca0-e8aa-49c0-a502-25badef479aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc0cca0-e8aa-49c0-a502-25badef479aa-credential-socket\") pod \"dcc0cca0-e8aa-49c0-a502-25badef479aa\" (UID: \"dcc0cca0-e8aa-49c0-a502-25badef479aa\") " Nov 03 23:01:02.040115 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.039350 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcc0cca0-e8aa-49c0-a502-25badef479aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcc0cca0-e8aa-49c0-a502-25badef479aa-istio-envoy\") pod \"dcc0cca0-e8aa-49c0-a502-25badef479aa\" (UID: \"dcc0cca0-e8aa-49c0-a502-25badef479aa\") " Nov 03 23:01:02.040115 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.039381 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc0cca0-e8aa-49c0-a502-25badef479aa-credential-socket" pod "dcc0cca0-e8aa-49c0-a502-25badef479aa" (UID: "dcc0cca0-e8aa-49c0-a502-25badef479aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:02.040115 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.039390 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcc0cca0-e8aa-49c0-a502-25badef479aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc0cca0-e8aa-49c0-a502-25badef479aa-workload-socket\") pod \"dcc0cca0-e8aa-49c0-a502-25badef479aa\" (UID: \"dcc0cca0-e8aa-49c0-a502-25badef479aa\") " Nov 03 23:01:02.040115 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.039451 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcc0cca0-e8aa-49c0-a502-25badef479aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcc0cca0-e8aa-49c0-a502-25badef479aa-workload-certs\") pod \"dcc0cca0-e8aa-49c0-a502-25badef479aa\" (UID: \"dcc0cca0-e8aa-49c0-a502-25badef479aa\") " Nov 03 23:01:02.040115 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.039898 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc0cca0-e8aa-49c0-a502-25badef479aa-workload-socket" pod "dcc0cca0-e8aa-49c0-a502-25badef479aa" (UID: "dcc0cca0-e8aa-49c0-a502-25badef479aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:02.040115 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.039937 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc0cca0-e8aa-49c0-a502-25badef479aa-workload-certs" pod "dcc0cca0-e8aa-49c0-a502-25badef479aa" (UID: "dcc0cca0-e8aa-49c0-a502-25badef479aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:02.040115 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.040054 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcc0cca0-e8aa-49c0-a502-25badef479aa-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:02.040115 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.040073 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc0cca0-e8aa-49c0-a502-25badef479aa-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:02.040115 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.040085 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc0cca0-e8aa-49c0-a502-25badef479aa-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:02.040115 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.040097 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcc0cca0-e8aa-49c0-a502-25badef479aa-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:02.040115 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.040110 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcc0cca0-e8aa-49c0-a502-25badef479aa-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:02.041629 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.041601 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcc0cca0-e8aa-49c0-a502-25badef479aa-istio-podinfo" pod "dcc0cca0-e8aa-49c0-a502-25badef479aa" (UID: "dcc0cca0-e8aa-49c0-a502-25badef479aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:01:02.041917 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.041883 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcc0cca0-e8aa-49c0-a502-25badef479aa-kube-api-access-hgtlz" pod "dcc0cca0-e8aa-49c0-a502-25badef479aa" (UID: "dcc0cca0-e8aa-49c0-a502-25badef479aa"). InnerVolumeSpecName "kube-api-access-hgtlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:02.042013 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.041971 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcc0cca0-e8aa-49c0-a502-25badef479aa-istio-token" pod "dcc0cca0-e8aa-49c0-a502-25badef479aa" (UID: "dcc0cca0-e8aa-49c0-a502-25badef479aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:02.042737 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.042711 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc0cca0-e8aa-49c0-a502-25badef479aa-istio-envoy" pod "dcc0cca0-e8aa-49c0-a502-25badef479aa" (UID: "dcc0cca0-e8aa-49c0-a502-25badef479aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:02.141419 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.141388 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcc0cca0-e8aa-49c0-a502-25badef479aa-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:02.141419 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.141416 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgtlz\" (UniqueName: \"kubernetes.io/projected/dcc0cca0-e8aa-49c0-a502-25badef479aa-kube-api-access-hgtlz\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:02.141419 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.141427 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcc0cca0-e8aa-49c0-a502-25badef479aa-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:02.141776 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.141438 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcc0cca0-e8aa-49c0-a502-25badef479aa-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:02.364135 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.364105 5196 generic.go:356] "Generic (PLEG): container finished" podID="dcc0cca0-e8aa-49c0-a502-25badef479aa" containerID="2d8e84c9f92d492a9a24402f80d9a79e32721350f3b1e272e933542ee06eb32c" exitCode=0 Nov 03 23:01:02.364454 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.364440 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" event={"ID":"dcc0cca0-e8aa-49c0-a502-25badef479aa","Type":"ContainerDied","Data":"2d8e84c9f92d492a9a24402f80d9a79e32721350f3b1e272e933542ee06eb32c"} Nov 03 23:01:02.364551 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.364541 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" event={"ID":"dcc0cca0-e8aa-49c0-a502-25badef479aa","Type":"ContainerDied","Data":"df1e5ebe7aecbb73f9c14742e6b53d8c70702b04b9f8065bf30d5c050a386c08"} Nov 03 23:01:02.364647 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.364617 5196 scope.go:122] "RemoveContainer" containerID="2d8e84c9f92d492a9a24402f80d9a79e32721350f3b1e272e933542ee06eb32c" Nov 03 23:01:02.365012 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.364994 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr" Nov 03 23:01:02.383593 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.382261 5196 scope.go:122] "RemoveContainer" containerID="2d8e84c9f92d492a9a24402f80d9a79e32721350f3b1e272e933542ee06eb32c" Nov 03 23:01:02.383593 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:01:02.382995 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d8e84c9f92d492a9a24402f80d9a79e32721350f3b1e272e933542ee06eb32c\": container with ID starting with 2d8e84c9f92d492a9a24402f80d9a79e32721350f3b1e272e933542ee06eb32c not found: ID does not exist" containerID="2d8e84c9f92d492a9a24402f80d9a79e32721350f3b1e272e933542ee06eb32c" Nov 03 23:01:02.383593 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.383033 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d8e84c9f92d492a9a24402f80d9a79e32721350f3b1e272e933542ee06eb32c"} err="failed to get container status \"2d8e84c9f92d492a9a24402f80d9a79e32721350f3b1e272e933542ee06eb32c\": rpc error: code = NotFound desc = could not find container \"2d8e84c9f92d492a9a24402f80d9a79e32721350f3b1e272e933542ee06eb32c\": container with ID starting with 2d8e84c9f92d492a9a24402f80d9a79e32721350f3b1e272e933542ee06eb32c not found: ID does not exist" Nov 03 23:01:02.545469 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.545301 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr"] Nov 03 23:01:02.548628 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.548602 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1d31f8-openshift-default-69fb4dfb5c-pwwqr"] Nov 03 23:01:02.826405 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.826319 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcc0cca0-e8aa-49c0-a502-25badef479aa" path="/var/lib/kubelet/pods/dcc0cca0-e8aa-49c0-a502-25badef479aa/volumes" Nov 03 23:01:02.857087 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.857055 5196 patch_prober.go:29] interesting pod/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 03 23:01:02.857255 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.857105 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" podUID="c332f22b-f8b4-4b95-a2dc-bac68ef527d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 03 23:01:02.971918 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:02.971886 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:03.200878 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:03.196842 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:03.353685 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:03.353653 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:03.857845 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:03.857806 5196 patch_prober.go:29] interesting pod/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 03 23:01:03.858374 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:03.857893 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" podUID="c332f22b-f8b4-4b95-a2dc-bac68ef527d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 03 23:01:04.055282 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:04.055246 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:04.857066 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:04.856915 5196 patch_prober.go:29] interesting pod/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 03 23:01:04.857066 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:04.856976 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" podUID="c332f22b-f8b4-4b95-a2dc-bac68ef527d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 03 23:01:05.914729 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:05.914699 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 23:01:05.915329 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:05.915310 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 23:01:05.916526 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:05.916503 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 23:01:07.926924 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:07.926827 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:08.116807 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:08.116771 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:08.276256 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:08.276162 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:09.235286 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:09.235251 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:09.555104 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:09.555026 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:11.311528 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:11.311449 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:11.497934 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:11.497897 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:14.441374 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:14.441340 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:14.937596 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:14.937567 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:19.504612 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:19.504575 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:24.379381 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:24.379350 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s"] Nov 03 23:01:24.380067 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:24.380046 5196 cpu_manager.go:414] "containerMap: removing container" podUID="dcc0cca0-e8aa-49c0-a502-25badef479aa" containerName="istio-proxy" Nov 03 23:01:24.380172 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:24.380077 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcc0cca0-e8aa-49c0-a502-25badef479aa" containerName="istio-proxy" Nov 03 23:01:24.380208 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:24.380183 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcc0cca0-e8aa-49c0-a502-25badef479aa" containerName="istio-proxy" Nov 03 23:01:24.382661 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:24.382646 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s" Nov 03 23:01:24.435948 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:24.435910 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-e66d59-serving-cert\"" Nov 03 23:01:24.467762 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:24.467731 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41e5ef76-ed91-4c59-a3e5-36a288fbc075-maas-api-tls\") pod \"maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s\" (UID: \"41e5ef76-ed91-4c59-a3e5-36a288fbc075\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s" Nov 03 23:01:24.467954 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:24.467785 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p24lq\" (UniqueName: \"kubernetes.io/projected/41e5ef76-ed91-4c59-a3e5-36a288fbc075-kube-api-access-p24lq\") pod \"maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s\" (UID: \"41e5ef76-ed91-4c59-a3e5-36a288fbc075\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s" Nov 03 23:01:24.467954 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:24.467910 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41e5ef76-ed91-4c59-a3e5-36a288fbc075-metrics-certs\") pod \"maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s\" (UID: \"41e5ef76-ed91-4c59-a3e5-36a288fbc075\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s" Nov 03 23:01:24.468039 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:24.468018 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e5ef76-ed91-4c59-a3e5-36a288fbc075-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s\" (UID: \"41e5ef76-ed91-4c59-a3e5-36a288fbc075\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s" Nov 03 23:01:24.511880 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:24.490266 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s"] Nov 03 23:01:24.569538 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:24.569507 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p24lq\" (UniqueName: \"kubernetes.io/projected/41e5ef76-ed91-4c59-a3e5-36a288fbc075-kube-api-access-p24lq\") pod \"maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s\" (UID: \"41e5ef76-ed91-4c59-a3e5-36a288fbc075\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s" Nov 03 23:01:24.569727 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:24.569565 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41e5ef76-ed91-4c59-a3e5-36a288fbc075-metrics-certs\") pod \"maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s\" (UID: \"41e5ef76-ed91-4c59-a3e5-36a288fbc075\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s" Nov 03 23:01:24.569727 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:24.569642 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e5ef76-ed91-4c59-a3e5-36a288fbc075-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s\" (UID: \"41e5ef76-ed91-4c59-a3e5-36a288fbc075\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s" Nov 03 23:01:24.569884 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:24.569763 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41e5ef76-ed91-4c59-a3e5-36a288fbc075-maas-api-tls\") pod \"maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s\" (UID: \"41e5ef76-ed91-4c59-a3e5-36a288fbc075\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s" Nov 03 23:01:24.569940 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:01:24.569909 5196 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-e66d59-serving-cert: secret "maas-api-e2e-inf-b-e66d59-serving-cert" not found Nov 03 23:01:24.570020 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:01:24.570008 5196 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41e5ef76-ed91-4c59-a3e5-36a288fbc075-maas-api-tls podName:41e5ef76-ed91-4c59-a3e5-36a288fbc075 nodeName:}" failed. No retries permitted until 2026-11-03 23:01:25.069986289 +0000 UTC m=+2314.379545560 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/41e5ef76-ed91-4c59-a3e5-36a288fbc075-maas-api-tls") pod "maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s" (UID: "41e5ef76-ed91-4c59-a3e5-36a288fbc075") : secret "maas-api-e2e-inf-b-e66d59-serving-cert" not found Nov 03 23:01:24.570420 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:24.570395 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e5ef76-ed91-4c59-a3e5-36a288fbc075-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s\" (UID: \"41e5ef76-ed91-4c59-a3e5-36a288fbc075\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s" Nov 03 23:01:24.572366 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:24.572344 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41e5ef76-ed91-4c59-a3e5-36a288fbc075-metrics-certs\") pod \"maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s\" (UID: \"41e5ef76-ed91-4c59-a3e5-36a288fbc075\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s" Nov 03 23:01:24.574544 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:24.574522 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:24.635269 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:24.635126 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p24lq\" (UniqueName: \"kubernetes.io/projected/41e5ef76-ed91-4c59-a3e5-36a288fbc075-kube-api-access-p24lq\") pod \"maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s\" (UID: \"41e5ef76-ed91-4c59-a3e5-36a288fbc075\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s" Nov 03 23:01:24.738685 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:24.738645 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:24.888113 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:24.888029 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:25.076003 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:25.075974 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41e5ef76-ed91-4c59-a3e5-36a288fbc075-maas-api-tls\") pod \"maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s\" (UID: \"41e5ef76-ed91-4c59-a3e5-36a288fbc075\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s" Nov 03 23:01:25.078479 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:25.078456 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41e5ef76-ed91-4c59-a3e5-36a288fbc075-maas-api-tls\") pod \"maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s\" (UID: \"41e5ef76-ed91-4c59-a3e5-36a288fbc075\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s" Nov 03 23:01:25.316323 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:25.316229 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s" Nov 03 23:01:25.734450 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:25.734419 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s"] Nov 03 23:01:25.745938 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:25.745909 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:25.957379 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:01:25.957344 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:01:26.593695 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:26.593656 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s" event={"ID":"41e5ef76-ed91-4c59-a3e5-36a288fbc075","Type":"ContainerStarted","Data":"1d63df83cb701fad6c63dc871562c06843bfe7c768b900aa07cf439ca5611f36"} Nov 03 23:01:26.628273 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:26.628248 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:27.505749 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:27.505662 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:27.626496 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:27.626461 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s" event={"ID":"41e5ef76-ed91-4c59-a3e5-36a288fbc075","Type":"ContainerStarted","Data":"d28feca9800963b68e65bc185de8fb38358d45436094b77d00af0679f2181800"} Nov 03 23:01:27.626710 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:27.626602 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s" Nov 03 23:01:27.804885 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:27.804755 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s" podStartSLOduration=2.963458753 podStartE2EDuration="3.80473576s" totalImagesPullingTime="841.277007ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:01:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:01:27.800440092 +0000 UTC m=+2317.109999389" watchObservedRunningTime="2026-11-03 23:01:27.80473576 +0000 UTC m=+2317.114295054" Nov 03 23:01:30.097957 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:30.097924 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:32.418503 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:32.418435 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:32.570509 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:32.570475 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:32.757147 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:32.757063 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt"] Nov 03 23:01:32.757616 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:32.757582 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" podUID="625eb3b0-400d-4fa7-b5e7-80a08e17a42e" containerName="istio-proxy" containerID="cri-o://90d15a672be3ebd5c87943d5230353a982153933c55a33009bdc1af10c7b5517" gracePeriod=30 Nov 03 23:01:32.768067 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:32.767991 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9"] Nov 03 23:01:32.768235 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:32.768212 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" podUID="4deb89b6-4d18-4423-a47e-6ab4c477145d" containerName="istio-proxy" containerID="cri-o://660620dd08ad7d07fe7b886864e14355268aaaed7f0c36db17beb5fdc45280c3" gracePeriod=30 Nov 03 23:01:32.779766 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:32.779736 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:32.802227 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:32.802200 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6"] Nov 03 23:01:32.802468 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:32.802442 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" podUID="a6166d64-362f-4ada-b2fd-ecf68bc023a7" containerName="istio-proxy" containerID="cri-o://fd25979b4993272a6e065da57b2cd211a2f592f762ef7a886a246f93820166cc" gracePeriod=30 Nov 03 23:01:32.804994 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:32.804969 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t"] Nov 03 23:01:32.805160 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:32.805136 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" podUID="9d0e7774-f0b0-427d-bbd4-7070d70b8fca" containerName="istio-proxy" containerID="cri-o://c52e75cd15a3b17b8ddcc2930732c1275843e2434da7cf7c47b4a8205e8ffc75" gracePeriod=30 Nov 03 23:01:33.202579 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:33.202549 5196 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 23:01:33.202790 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:33.202600 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" podUID="9d0e7774-f0b0-427d-bbd4-7070d70b8fca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 23:01:33.443846 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:33.443812 5196 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 23:01:33.444242 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:33.443893 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" podUID="a6166d64-362f-4ada-b2fd-ecf68bc023a7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 23:01:33.530699 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:33.530613 5196 patch_prober.go:29] interesting pod/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 23:01:33.530699 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:33.530669 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" podUID="4deb89b6-4d18-4423-a47e-6ab4c477145d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 23:01:33.638369 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:33.638346 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s" Nov 03 23:01:33.672479 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:33.672453 5196 generic.go:356] "Generic (PLEG): container finished" podID="0d375a3f-2f84-4495-a683-42a98477ff52" containerID="162270799d34e633401238397fd27799cf07fc10e603e751e08527bd0178b50a" exitCode=0 Nov 03 23:01:33.672611 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:33.672524 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" event={"ID":"0d375a3f-2f84-4495-a683-42a98477ff52","Type":"ContainerDied","Data":"162270799d34e633401238397fd27799cf07fc10e603e751e08527bd0178b50a"} Nov 03 23:01:33.672901 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:33.672887 5196 scope.go:122] "RemoveContainer" containerID="162270799d34e633401238397fd27799cf07fc10e603e751e08527bd0178b50a" Nov 03 23:01:33.674536 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:33.674507 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:01:33.674633 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:33.674572 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:01:33.674633 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:33.674604 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:01:34.681310 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:34.681274 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" event={"ID":"0d375a3f-2f84-4495-a683-42a98477ff52","Type":"ContainerStarted","Data":"ffbd4f361bac15f9b185f79737d3e7b7fa9dc6cabe4e44eefdcb8c2b61c3b03c"} Nov 03 23:01:35.117383 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:35.117347 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:35.657603 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:35.657572 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 23:01:35.658716 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:35.658689 5196 patch_prober.go:29] interesting pod/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 03 23:01:35.658871 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:35.658735 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" podUID="0d375a3f-2f84-4495-a683-42a98477ff52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 03 23:01:36.383868 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:36.383835 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 23:01:36.658302 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:36.658229 5196 patch_prober.go:29] interesting pod/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 03 23:01:36.658302 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:36.658287 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" podUID="0d375a3f-2f84-4495-a683-42a98477ff52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 03 23:01:37.657623 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:37.657587 5196 patch_prober.go:29] interesting pod/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 03 23:01:37.658048 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:37.657649 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" podUID="0d375a3f-2f84-4495-a683-42a98477ff52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 03 23:01:38.130879 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.130724 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" Nov 03 23:01:38.187113 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.187079 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" Nov 03 23:01:38.223618 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.223397 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-istio-podinfo\") pod \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\" (UID: \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\") " Nov 03 23:01:38.226077 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.224400 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-istiod-ca-cert\") pod \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\" (UID: \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\") " Nov 03 23:01:38.226077 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.224459 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-istiod-ca-cert\") pod \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\" (UID: \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\") " Nov 03 23:01:38.226077 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.224500 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-istio-podinfo\") pod \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\" (UID: \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\") " Nov 03 23:01:38.226077 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.224539 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-workload-socket\") pod \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\" (UID: \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\") " Nov 03 23:01:38.226077 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.224594 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-istio-data\") pod \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\" (UID: \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\") " Nov 03 23:01:38.226077 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.224641 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-istio-data\") pod \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\" (UID: \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\") " Nov 03 23:01:38.226077 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.224741 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-workload-socket\") pod \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\" (UID: \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\") " Nov 03 23:01:38.226077 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.224789 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-istio-envoy\") pod \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\" (UID: \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\") " Nov 03 23:01:38.226077 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.224818 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-workload-certs\") pod \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\" (UID: \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\") " Nov 03 23:01:38.226077 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.224851 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-workload-certs\") pod \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\" (UID: \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\") " Nov 03 23:01:38.226077 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.224902 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-istio-envoy\") pod \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\" (UID: \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\") " Nov 03 23:01:38.226077 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.224912 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-istiod-ca-cert" pod "625eb3b0-400d-4fa7-b5e7-80a08e17a42e" (UID: "625eb3b0-400d-4fa7-b5e7-80a08e17a42e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:01:38.226077 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.224954 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-credential-socket\") pod \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\" (UID: \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\") " Nov 03 23:01:38.226077 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.224995 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-kube-api-access-qx8lm\" (UniqueName: \"kubernetes.io/projected/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-kube-api-access-qx8lm\") pod \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\" (UID: \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\") " Nov 03 23:01:38.226077 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.225016 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-kube-api-access-gmbgb\" (UniqueName: \"kubernetes.io/projected/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-kube-api-access-gmbgb\") pod \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\" (UID: \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\") " Nov 03 23:01:38.226077 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.225035 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-istio-token\" (UniqueName: \"kubernetes.io/projected/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-istio-token\") pod \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\" (UID: \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\") " Nov 03 23:01:38.226077 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.225123 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-istio-token\" (UniqueName: \"kubernetes.io/projected/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-istio-token\") pod \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\" (UID: \"625eb3b0-400d-4fa7-b5e7-80a08e17a42e\") " Nov 03 23:01:38.226077 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.225178 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-credential-socket\") pod \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\" (UID: \"9d0e7774-f0b0-427d-bbd4-7070d70b8fca\") " Nov 03 23:01:38.226077 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.226060 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-workload-socket" pod "625eb3b0-400d-4fa7-b5e7-80a08e17a42e" (UID: "625eb3b0-400d-4fa7-b5e7-80a08e17a42e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:38.227514 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.226319 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-workload-certs" pod "625eb3b0-400d-4fa7-b5e7-80a08e17a42e" (UID: "625eb3b0-400d-4fa7-b5e7-80a08e17a42e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:38.227514 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.226736 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-istio-data" pod "9d0e7774-f0b0-427d-bbd4-7070d70b8fca" (UID: "9d0e7774-f0b0-427d-bbd4-7070d70b8fca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:38.227514 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.226967 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-istio-data" pod "625eb3b0-400d-4fa7-b5e7-80a08e17a42e" (UID: "625eb3b0-400d-4fa7-b5e7-80a08e17a42e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:38.227514 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.226985 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-workload-socket" pod "9d0e7774-f0b0-427d-bbd4-7070d70b8fca" (UID: "9d0e7774-f0b0-427d-bbd4-7070d70b8fca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:38.227514 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.227296 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-workload-certs" pod "9d0e7774-f0b0-427d-bbd4-7070d70b8fca" (UID: "9d0e7774-f0b0-427d-bbd4-7070d70b8fca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:38.227514 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.227342 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-credential-socket" pod "625eb3b0-400d-4fa7-b5e7-80a08e17a42e" (UID: "625eb3b0-400d-4fa7-b5e7-80a08e17a42e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:38.227875 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.227566 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-istiod-ca-cert" pod "9d0e7774-f0b0-427d-bbd4-7070d70b8fca" (UID: "9d0e7774-f0b0-427d-bbd4-7070d70b8fca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:01:38.229042 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.228191 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-credential-socket" pod "9d0e7774-f0b0-427d-bbd4-7070d70b8fca" (UID: "9d0e7774-f0b0-427d-bbd4-7070d70b8fca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:38.229042 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.228445 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.229042 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.228468 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.229042 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.228484 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.229042 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.228565 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.229042 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.228673 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.229042 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.228687 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.229042 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.228701 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.229042 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.228715 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.229042 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.228730 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.229042 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.228744 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.229042 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.228892 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" Nov 03 23:01:38.229042 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.229008 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-istio-podinfo" pod "9d0e7774-f0b0-427d-bbd4-7070d70b8fca" (UID: "9d0e7774-f0b0-427d-bbd4-7070d70b8fca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:01:38.230694 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.230661 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-istio-podinfo" pod "625eb3b0-400d-4fa7-b5e7-80a08e17a42e" (UID: "625eb3b0-400d-4fa7-b5e7-80a08e17a42e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:01:38.231717 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.231624 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-istio-envoy" pod "9d0e7774-f0b0-427d-bbd4-7070d70b8fca" (UID: "9d0e7774-f0b0-427d-bbd4-7070d70b8fca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:38.231717 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.231668 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-kube-api-access-qx8lm" pod "625eb3b0-400d-4fa7-b5e7-80a08e17a42e" (UID: "625eb3b0-400d-4fa7-b5e7-80a08e17a42e"). InnerVolumeSpecName "kube-api-access-qx8lm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:38.231717 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.231692 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-istio-envoy" pod "625eb3b0-400d-4fa7-b5e7-80a08e17a42e" (UID: "625eb3b0-400d-4fa7-b5e7-80a08e17a42e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:38.234278 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.233996 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-istio-token" pod "625eb3b0-400d-4fa7-b5e7-80a08e17a42e" (UID: "625eb3b0-400d-4fa7-b5e7-80a08e17a42e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:38.234278 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.234185 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-istio-token" pod "9d0e7774-f0b0-427d-bbd4-7070d70b8fca" (UID: "9d0e7774-f0b0-427d-bbd4-7070d70b8fca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:38.234278 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.234244 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-kube-api-access-gmbgb" pod "9d0e7774-f0b0-427d-bbd4-7070d70b8fca" (UID: "9d0e7774-f0b0-427d-bbd4-7070d70b8fca"). InnerVolumeSpecName "kube-api-access-gmbgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:38.271167 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.271101 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" Nov 03 23:01:38.329688 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.329649 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6166d64-362f-4ada-b2fd-ecf68bc023a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6166d64-362f-4ada-b2fd-ecf68bc023a7-istio-data\") pod \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\" (UID: \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\") " Nov 03 23:01:38.329688 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.329686 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4deb89b6-4d18-4423-a47e-6ab4c477145d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4deb89b6-4d18-4423-a47e-6ab4c477145d-istio-envoy\") pod \"4deb89b6-4d18-4423-a47e-6ab4c477145d\" (UID: \"4deb89b6-4d18-4423-a47e-6ab4c477145d\") " Nov 03 23:01:38.330013 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.329735 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4deb89b6-4d18-4423-a47e-6ab4c477145d-istio-token\" (UniqueName: \"kubernetes.io/projected/4deb89b6-4d18-4423-a47e-6ab4c477145d-istio-token\") pod \"4deb89b6-4d18-4423-a47e-6ab4c477145d\" (UID: \"4deb89b6-4d18-4423-a47e-6ab4c477145d\") " Nov 03 23:01:38.330013 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.329759 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6166d64-362f-4ada-b2fd-ecf68bc023a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6166d64-362f-4ada-b2fd-ecf68bc023a7-istio-envoy\") pod \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\" (UID: \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\") " Nov 03 23:01:38.330013 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.329807 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6166d64-362f-4ada-b2fd-ecf68bc023a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6166d64-362f-4ada-b2fd-ecf68bc023a7-credential-socket\") pod \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\" (UID: \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\") " Nov 03 23:01:38.330013 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.329849 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6166d64-362f-4ada-b2fd-ecf68bc023a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6166d64-362f-4ada-b2fd-ecf68bc023a7-workload-socket\") pod \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\" (UID: \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\") " Nov 03 23:01:38.330013 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.329896 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4deb89b6-4d18-4423-a47e-6ab4c477145d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4deb89b6-4d18-4423-a47e-6ab4c477145d-istio-data\") pod \"4deb89b6-4d18-4423-a47e-6ab4c477145d\" (UID: \"4deb89b6-4d18-4423-a47e-6ab4c477145d\") " Nov 03 23:01:38.330013 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.329924 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6166d64-362f-4ada-b2fd-ecf68bc023a7-istio-data" pod "a6166d64-362f-4ada-b2fd-ecf68bc023a7" (UID: "a6166d64-362f-4ada-b2fd-ecf68bc023a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:38.330013 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.329928 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6166d64-362f-4ada-b2fd-ecf68bc023a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6166d64-362f-4ada-b2fd-ecf68bc023a7-workload-certs\") pod \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\" (UID: \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\") " Nov 03 23:01:38.330013 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.329996 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4deb89b6-4d18-4423-a47e-6ab4c477145d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4deb89b6-4d18-4423-a47e-6ab4c477145d-istiod-ca-cert\") pod \"4deb89b6-4d18-4423-a47e-6ab4c477145d\" (UID: \"4deb89b6-4d18-4423-a47e-6ab4c477145d\") " Nov 03 23:01:38.331796 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.330025 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6166d64-362f-4ada-b2fd-ecf68bc023a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6166d64-362f-4ada-b2fd-ecf68bc023a7-istio-podinfo\") pod \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\" (UID: \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\") " Nov 03 23:01:38.331796 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.330116 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6166d64-362f-4ada-b2fd-ecf68bc023a7-workload-certs" pod "a6166d64-362f-4ada-b2fd-ecf68bc023a7" (UID: "a6166d64-362f-4ada-b2fd-ecf68bc023a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:38.331796 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.330293 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6166d64-362f-4ada-b2fd-ecf68bc023a7-istio-token\" (UniqueName: \"kubernetes.io/projected/a6166d64-362f-4ada-b2fd-ecf68bc023a7-istio-token\") pod \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\" (UID: \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\") " Nov 03 23:01:38.331796 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.330325 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6166d64-362f-4ada-b2fd-ecf68bc023a7-workload-socket" pod "a6166d64-362f-4ada-b2fd-ecf68bc023a7" (UID: "a6166d64-362f-4ada-b2fd-ecf68bc023a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:38.331796 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.330356 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6166d64-362f-4ada-b2fd-ecf68bc023a7-credential-socket" pod "a6166d64-362f-4ada-b2fd-ecf68bc023a7" (UID: "a6166d64-362f-4ada-b2fd-ecf68bc023a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:38.331796 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.330370 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4deb89b6-4d18-4423-a47e-6ab4c477145d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4deb89b6-4d18-4423-a47e-6ab4c477145d-workload-socket\") pod \"4deb89b6-4d18-4423-a47e-6ab4c477145d\" (UID: \"4deb89b6-4d18-4423-a47e-6ab4c477145d\") " Nov 03 23:01:38.331796 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.330408 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6166d64-362f-4ada-b2fd-ecf68bc023a7-kube-api-access-t29bc\" (UniqueName: \"kubernetes.io/projected/a6166d64-362f-4ada-b2fd-ecf68bc023a7-kube-api-access-t29bc\") pod \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\" (UID: \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\") " Nov 03 23:01:38.331796 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.330466 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6166d64-362f-4ada-b2fd-ecf68bc023a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6166d64-362f-4ada-b2fd-ecf68bc023a7-istiod-ca-cert\") pod \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\" (UID: \"a6166d64-362f-4ada-b2fd-ecf68bc023a7\") " Nov 03 23:01:38.331796 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.330498 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4deb89b6-4d18-4423-a47e-6ab4c477145d-kube-api-access-dwbjg\" (UniqueName: \"kubernetes.io/projected/4deb89b6-4d18-4423-a47e-6ab4c477145d-kube-api-access-dwbjg\") pod \"4deb89b6-4d18-4423-a47e-6ab4c477145d\" (UID: \"4deb89b6-4d18-4423-a47e-6ab4c477145d\") " Nov 03 23:01:38.331796 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.330535 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4deb89b6-4d18-4423-a47e-6ab4c477145d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4deb89b6-4d18-4423-a47e-6ab4c477145d-istio-podinfo\") pod \"4deb89b6-4d18-4423-a47e-6ab4c477145d\" (UID: \"4deb89b6-4d18-4423-a47e-6ab4c477145d\") " Nov 03 23:01:38.331796 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.330568 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4deb89b6-4d18-4423-a47e-6ab4c477145d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4deb89b6-4d18-4423-a47e-6ab4c477145d-credential-socket\") pod \"4deb89b6-4d18-4423-a47e-6ab4c477145d\" (UID: \"4deb89b6-4d18-4423-a47e-6ab4c477145d\") " Nov 03 23:01:38.331796 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.330612 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4deb89b6-4d18-4423-a47e-6ab4c477145d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4deb89b6-4d18-4423-a47e-6ab4c477145d-workload-certs\") pod \"4deb89b6-4d18-4423-a47e-6ab4c477145d\" (UID: \"4deb89b6-4d18-4423-a47e-6ab4c477145d\") " Nov 03 23:01:38.331796 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.331071 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6166d64-362f-4ada-b2fd-ecf68bc023a7-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.331796 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.331092 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6166d64-362f-4ada-b2fd-ecf68bc023a7-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.331796 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.331105 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6166d64-362f-4ada-b2fd-ecf68bc023a7-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.331796 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.331115 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.331796 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.331124 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.331796 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.331134 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx8lm\" (UniqueName: \"kubernetes.io/projected/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-kube-api-access-qx8lm\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.331796 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.331143 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmbgb\" (UniqueName: \"kubernetes.io/projected/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-kube-api-access-gmbgb\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.331796 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.331137 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4deb89b6-4d18-4423-a47e-6ab4c477145d-workload-socket" pod "4deb89b6-4d18-4423-a47e-6ab4c477145d" (UID: "4deb89b6-4d18-4423-a47e-6ab4c477145d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:38.331796 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.331153 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.331796 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.331166 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.331796 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.331178 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d0e7774-f0b0-427d-bbd4-7070d70b8fca-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.331796 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.331186 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/625eb3b0-400d-4fa7-b5e7-80a08e17a42e-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.331796 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.331195 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6166d64-362f-4ada-b2fd-ecf68bc023a7-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.331796 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.331432 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4deb89b6-4d18-4423-a47e-6ab4c477145d-istiod-ca-cert" pod "4deb89b6-4d18-4423-a47e-6ab4c477145d" (UID: "4deb89b6-4d18-4423-a47e-6ab4c477145d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:01:38.331796 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.331465 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4deb89b6-4d18-4423-a47e-6ab4c477145d-istio-data" pod "4deb89b6-4d18-4423-a47e-6ab4c477145d" (UID: "4deb89b6-4d18-4423-a47e-6ab4c477145d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:38.331796 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.331554 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6166d64-362f-4ada-b2fd-ecf68bc023a7-istiod-ca-cert" pod "a6166d64-362f-4ada-b2fd-ecf68bc023a7" (UID: "a6166d64-362f-4ada-b2fd-ecf68bc023a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:01:38.331796 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.331774 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4deb89b6-4d18-4423-a47e-6ab4c477145d-credential-socket" pod "4deb89b6-4d18-4423-a47e-6ab4c477145d" (UID: "4deb89b6-4d18-4423-a47e-6ab4c477145d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:38.333401 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.332643 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4deb89b6-4d18-4423-a47e-6ab4c477145d-workload-certs" pod "4deb89b6-4d18-4423-a47e-6ab4c477145d" (UID: "4deb89b6-4d18-4423-a47e-6ab4c477145d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:38.333401 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.333236 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6166d64-362f-4ada-b2fd-ecf68bc023a7-istio-podinfo" pod "a6166d64-362f-4ada-b2fd-ecf68bc023a7" (UID: "a6166d64-362f-4ada-b2fd-ecf68bc023a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:01:38.333949 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.333905 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4deb89b6-4d18-4423-a47e-6ab4c477145d-istio-envoy" pod "4deb89b6-4d18-4423-a47e-6ab4c477145d" (UID: "4deb89b6-4d18-4423-a47e-6ab4c477145d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:38.333949 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.333908 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4deb89b6-4d18-4423-a47e-6ab4c477145d-istio-token" pod "4deb89b6-4d18-4423-a47e-6ab4c477145d" (UID: "4deb89b6-4d18-4423-a47e-6ab4c477145d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:38.334695 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.334659 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6166d64-362f-4ada-b2fd-ecf68bc023a7-istio-token" pod "a6166d64-362f-4ada-b2fd-ecf68bc023a7" (UID: "a6166d64-362f-4ada-b2fd-ecf68bc023a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:38.335172 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.335071 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6166d64-362f-4ada-b2fd-ecf68bc023a7-kube-api-access-t29bc" pod "a6166d64-362f-4ada-b2fd-ecf68bc023a7" (UID: "a6166d64-362f-4ada-b2fd-ecf68bc023a7"). InnerVolumeSpecName "kube-api-access-t29bc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:38.335172 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.335149 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4deb89b6-4d18-4423-a47e-6ab4c477145d-kube-api-access-dwbjg" pod "4deb89b6-4d18-4423-a47e-6ab4c477145d" (UID: "4deb89b6-4d18-4423-a47e-6ab4c477145d"). InnerVolumeSpecName "kube-api-access-dwbjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:38.335488 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.335469 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6166d64-362f-4ada-b2fd-ecf68bc023a7-istio-envoy" pod "a6166d64-362f-4ada-b2fd-ecf68bc023a7" (UID: "a6166d64-362f-4ada-b2fd-ecf68bc023a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:38.336754 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.336722 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4deb89b6-4d18-4423-a47e-6ab4c477145d-istio-podinfo" pod "4deb89b6-4d18-4423-a47e-6ab4c477145d" (UID: "4deb89b6-4d18-4423-a47e-6ab4c477145d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:01:38.432534 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.432447 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4deb89b6-4d18-4423-a47e-6ab4c477145d-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.432534 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.432478 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4deb89b6-4d18-4423-a47e-6ab4c477145d-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.432534 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.432493 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6166d64-362f-4ada-b2fd-ecf68bc023a7-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.432534 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.432506 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4deb89b6-4d18-4423-a47e-6ab4c477145d-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.432534 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.432518 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4deb89b6-4d18-4423-a47e-6ab4c477145d-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.432534 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.432532 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6166d64-362f-4ada-b2fd-ecf68bc023a7-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.432888 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.432545 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6166d64-362f-4ada-b2fd-ecf68bc023a7-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.432888 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.432557 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4deb89b6-4d18-4423-a47e-6ab4c477145d-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.432888 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.432569 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t29bc\" (UniqueName: \"kubernetes.io/projected/a6166d64-362f-4ada-b2fd-ecf68bc023a7-kube-api-access-t29bc\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.432888 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.432583 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6166d64-362f-4ada-b2fd-ecf68bc023a7-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.432888 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.432595 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwbjg\" (UniqueName: \"kubernetes.io/projected/4deb89b6-4d18-4423-a47e-6ab4c477145d-kube-api-access-dwbjg\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.432888 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.432606 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4deb89b6-4d18-4423-a47e-6ab4c477145d-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.432888 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.432618 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4deb89b6-4d18-4423-a47e-6ab4c477145d-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.432888 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.432631 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4deb89b6-4d18-4423-a47e-6ab4c477145d-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:01:38.657692 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.657649 5196 patch_prober.go:29] interesting pod/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 03 23:01:38.658087 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.657721 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" podUID="0d375a3f-2f84-4495-a683-42a98477ff52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 03 23:01:38.734101 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.734009 5196 generic.go:356] "Generic (PLEG): container finished" podID="4deb89b6-4d18-4423-a47e-6ab4c477145d" containerID="660620dd08ad7d07fe7b886864e14355268aaaed7f0c36db17beb5fdc45280c3" exitCode=0 Nov 03 23:01:38.734440 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.734109 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" event={"ID":"4deb89b6-4d18-4423-a47e-6ab4c477145d","Type":"ContainerDied","Data":"660620dd08ad7d07fe7b886864e14355268aaaed7f0c36db17beb5fdc45280c3"} Nov 03 23:01:38.734513 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.734493 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" event={"ID":"4deb89b6-4d18-4423-a47e-6ab4c477145d","Type":"ContainerDied","Data":"892087a140e27cff7f952d1e36d565cd8e09d514e7d92ca7d8b77da8c0b4b8d7"} Nov 03 23:01:38.734571 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.734122 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9" Nov 03 23:01:38.734571 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.734536 5196 scope.go:122] "RemoveContainer" containerID="660620dd08ad7d07fe7b886864e14355268aaaed7f0c36db17beb5fdc45280c3" Nov 03 23:01:38.736026 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.736001 5196 generic.go:356] "Generic (PLEG): container finished" podID="625eb3b0-400d-4fa7-b5e7-80a08e17a42e" containerID="90d15a672be3ebd5c87943d5230353a982153933c55a33009bdc1af10c7b5517" exitCode=0 Nov 03 23:01:38.736176 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.736052 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" Nov 03 23:01:38.736176 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.736095 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" event={"ID":"625eb3b0-400d-4fa7-b5e7-80a08e17a42e","Type":"ContainerDied","Data":"90d15a672be3ebd5c87943d5230353a982153933c55a33009bdc1af10c7b5517"} Nov 03 23:01:38.736176 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.736148 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt" event={"ID":"625eb3b0-400d-4fa7-b5e7-80a08e17a42e","Type":"ContainerDied","Data":"0ace3c0fdee35a380ff36952d7c921cd7f16ade10af93dedb7d67af13970fe32"} Nov 03 23:01:38.738459 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.738435 5196 generic.go:356] "Generic (PLEG): container finished" podID="a6166d64-362f-4ada-b2fd-ecf68bc023a7" containerID="fd25979b4993272a6e065da57b2cd211a2f592f762ef7a886a246f93820166cc" exitCode=0 Nov 03 23:01:38.738627 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.738527 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" event={"ID":"a6166d64-362f-4ada-b2fd-ecf68bc023a7","Type":"ContainerDied","Data":"fd25979b4993272a6e065da57b2cd211a2f592f762ef7a886a246f93820166cc"} Nov 03 23:01:38.738627 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.738567 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" event={"ID":"a6166d64-362f-4ada-b2fd-ecf68bc023a7","Type":"ContainerDied","Data":"53ce57e00f416bbb1133ab97e71b2cadccdbdb17891a6e6ada81943834a18ee3"} Nov 03 23:01:38.738785 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.738635 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6" Nov 03 23:01:38.744577 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.744554 5196 generic.go:356] "Generic (PLEG): container finished" podID="9d0e7774-f0b0-427d-bbd4-7070d70b8fca" containerID="c52e75cd15a3b17b8ddcc2930732c1275843e2434da7cf7c47b4a8205e8ffc75" exitCode=0 Nov 03 23:01:38.744664 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.744654 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" event={"ID":"9d0e7774-f0b0-427d-bbd4-7070d70b8fca","Type":"ContainerDied","Data":"c52e75cd15a3b17b8ddcc2930732c1275843e2434da7cf7c47b4a8205e8ffc75"} Nov 03 23:01:38.744725 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.744696 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" event={"ID":"9d0e7774-f0b0-427d-bbd4-7070d70b8fca","Type":"ContainerDied","Data":"f150be4bc397750f4cb584e9996ad460db41d469f90bcd0d364909fbd9a173ad"} Nov 03 23:01:38.744778 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.744767 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t" Nov 03 23:01:38.748911 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.748890 5196 scope.go:122] "RemoveContainer" containerID="660620dd08ad7d07fe7b886864e14355268aaaed7f0c36db17beb5fdc45280c3" Nov 03 23:01:38.749448 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:01:38.749372 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"660620dd08ad7d07fe7b886864e14355268aaaed7f0c36db17beb5fdc45280c3\": container with ID starting with 660620dd08ad7d07fe7b886864e14355268aaaed7f0c36db17beb5fdc45280c3 not found: ID does not exist" containerID="660620dd08ad7d07fe7b886864e14355268aaaed7f0c36db17beb5fdc45280c3" Nov 03 23:01:38.749550 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.749476 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"660620dd08ad7d07fe7b886864e14355268aaaed7f0c36db17beb5fdc45280c3"} err="failed to get container status \"660620dd08ad7d07fe7b886864e14355268aaaed7f0c36db17beb5fdc45280c3\": rpc error: code = NotFound desc = could not find container \"660620dd08ad7d07fe7b886864e14355268aaaed7f0c36db17beb5fdc45280c3\": container with ID starting with 660620dd08ad7d07fe7b886864e14355268aaaed7f0c36db17beb5fdc45280c3 not found: ID does not exist" Nov 03 23:01:38.749550 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.749506 5196 scope.go:122] "RemoveContainer" containerID="90d15a672be3ebd5c87943d5230353a982153933c55a33009bdc1af10c7b5517" Nov 03 23:01:38.763377 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.763349 5196 scope.go:122] "RemoveContainer" containerID="90d15a672be3ebd5c87943d5230353a982153933c55a33009bdc1af10c7b5517" Nov 03 23:01:38.763692 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:01:38.763676 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90d15a672be3ebd5c87943d5230353a982153933c55a33009bdc1af10c7b5517\": container with ID starting with 90d15a672be3ebd5c87943d5230353a982153933c55a33009bdc1af10c7b5517 not found: ID does not exist" containerID="90d15a672be3ebd5c87943d5230353a982153933c55a33009bdc1af10c7b5517" Nov 03 23:01:38.763780 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.763699 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90d15a672be3ebd5c87943d5230353a982153933c55a33009bdc1af10c7b5517"} err="failed to get container status \"90d15a672be3ebd5c87943d5230353a982153933c55a33009bdc1af10c7b5517\": rpc error: code = NotFound desc = could not find container \"90d15a672be3ebd5c87943d5230353a982153933c55a33009bdc1af10c7b5517\": container with ID starting with 90d15a672be3ebd5c87943d5230353a982153933c55a33009bdc1af10c7b5517 not found: ID does not exist" Nov 03 23:01:38.763780 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.763715 5196 scope.go:122] "RemoveContainer" containerID="fd25979b4993272a6e065da57b2cd211a2f592f762ef7a886a246f93820166cc" Nov 03 23:01:38.774400 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.774380 5196 scope.go:122] "RemoveContainer" containerID="fd25979b4993272a6e065da57b2cd211a2f592f762ef7a886a246f93820166cc" Nov 03 23:01:38.774623 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:01:38.774603 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd25979b4993272a6e065da57b2cd211a2f592f762ef7a886a246f93820166cc\": container with ID starting with fd25979b4993272a6e065da57b2cd211a2f592f762ef7a886a246f93820166cc not found: ID does not exist" containerID="fd25979b4993272a6e065da57b2cd211a2f592f762ef7a886a246f93820166cc" Nov 03 23:01:38.774694 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.774633 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd25979b4993272a6e065da57b2cd211a2f592f762ef7a886a246f93820166cc"} err="failed to get container status \"fd25979b4993272a6e065da57b2cd211a2f592f762ef7a886a246f93820166cc\": rpc error: code = NotFound desc = could not find container \"fd25979b4993272a6e065da57b2cd211a2f592f762ef7a886a246f93820166cc\": container with ID starting with fd25979b4993272a6e065da57b2cd211a2f592f762ef7a886a246f93820166cc not found: ID does not exist" Nov 03 23:01:38.774694 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.774660 5196 scope.go:122] "RemoveContainer" containerID="c52e75cd15a3b17b8ddcc2930732c1275843e2434da7cf7c47b4a8205e8ffc75" Nov 03 23:01:38.783555 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.783540 5196 scope.go:122] "RemoveContainer" containerID="c52e75cd15a3b17b8ddcc2930732c1275843e2434da7cf7c47b4a8205e8ffc75" Nov 03 23:01:38.783807 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:01:38.783785 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c52e75cd15a3b17b8ddcc2930732c1275843e2434da7cf7c47b4a8205e8ffc75\": container with ID starting with c52e75cd15a3b17b8ddcc2930732c1275843e2434da7cf7c47b4a8205e8ffc75 not found: ID does not exist" containerID="c52e75cd15a3b17b8ddcc2930732c1275843e2434da7cf7c47b4a8205e8ffc75" Nov 03 23:01:38.783910 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.783810 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c52e75cd15a3b17b8ddcc2930732c1275843e2434da7cf7c47b4a8205e8ffc75"} err="failed to get container status \"c52e75cd15a3b17b8ddcc2930732c1275843e2434da7cf7c47b4a8205e8ffc75\": rpc error: code = NotFound desc = could not find container \"c52e75cd15a3b17b8ddcc2930732c1275843e2434da7cf7c47b4a8205e8ffc75\": container with ID starting with c52e75cd15a3b17b8ddcc2930732c1275843e2434da7cf7c47b4a8205e8ffc75 not found: ID does not exist" Nov 03 23:01:38.920065 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.920029 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt"] Nov 03 23:01:38.922777 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:38.922757 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-5wnpt"] Nov 03 23:01:39.030631 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:39.030541 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9"] Nov 03 23:01:39.034364 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:39.034339 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-27kc9"] Nov 03 23:01:39.154477 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:39.154429 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6"] Nov 03 23:01:39.157157 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:39.157129 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-lj7c6"] Nov 03 23:01:39.276214 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:39.276168 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t"] Nov 03 23:01:39.279809 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:39.279784 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a9ab23c4-openshift-default-766c495bb9-kc75t"] Nov 03 23:01:39.657954 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:39.657911 5196 patch_prober.go:29] interesting pod/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 03 23:01:39.658343 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:39.657967 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" podUID="0d375a3f-2f84-4495-a683-42a98477ff52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 03 23:01:40.056847 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:40.056767 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:40.662362 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:40.662336 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 23:01:40.663611 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:40.663590 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 23:01:40.815350 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:40.815317 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4deb89b6-4d18-4423-a47e-6ab4c477145d" path="/var/lib/kubelet/pods/4deb89b6-4d18-4423-a47e-6ab4c477145d/volumes" Nov 03 23:01:40.815764 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:40.815750 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="625eb3b0-400d-4fa7-b5e7-80a08e17a42e" path="/var/lib/kubelet/pods/625eb3b0-400d-4fa7-b5e7-80a08e17a42e/volumes" Nov 03 23:01:40.816214 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:40.816200 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d0e7774-f0b0-427d-bbd4-7070d70b8fca" path="/var/lib/kubelet/pods/9d0e7774-f0b0-427d-bbd4-7070d70b8fca/volumes" Nov 03 23:01:40.816937 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:40.816920 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6166d64-362f-4ada-b2fd-ecf68bc023a7" path="/var/lib/kubelet/pods/a6166d64-362f-4ada-b2fd-ecf68bc023a7/volumes" Nov 03 23:01:44.825230 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:44.825175 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:49.718660 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:49.718619 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:54.286509 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:54.286474 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5"] Nov 03 23:01:54.287038 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:54.287023 5196 cpu_manager.go:414] "containerMap: removing container" podUID="9d0e7774-f0b0-427d-bbd4-7070d70b8fca" containerName="istio-proxy" Nov 03 23:01:54.287096 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:54.287039 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d0e7774-f0b0-427d-bbd4-7070d70b8fca" containerName="istio-proxy" Nov 03 23:01:54.287096 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:54.287052 5196 cpu_manager.go:414] "containerMap: removing container" podUID="625eb3b0-400d-4fa7-b5e7-80a08e17a42e" containerName="istio-proxy" Nov 03 23:01:54.287096 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:54.287057 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="625eb3b0-400d-4fa7-b5e7-80a08e17a42e" containerName="istio-proxy" Nov 03 23:01:54.287096 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:54.287071 5196 cpu_manager.go:414] "containerMap: removing container" podUID="4deb89b6-4d18-4423-a47e-6ab4c477145d" containerName="istio-proxy" Nov 03 23:01:54.287096 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:54.287077 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4deb89b6-4d18-4423-a47e-6ab4c477145d" containerName="istio-proxy" Nov 03 23:01:54.287096 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:54.287095 5196 cpu_manager.go:414] "containerMap: removing container" podUID="a6166d64-362f-4ada-b2fd-ecf68bc023a7" containerName="istio-proxy" Nov 03 23:01:54.287283 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:54.287100 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6166d64-362f-4ada-b2fd-ecf68bc023a7" containerName="istio-proxy" Nov 03 23:01:54.287283 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:54.287169 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="625eb3b0-400d-4fa7-b5e7-80a08e17a42e" containerName="istio-proxy" Nov 03 23:01:54.287283 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:54.287178 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6166d64-362f-4ada-b2fd-ecf68bc023a7" containerName="istio-proxy" Nov 03 23:01:54.287283 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:54.287190 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="4deb89b6-4d18-4423-a47e-6ab4c477145d" containerName="istio-proxy" Nov 03 23:01:54.287283 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:54.287199 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d0e7774-f0b0-427d-bbd4-7070d70b8fca" containerName="istio-proxy" Nov 03 23:01:54.290590 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:54.290568 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5" Nov 03 23:01:54.347156 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:54.347127 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-dcf5a5-serving-cert\"" Nov 03 23:01:54.351917 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:54.351888 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5"] Nov 03 23:01:54.448740 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:54.448692 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7914cbce-00fd-4d6b-84c9-4519c50af650-metrics-certs\") pod \"maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5\" (UID: \"7914cbce-00fd-4d6b-84c9-4519c50af650\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5" Nov 03 23:01:54.449052 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:54.449027 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srtb5\" (UniqueName: \"kubernetes.io/projected/7914cbce-00fd-4d6b-84c9-4519c50af650-kube-api-access-srtb5\") pod \"maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5\" (UID: \"7914cbce-00fd-4d6b-84c9-4519c50af650\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5" Nov 03 23:01:54.449259 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:54.449239 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7914cbce-00fd-4d6b-84c9-4519c50af650-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5\" (UID: \"7914cbce-00fd-4d6b-84c9-4519c50af650\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5" Nov 03 23:01:54.449511 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:54.449487 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7914cbce-00fd-4d6b-84c9-4519c50af650-maas-api-tls\") pod \"maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5\" (UID: \"7914cbce-00fd-4d6b-84c9-4519c50af650\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5" Nov 03 23:01:54.494431 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:54.494395 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:54.556983 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:54.554726 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7914cbce-00fd-4d6b-84c9-4519c50af650-metrics-certs\") pod \"maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5\" (UID: \"7914cbce-00fd-4d6b-84c9-4519c50af650\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5" Nov 03 23:01:54.556983 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:54.554810 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srtb5\" (UniqueName: \"kubernetes.io/projected/7914cbce-00fd-4d6b-84c9-4519c50af650-kube-api-access-srtb5\") pod \"maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5\" (UID: \"7914cbce-00fd-4d6b-84c9-4519c50af650\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5" Nov 03 23:01:54.556983 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:54.554954 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7914cbce-00fd-4d6b-84c9-4519c50af650-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5\" (UID: \"7914cbce-00fd-4d6b-84c9-4519c50af650\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5" Nov 03 23:01:54.556983 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:54.555168 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7914cbce-00fd-4d6b-84c9-4519c50af650-maas-api-tls\") pod \"maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5\" (UID: \"7914cbce-00fd-4d6b-84c9-4519c50af650\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5" Nov 03 23:01:54.556983 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:01:54.555540 5196 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-dcf5a5-serving-cert: secret "maas-api-e2e-inf-b-dcf5a5-serving-cert" not found Nov 03 23:01:54.556983 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:01:54.555682 5196 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7914cbce-00fd-4d6b-84c9-4519c50af650-maas-api-tls podName:7914cbce-00fd-4d6b-84c9-4519c50af650 nodeName:}" failed. No retries permitted until 2026-11-03 23:01:55.055628475 +0000 UTC m=+2344.365187814 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7914cbce-00fd-4d6b-84c9-4519c50af650-maas-api-tls") pod "maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5" (UID: "7914cbce-00fd-4d6b-84c9-4519c50af650") : secret "maas-api-e2e-inf-b-dcf5a5-serving-cert" not found Nov 03 23:01:54.574876 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:54.559320 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7914cbce-00fd-4d6b-84c9-4519c50af650-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5\" (UID: \"7914cbce-00fd-4d6b-84c9-4519c50af650\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5" Nov 03 23:01:54.589220 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:54.578463 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7914cbce-00fd-4d6b-84c9-4519c50af650-metrics-certs\") pod \"maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5\" (UID: \"7914cbce-00fd-4d6b-84c9-4519c50af650\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5" Nov 03 23:01:54.638849 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:54.638772 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srtb5\" (UniqueName: \"kubernetes.io/projected/7914cbce-00fd-4d6b-84c9-4519c50af650-kube-api-access-srtb5\") pod \"maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5\" (UID: \"7914cbce-00fd-4d6b-84c9-4519c50af650\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5" Nov 03 23:01:54.663371 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:54.663211 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:54.824994 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:54.824915 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:55.065874 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:55.065815 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7914cbce-00fd-4d6b-84c9-4519c50af650-maas-api-tls\") pod \"maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5\" (UID: \"7914cbce-00fd-4d6b-84c9-4519c50af650\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5" Nov 03 23:01:55.068832 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:55.068798 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7914cbce-00fd-4d6b-84c9-4519c50af650-maas-api-tls\") pod \"maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5\" (UID: \"7914cbce-00fd-4d6b-84c9-4519c50af650\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5" Nov 03 23:01:55.207009 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:55.206979 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5" Nov 03 23:01:55.601670 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:55.601636 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5"] Nov 03 23:01:55.602136 ip-10-0-143-132 kubenswrapper[5196]: W1103 23:01:55.601990 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7914cbce_00fd_4d6b_84c9_4519c50af650.slice/crio-93b255f4c3e987a320b57272652e65cdbdc59d947f4f6470677624374f5e00b2 WatchSource:0}: Error finding container 93b255f4c3e987a320b57272652e65cdbdc59d947f4f6470677624374f5e00b2: Status 404 returned error can't find the container with id 93b255f4c3e987a320b57272652e65cdbdc59d947f4f6470677624374f5e00b2 Nov 03 23:01:55.883575 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:55.883478 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5" event={"ID":"7914cbce-00fd-4d6b-84c9-4519c50af650","Type":"ContainerStarted","Data":"93b255f4c3e987a320b57272652e65cdbdc59d947f4f6470677624374f5e00b2"} Nov 03 23:01:55.922350 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:01:55.922308 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:01:56.893754 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:56.893711 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5" event={"ID":"7914cbce-00fd-4d6b-84c9-4519c50af650","Type":"ContainerStarted","Data":"7d0bdc6396e6d713d6b27723c120a88fff35255a4d85edb98eadf14947921e6e"} Nov 03 23:01:56.894271 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:56.893798 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5" Nov 03 23:01:57.013339 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:57.013282 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5" podStartSLOduration=2.18035755 podStartE2EDuration="3.013268723s" totalImagesPullingTime="832.911173ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:01:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:01:57.012278535 +0000 UTC m=+2346.321837827" watchObservedRunningTime="2026-11-03 23:01:57.013268723 +0000 UTC m=+2346.322828016" Nov 03 23:01:57.783035 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:57.783000 5196 patch_prober.go:29] interesting pod/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:01:57.783221 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:57.783056 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" podUID="59200876-e179-4170-832d-d85cba111281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:01:58.907779 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:58.907746 5196 generic.go:356] "Generic (PLEG): container finished" podID="59200876-e179-4170-832d-d85cba111281" containerID="159fb8a18f6a38943292cbe931ada2d53535627cd6be0c70cda2d95bb733c204" exitCode=0 Nov 03 23:01:58.908278 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:58.907818 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" event={"ID":"59200876-e179-4170-832d-d85cba111281","Type":"ContainerDied","Data":"159fb8a18f6a38943292cbe931ada2d53535627cd6be0c70cda2d95bb733c204"} Nov 03 23:01:58.908278 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:58.908224 5196 scope.go:122] "RemoveContainer" containerID="159fb8a18f6a38943292cbe931ada2d53535627cd6be0c70cda2d95bb733c204" Nov 03 23:01:58.909784 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:58.909753 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:01:58.909843 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:58.909826 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:01:58.909913 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:58.909878 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:01:59.453342 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:59.453311 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:01:59.916219 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:59.916184 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" event={"ID":"59200876-e179-4170-832d-d85cba111281","Type":"ContainerStarted","Data":"f9b6ad04597760bd9788d8c14229e49341b2467d81c993c53bd42fb1ddf7bb7d"} Nov 03 23:01:59.970534 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:59.970501 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 23:01:59.970707 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:59.970672 5196 patch_prober.go:29] interesting pod/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 23:01:59.970753 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:01:59.970720 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" podUID="59200876-e179-4170-832d-d85cba111281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 23:02:00.971273 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:00.971233 5196 patch_prober.go:29] interesting pod/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 23:02:00.971837 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:00.971290 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" podUID="59200876-e179-4170-832d-d85cba111281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 23:02:01.971314 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:01.971282 5196 patch_prober.go:29] interesting pod/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 23:02:01.971704 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:01.971338 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" podUID="59200876-e179-4170-832d-d85cba111281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 23:02:02.920563 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:02.920532 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5" Nov 03 23:02:02.971254 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:02.971223 5196 patch_prober.go:29] interesting pod/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 23:02:02.971407 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:02.971278 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" podUID="59200876-e179-4170-832d-d85cba111281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 23:02:03.971243 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:03.971206 5196 patch_prober.go:29] interesting pod/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 23:02:03.971440 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:03.971268 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" podUID="59200876-e179-4170-832d-d85cba111281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 23:02:04.241783 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:04.241707 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:04.971384 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:04.971351 5196 patch_prober.go:29] interesting pod/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 23:02:04.971561 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:04.971403 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" podUID="59200876-e179-4170-832d-d85cba111281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 23:02:06.033621 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:06.033561 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 23:02:06.034226 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:06.034054 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 23:02:06.035281 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:06.035259 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 23:02:09.335360 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:09.335321 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:09.895016 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:09.894985 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:14.355311 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:14.355284 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:15.278798 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:15.278763 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:19.411570 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:19.411528 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:21.322354 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:21.322321 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:22.006673 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:22.006634 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:22.690272 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:22.690207 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:23.086669 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:23.086574 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:23.118132 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:23.118099 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ea5c8f-7db7d5b55f-qnbc8"] Nov 03 23:02:23.364354 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:23.364325 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:23.519492 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:23.519462 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:24.556568 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:24.556509 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:25.919268 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:02:25.919234 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:02:27.291970 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:27.291922 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:27.465256 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:27.465226 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:27.629753 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:27.629721 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:28.118758 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:28.118697 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ea5c8f-7db7d5b55f-qnbc8" podUID="286e0af7-6c70-47b8-924c-5f584a4ba2f8" containerName="payload-pre-processing" containerID="cri-o://bca8602399801d660bfd7ee4d4b6359b9db494795fa74b77a76b65a4350cacb5" gracePeriod=25 Nov 03 23:02:28.405819 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:28.405791 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ea5c8f-7db7d5b55f-qnbc8" Nov 03 23:02:28.495599 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:28.495562 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/286e0af7-6c70-47b8-924c-5f584a4ba2f8-kube-api-access-t8gk9\" (UniqueName: \"kubernetes.io/projected/286e0af7-6c70-47b8-924c-5f584a4ba2f8-kube-api-access-t8gk9\") pod \"286e0af7-6c70-47b8-924c-5f584a4ba2f8\" (UID: \"286e0af7-6c70-47b8-924c-5f584a4ba2f8\") " Nov 03 23:02:28.495782 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:28.495761 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/286e0af7-6c70-47b8-924c-5f584a4ba2f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/286e0af7-6c70-47b8-924c-5f584a4ba2f8-plugins-config-volume\") pod \"286e0af7-6c70-47b8-924c-5f584a4ba2f8\" (UID: \"286e0af7-6c70-47b8-924c-5f584a4ba2f8\") " Nov 03 23:02:28.496175 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:28.496150 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/286e0af7-6c70-47b8-924c-5f584a4ba2f8-plugins-config-volume" pod "286e0af7-6c70-47b8-924c-5f584a4ba2f8" (UID: "286e0af7-6c70-47b8-924c-5f584a4ba2f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:02:28.496414 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:28.496378 5196 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/286e0af7-6c70-47b8-924c-5f584a4ba2f8-plugins-config-volume\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:02:28.498441 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:28.498418 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/286e0af7-6c70-47b8-924c-5f584a4ba2f8-kube-api-access-t8gk9" pod "286e0af7-6c70-47b8-924c-5f584a4ba2f8" (UID: "286e0af7-6c70-47b8-924c-5f584a4ba2f8"). InnerVolumeSpecName "kube-api-access-t8gk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:02:28.597481 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:28.597447 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8gk9\" (UniqueName: \"kubernetes.io/projected/286e0af7-6c70-47b8-924c-5f584a4ba2f8-kube-api-access-t8gk9\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:02:29.098465 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:29.098431 5196 generic.go:356] "Generic (PLEG): container finished" podID="286e0af7-6c70-47b8-924c-5f584a4ba2f8" containerID="bca8602399801d660bfd7ee4d4b6359b9db494795fa74b77a76b65a4350cacb5" exitCode=0 Nov 03 23:02:29.098649 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:29.098522 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ea5c8f-7db7d5b55f-qnbc8" event={"ID":"286e0af7-6c70-47b8-924c-5f584a4ba2f8","Type":"ContainerDied","Data":"bca8602399801d660bfd7ee4d4b6359b9db494795fa74b77a76b65a4350cacb5"} Nov 03 23:02:29.098649 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:29.098545 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ea5c8f-7db7d5b55f-qnbc8" Nov 03 23:02:29.098649 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:29.098576 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ea5c8f-7db7d5b55f-qnbc8" event={"ID":"286e0af7-6c70-47b8-924c-5f584a4ba2f8","Type":"ContainerDied","Data":"27d0fc94ca9cac4d78c843b5360cf3de87590082b9c3f9157a61ab8688722c8a"} Nov 03 23:02:29.098649 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:29.098605 5196 scope.go:122] "RemoveContainer" containerID="bca8602399801d660bfd7ee4d4b6359b9db494795fa74b77a76b65a4350cacb5" Nov 03 23:02:29.109232 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:29.109211 5196 scope.go:122] "RemoveContainer" containerID="bca8602399801d660bfd7ee4d4b6359b9db494795fa74b77a76b65a4350cacb5" Nov 03 23:02:29.109505 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:02:29.109482 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bca8602399801d660bfd7ee4d4b6359b9db494795fa74b77a76b65a4350cacb5\": container with ID starting with bca8602399801d660bfd7ee4d4b6359b9db494795fa74b77a76b65a4350cacb5 not found: ID does not exist" containerID="bca8602399801d660bfd7ee4d4b6359b9db494795fa74b77a76b65a4350cacb5" Nov 03 23:02:29.109586 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:29.109512 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bca8602399801d660bfd7ee4d4b6359b9db494795fa74b77a76b65a4350cacb5"} err="failed to get container status \"bca8602399801d660bfd7ee4d4b6359b9db494795fa74b77a76b65a4350cacb5\": rpc error: code = NotFound desc = could not find container \"bca8602399801d660bfd7ee4d4b6359b9db494795fa74b77a76b65a4350cacb5\": container with ID starting with bca8602399801d660bfd7ee4d4b6359b9db494795fa74b77a76b65a4350cacb5 not found: ID does not exist" Nov 03 23:02:29.265983 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:29.265949 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ea5c8f-7db7d5b55f-qnbc8"] Nov 03 23:02:29.316927 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:29.316898 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ea5c8f-7db7d5b55f-qnbc8"] Nov 03 23:02:29.743681 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:29.743627 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:30.532778 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:30.532750 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:30.714543 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:30.714512 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:30.849456 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:30.840072 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="286e0af7-6c70-47b8-924c-5f584a4ba2f8" path="/var/lib/kubelet/pods/286e0af7-6c70-47b8-924c-5f584a4ba2f8/volumes" Nov 03 23:02:34.919501 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:34.919461 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:39.304914 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:39.304871 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2"] Nov 03 23:02:39.305771 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:39.305719 5196 cpu_manager.go:414] "containerMap: removing container" podUID="286e0af7-6c70-47b8-924c-5f584a4ba2f8" containerName="payload-pre-processing" Nov 03 23:02:39.305915 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:39.305771 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="286e0af7-6c70-47b8-924c-5f584a4ba2f8" containerName="payload-pre-processing" Nov 03 23:02:39.306028 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:39.306014 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="286e0af7-6c70-47b8-924c-5f584a4ba2f8" containerName="payload-pre-processing" Nov 03 23:02:39.310959 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:39.310942 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2" Nov 03 23:02:39.338921 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:39.338888 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjplx\" (UniqueName: \"kubernetes.io/projected/5f4dad96-aaa9-467a-b599-6b45196c5eaa-kube-api-access-cjplx\") pod \"maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2\" (UID: \"5f4dad96-aaa9-467a-b599-6b45196c5eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2" Nov 03 23:02:39.339076 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:39.339055 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f4dad96-aaa9-467a-b599-6b45196c5eaa-maas-api-tls\") pod \"maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2\" (UID: \"5f4dad96-aaa9-467a-b599-6b45196c5eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2" Nov 03 23:02:39.339154 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:39.339140 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f4dad96-aaa9-467a-b599-6b45196c5eaa-openshift-service-ca\") pod \"maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2\" (UID: \"5f4dad96-aaa9-467a-b599-6b45196c5eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2" Nov 03 23:02:39.339312 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:39.339292 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f4dad96-aaa9-467a-b599-6b45196c5eaa-metrics-certs\") pod \"maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2\" (UID: \"5f4dad96-aaa9-467a-b599-6b45196c5eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2" Nov 03 23:02:39.364569 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:39.364543 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-78be2c-serving-cert\"" Nov 03 23:02:39.418370 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:39.418338 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2"] Nov 03 23:02:39.441004 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:39.440976 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f4dad96-aaa9-467a-b599-6b45196c5eaa-openshift-service-ca\") pod \"maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2\" (UID: \"5f4dad96-aaa9-467a-b599-6b45196c5eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2" Nov 03 23:02:39.441145 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:39.441054 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f4dad96-aaa9-467a-b599-6b45196c5eaa-metrics-certs\") pod \"maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2\" (UID: \"5f4dad96-aaa9-467a-b599-6b45196c5eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2" Nov 03 23:02:39.441145 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:39.441128 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjplx\" (UniqueName: \"kubernetes.io/projected/5f4dad96-aaa9-467a-b599-6b45196c5eaa-kube-api-access-cjplx\") pod \"maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2\" (UID: \"5f4dad96-aaa9-467a-b599-6b45196c5eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2" Nov 03 23:02:39.441262 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:39.441246 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f4dad96-aaa9-467a-b599-6b45196c5eaa-maas-api-tls\") pod \"maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2\" (UID: \"5f4dad96-aaa9-467a-b599-6b45196c5eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2" Nov 03 23:02:39.441420 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:02:39.441397 5196 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-78be2c-serving-cert: secret "maas-api-e2e-inf-b-78be2c-serving-cert" not found Nov 03 23:02:39.441509 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:02:39.441494 5196 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f4dad96-aaa9-467a-b599-6b45196c5eaa-maas-api-tls podName:5f4dad96-aaa9-467a-b599-6b45196c5eaa nodeName:}" failed. No retries permitted until 2026-11-03 23:02:39.941471047 +0000 UTC m=+2389.251030334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5f4dad96-aaa9-467a-b599-6b45196c5eaa-maas-api-tls") pod "maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2" (UID: "5f4dad96-aaa9-467a-b599-6b45196c5eaa") : secret "maas-api-e2e-inf-b-78be2c-serving-cert" not found Nov 03 23:02:39.441708 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:39.441691 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f4dad96-aaa9-467a-b599-6b45196c5eaa-openshift-service-ca\") pod \"maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2\" (UID: \"5f4dad96-aaa9-467a-b599-6b45196c5eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2" Nov 03 23:02:39.443706 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:39.443689 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f4dad96-aaa9-467a-b599-6b45196c5eaa-metrics-certs\") pod \"maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2\" (UID: \"5f4dad96-aaa9-467a-b599-6b45196c5eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2" Nov 03 23:02:39.498072 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:39.498026 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:39.504480 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:39.504451 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjplx\" (UniqueName: \"kubernetes.io/projected/5f4dad96-aaa9-467a-b599-6b45196c5eaa-kube-api-access-cjplx\") pod \"maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2\" (UID: \"5f4dad96-aaa9-467a-b599-6b45196c5eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2" Nov 03 23:02:39.659726 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:39.659691 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:39.812342 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:39.812306 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:39.956826 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:39.956743 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f4dad96-aaa9-467a-b599-6b45196c5eaa-maas-api-tls\") pod \"maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2\" (UID: \"5f4dad96-aaa9-467a-b599-6b45196c5eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2" Nov 03 23:02:39.960075 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:39.960043 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f4dad96-aaa9-467a-b599-6b45196c5eaa-maas-api-tls\") pod \"maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2\" (UID: \"5f4dad96-aaa9-467a-b599-6b45196c5eaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2" Nov 03 23:02:40.224535 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:40.224439 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2" Nov 03 23:02:40.677756 ip-10-0-143-132 kubenswrapper[5196]: W1103 23:02:40.677709 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f4dad96_aaa9_467a_b599_6b45196c5eaa.slice/crio-294ba27ecd0bbed9aec336f6ac158835d02f5e7112328d7df9f2e644ea051076 WatchSource:0}: Error finding container 294ba27ecd0bbed9aec336f6ac158835d02f5e7112328d7df9f2e644ea051076: Status 404 returned error can't find the container with id 294ba27ecd0bbed9aec336f6ac158835d02f5e7112328d7df9f2e644ea051076 Nov 03 23:02:40.678633 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:40.678603 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2"] Nov 03 23:02:41.233201 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:41.233147 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2" event={"ID":"5f4dad96-aaa9-467a-b599-6b45196c5eaa","Type":"ContainerStarted","Data":"294ba27ecd0bbed9aec336f6ac158835d02f5e7112328d7df9f2e644ea051076"} Nov 03 23:02:42.480789 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:42.480741 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2" event={"ID":"5f4dad96-aaa9-467a-b599-6b45196c5eaa","Type":"ContainerStarted","Data":"e8ee163f30526671fc382c4ab7368088664d7caed8e0b46c4f7bfdbc6901c5db"} Nov 03 23:02:42.501884 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:42.483776 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2" Nov 03 23:02:42.618434 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:42.618370 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2" podStartSLOduration=2.713828368 podStartE2EDuration="3.618347899s" totalImagesPullingTime="904.519531ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:02:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:02:42.609312896 +0000 UTC m=+2391.918872190" watchObservedRunningTime="2026-11-03 23:02:42.618347899 +0000 UTC m=+2391.927907193" Nov 03 23:02:44.842955 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:44.842922 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:49.516901 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:49.516849 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2" Nov 03 23:02:49.615866 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:49.615830 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:50.553791 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:50.553758 5196 generic.go:356] "Generic (PLEG): container finished" podID="4b847dde-7257-44e0-b25c-9ebe4cdad697" containerID="031184b6b8125e3f772103763af015ba5f2dff52af37f785d5fbdac5b41e5c00" exitCode=0 Nov 03 23:02:50.554255 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:50.553829 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" event={"ID":"4b847dde-7257-44e0-b25c-9ebe4cdad697","Type":"ContainerDied","Data":"031184b6b8125e3f772103763af015ba5f2dff52af37f785d5fbdac5b41e5c00"} Nov 03 23:02:50.554255 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:50.554234 5196 scope.go:122] "RemoveContainer" containerID="031184b6b8125e3f772103763af015ba5f2dff52af37f785d5fbdac5b41e5c00" Nov 03 23:02:50.555868 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:50.555825 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:02:50.555933 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:50.555921 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:02:50.555974 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:50.555960 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:02:51.562227 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:51.562190 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" event={"ID":"4b847dde-7257-44e0-b25c-9ebe4cdad697","Type":"ContainerStarted","Data":"5a8d4cb035b182d99b74f47f82b198b0a4c1df99743dff6adc724a937ab08ce8"} Nov 03 23:02:52.078596 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:52.078562 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:02:52.079637 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:52.079597 5196 patch_prober.go:29] interesting pod/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 23:02:52.079785 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:52.079648 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" podUID="4b847dde-7257-44e0-b25c-9ebe4cdad697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 23:02:53.078053 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:53.078014 5196 patch_prober.go:29] interesting pod/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 23:02:53.078537 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:53.078074 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" podUID="4b847dde-7257-44e0-b25c-9ebe4cdad697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 23:02:53.999983 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:53.999949 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:02:54.078596 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:54.078563 5196 patch_prober.go:29] interesting pod/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 23:02:54.079004 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:54.078621 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" podUID="4b847dde-7257-44e0-b25c-9ebe4cdad697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 23:02:54.439226 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:54.439182 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:55.078526 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:55.078498 5196 patch_prober.go:29] interesting pod/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 23:02:55.078709 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:55.078551 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" podUID="4b847dde-7257-44e0-b25c-9ebe4cdad697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 23:02:55.940265 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:02:55.940226 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:02:56.078464 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:56.078424 5196 patch_prober.go:29] interesting pod/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 23:02:56.078632 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:56.078482 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" podUID="4b847dde-7257-44e0-b25c-9ebe4cdad697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 23:02:57.083646 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:57.083619 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:02:57.084924 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:57.084902 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:02:57.446639 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:57.446608 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442"] Nov 03 23:02:57.446866 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:57.446819 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" podUID="50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7" containerName="istio-proxy" containerID="cri-o://893a9a0e8febbc315aec0d20d348e68cee3c49de2eb97cc2a88ea1237da6bf29" gracePeriod=30 Nov 03 23:02:57.573262 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:57.573230 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:57.718638 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:57.718570 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:02:59.669316 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:02:59.669273 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:02.656354 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:02.656272 5196 generic.go:356] "Generic (PLEG): container finished" podID="50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7" containerID="893a9a0e8febbc315aec0d20d348e68cee3c49de2eb97cc2a88ea1237da6bf29" exitCode=0 Nov 03 23:03:02.656810 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:02.656437 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" event={"ID":"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7","Type":"ContainerDied","Data":"893a9a0e8febbc315aec0d20d348e68cee3c49de2eb97cc2a88ea1237da6bf29"} Nov 03 23:03:02.674893 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:02.664418 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:02.726079 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:02.726023 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" Nov 03 23:03:02.845428 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:02.845396 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-istio-data\") pod \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\" (UID: \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\") " Nov 03 23:03:02.845608 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:02.845495 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-istio-podinfo\") pod \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\" (UID: \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\") " Nov 03 23:03:02.845608 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:02.845535 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-istio-envoy\") pod \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\" (UID: \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\") " Nov 03 23:03:02.845608 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:02.845563 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-istiod-ca-cert\") pod \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\" (UID: \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\") " Nov 03 23:03:02.845608 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:02.845603 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-istio-token\" (UniqueName: \"kubernetes.io/projected/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-istio-token\") pod \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\" (UID: \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\") " Nov 03 23:03:02.845886 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:02.845638 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-workload-socket\") pod \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\" (UID: \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\") " Nov 03 23:03:02.845886 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:02.845671 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-istio-data" pod "50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7" (UID: "50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:02.845886 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:02.845685 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-workload-certs\") pod \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\" (UID: \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\") " Nov 03 23:03:02.845886 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:02.845742 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-kube-api-access-xrb52\" (UniqueName: \"kubernetes.io/projected/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-kube-api-access-xrb52\") pod \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\" (UID: \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\") " Nov 03 23:03:02.845886 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:02.845814 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-credential-socket\") pod \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\" (UID: \"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7\") " Nov 03 23:03:02.846207 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:02.845990 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-workload-socket" pod "50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7" (UID: "50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:02.846207 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:02.846051 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-istiod-ca-cert" pod "50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7" (UID: "50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:02.846342 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:02.846245 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-credential-socket" pod "50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7" (UID: "50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:02.846517 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:02.846474 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-workload-certs" pod "50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7" (UID: "50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:02.846790 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:02.846682 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:02.846790 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:02.846718 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:02.846790 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:02.846737 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:02.846790 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:02.846752 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:02.846790 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:02.846766 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:02.848538 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:02.848517 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-istio-envoy" pod "50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7" (UID: "50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:02.848829 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:02.848809 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-istio-podinfo" pod "50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7" (UID: "50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:03:02.848940 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:02.848850 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-istio-token" pod "50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7" (UID: "50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:02.848940 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:02.848892 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-kube-api-access-xrb52" pod "50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7" (UID: "50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7"). InnerVolumeSpecName "kube-api-access-xrb52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:02.948829 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:02.948795 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:02.948829 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:02.948827 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:02.948829 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:02.948837 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:02.949080 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:02.948851 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrb52\" (UniqueName: \"kubernetes.io/projected/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7-kube-api-access-xrb52\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:03.664020 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:03.663979 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" event={"ID":"50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7","Type":"ContainerDied","Data":"963b3633e70d0af4ce28479d18fcff917575c42c134c05c67ae42f52b5e6e85f"} Nov 03 23:03:03.664020 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:03.664009 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442" Nov 03 23:03:03.664563 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:03.664084 5196 scope.go:122] "RemoveContainer" containerID="893a9a0e8febbc315aec0d20d348e68cee3c49de2eb97cc2a88ea1237da6bf29" Nov 03 23:03:03.882357 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:03.882326 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442"] Nov 03 23:03:03.933606 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:03.933536 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-32449d-openshift-default-756d4bb7c7-sm442"] Nov 03 23:03:04.815765 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:04.815733 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7" path="/var/lib/kubelet/pods/50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7/volumes" Nov 03 23:03:04.910705 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:04.910672 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:07.622690 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:07.622657 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp"] Nov 03 23:03:07.868556 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:07.868528 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:08.843931 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:08.843899 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:09.016838 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:09.016803 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:10.187720 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:10.187689 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:10.336793 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:10.336761 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:14.932826 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:14.932789 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:15.165651 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:15.165621 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:15.632371 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:15.632335 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:20.043114 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:20.043076 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:22.643733 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:22.643690 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" podUID="abb461c4-9707-4b5c-aba0-793f7bb993e5" containerName="main" containerID="cri-o://beee901e27bf51231fbdbc562b681a34e9337823bd6b83d32898ad74583915ce" gracePeriod=45 Nov 03 23:03:22.791594 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:22.791565 5196 generic.go:356] "Generic (PLEG): container finished" podID="abb461c4-9707-4b5c-aba0-793f7bb993e5" containerID="beee901e27bf51231fbdbc562b681a34e9337823bd6b83d32898ad74583915ce" exitCode=0 Nov 03 23:03:22.791765 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:22.791644 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" event={"ID":"abb461c4-9707-4b5c-aba0-793f7bb993e5","Type":"ContainerDied","Data":"beee901e27bf51231fbdbc562b681a34e9337823bd6b83d32898ad74583915ce"} Nov 03 23:03:22.899050 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:22.898979 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" Nov 03 23:03:22.983997 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:22.983970 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abb461c4-9707-4b5c-aba0-793f7bb993e5-tls-certs\" (UniqueName: \"kubernetes.io/secret/abb461c4-9707-4b5c-aba0-793f7bb993e5-tls-certs\") pod \"abb461c4-9707-4b5c-aba0-793f7bb993e5\" (UID: \"abb461c4-9707-4b5c-aba0-793f7bb993e5\") " Nov 03 23:03:22.984189 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:22.984007 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abb461c4-9707-4b5c-aba0-793f7bb993e5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/abb461c4-9707-4b5c-aba0-793f7bb993e5-dshm\") pod \"abb461c4-9707-4b5c-aba0-793f7bb993e5\" (UID: \"abb461c4-9707-4b5c-aba0-793f7bb993e5\") " Nov 03 23:03:22.984189 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:22.984031 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abb461c4-9707-4b5c-aba0-793f7bb993e5-home\" (UniqueName: \"kubernetes.io/empty-dir/abb461c4-9707-4b5c-aba0-793f7bb993e5-home\") pod \"abb461c4-9707-4b5c-aba0-793f7bb993e5\" (UID: \"abb461c4-9707-4b5c-aba0-793f7bb993e5\") " Nov 03 23:03:22.984189 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:22.984064 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abb461c4-9707-4b5c-aba0-793f7bb993e5-kube-api-access-jbrg4\" (UniqueName: \"kubernetes.io/projected/abb461c4-9707-4b5c-aba0-793f7bb993e5-kube-api-access-jbrg4\") pod \"abb461c4-9707-4b5c-aba0-793f7bb993e5\" (UID: \"abb461c4-9707-4b5c-aba0-793f7bb993e5\") " Nov 03 23:03:22.984189 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:22.984103 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abb461c4-9707-4b5c-aba0-793f7bb993e5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abb461c4-9707-4b5c-aba0-793f7bb993e5-tmp-dir\") pod \"abb461c4-9707-4b5c-aba0-793f7bb993e5\" (UID: \"abb461c4-9707-4b5c-aba0-793f7bb993e5\") " Nov 03 23:03:22.984433 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:22.984199 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abb461c4-9707-4b5c-aba0-793f7bb993e5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abb461c4-9707-4b5c-aba0-793f7bb993e5-model-cache\") pod \"abb461c4-9707-4b5c-aba0-793f7bb993e5\" (UID: \"abb461c4-9707-4b5c-aba0-793f7bb993e5\") " Nov 03 23:03:22.984433 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:22.984333 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abb461c4-9707-4b5c-aba0-793f7bb993e5-home" pod "abb461c4-9707-4b5c-aba0-793f7bb993e5" (UID: "abb461c4-9707-4b5c-aba0-793f7bb993e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:22.984561 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:22.984418 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abb461c4-9707-4b5c-aba0-793f7bb993e5-tmp-dir" pod "abb461c4-9707-4b5c-aba0-793f7bb993e5" (UID: "abb461c4-9707-4b5c-aba0-793f7bb993e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:22.984561 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:22.984518 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abb461c4-9707-4b5c-aba0-793f7bb993e5-model-cache" pod "abb461c4-9707-4b5c-aba0-793f7bb993e5" (UID: "abb461c4-9707-4b5c-aba0-793f7bb993e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:22.984680 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:22.984645 5196 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abb461c4-9707-4b5c-aba0-793f7bb993e5-home\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:22.984680 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:22.984656 5196 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abb461c4-9707-4b5c-aba0-793f7bb993e5-tmp-dir\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:22.984680 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:22.984667 5196 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abb461c4-9707-4b5c-aba0-793f7bb993e5-model-cache\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:22.986436 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:22.986413 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abb461c4-9707-4b5c-aba0-793f7bb993e5-kube-api-access-jbrg4" pod "abb461c4-9707-4b5c-aba0-793f7bb993e5" (UID: "abb461c4-9707-4b5c-aba0-793f7bb993e5"). InnerVolumeSpecName "kube-api-access-jbrg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:22.986436 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:22.986425 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abb461c4-9707-4b5c-aba0-793f7bb993e5-dshm" pod "abb461c4-9707-4b5c-aba0-793f7bb993e5" (UID: "abb461c4-9707-4b5c-aba0-793f7bb993e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:22.986579 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:22.986458 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abb461c4-9707-4b5c-aba0-793f7bb993e5-tls-certs" pod "abb461c4-9707-4b5c-aba0-793f7bb993e5" (UID: "abb461c4-9707-4b5c-aba0-793f7bb993e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:03:23.086195 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:23.086167 5196 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abb461c4-9707-4b5c-aba0-793f7bb993e5-tls-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:23.086195 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:23.086192 5196 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abb461c4-9707-4b5c-aba0-793f7bb993e5-dshm\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:23.086390 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:23.086203 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbrg4\" (UniqueName: \"kubernetes.io/projected/abb461c4-9707-4b5c-aba0-793f7bb993e5-kube-api-access-jbrg4\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:23.799795 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:23.799762 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" Nov 03 23:03:23.800306 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:23.799796 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp" event={"ID":"abb461c4-9707-4b5c-aba0-793f7bb993e5","Type":"ContainerDied","Data":"d9592524f951d87d9e1e180c5e36d306e54c89fe1204f7385f44e8a6398c1e28"} Nov 03 23:03:23.800306 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:23.799843 5196 scope.go:122] "RemoveContainer" containerID="beee901e27bf51231fbdbc562b681a34e9337823bd6b83d32898ad74583915ce" Nov 03 23:03:23.966540 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:23.966498 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp"] Nov 03 23:03:24.017371 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:24.017337 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-7afb1c/distinct-2-w1-7afb1c-kserve-54d5d5dbc5-b2rkp"] Nov 03 23:03:24.814426 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:24.814390 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abb461c4-9707-4b5c-aba0-793f7bb993e5" path="/var/lib/kubelet/pods/abb461c4-9707-4b5c-aba0-793f7bb993e5/volumes" Nov 03 23:03:25.017967 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:25.017933 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:25.917796 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:03:25.917759 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:03:29.517622 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:29.515098 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr"] Nov 03 23:03:29.517622 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:29.515333 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr" podUID="8e42253b-644f-4b19-9520-1224e7d1746d" containerName="maas-api" containerID="cri-o://b5b8619a973adca607afdf1f0ce3ccf67721e6eda1b2a41d772aa8407339fe29" gracePeriod=30 Nov 03 23:03:29.620596 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:29.620558 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:29.839397 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:29.839365 5196 generic.go:356] "Generic (PLEG): container finished" podID="8e42253b-644f-4b19-9520-1224e7d1746d" containerID="b5b8619a973adca607afdf1f0ce3ccf67721e6eda1b2a41d772aa8407339fe29" exitCode=0 Nov 03 23:03:29.839520 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:29.839455 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr" event={"ID":"8e42253b-644f-4b19-9520-1224e7d1746d","Type":"ContainerDied","Data":"b5b8619a973adca607afdf1f0ce3ccf67721e6eda1b2a41d772aa8407339fe29"} Nov 03 23:03:29.839597 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:29.839527 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr" event={"ID":"8e42253b-644f-4b19-9520-1224e7d1746d","Type":"ContainerDied","Data":"40f29e5e4d6d4f5afbf7b08298638ce2d01a0b4152db3674cd7ed476e58074d3"} Nov 03 23:03:29.839597 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:29.839559 5196 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40f29e5e4d6d4f5afbf7b08298638ce2d01a0b4152db3674cd7ed476e58074d3" Nov 03 23:03:29.850353 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:29.850331 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr" Nov 03 23:03:29.975974 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:29.975936 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e42253b-644f-4b19-9520-1224e7d1746d-kube-api-access-vxgqp\" (UniqueName: \"kubernetes.io/projected/8e42253b-644f-4b19-9520-1224e7d1746d-kube-api-access-vxgqp\") pod \"8e42253b-644f-4b19-9520-1224e7d1746d\" (UID: \"8e42253b-644f-4b19-9520-1224e7d1746d\") " Nov 03 23:03:29.976168 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:29.976004 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e42253b-644f-4b19-9520-1224e7d1746d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e42253b-644f-4b19-9520-1224e7d1746d-metrics-certs\") pod \"8e42253b-644f-4b19-9520-1224e7d1746d\" (UID: \"8e42253b-644f-4b19-9520-1224e7d1746d\") " Nov 03 23:03:29.976168 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:29.976152 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e42253b-644f-4b19-9520-1224e7d1746d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e42253b-644f-4b19-9520-1224e7d1746d-openshift-service-ca\") pod \"8e42253b-644f-4b19-9520-1224e7d1746d\" (UID: \"8e42253b-644f-4b19-9520-1224e7d1746d\") " Nov 03 23:03:29.976296 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:29.976210 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e42253b-644f-4b19-9520-1224e7d1746d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e42253b-644f-4b19-9520-1224e7d1746d-maas-api-tls\") pod \"8e42253b-644f-4b19-9520-1224e7d1746d\" (UID: \"8e42253b-644f-4b19-9520-1224e7d1746d\") " Nov 03 23:03:29.977748 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:29.977693 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e42253b-644f-4b19-9520-1224e7d1746d-openshift-service-ca" pod "8e42253b-644f-4b19-9520-1224e7d1746d" (UID: "8e42253b-644f-4b19-9520-1224e7d1746d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:29.981563 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:29.981469 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e42253b-644f-4b19-9520-1224e7d1746d-metrics-certs" pod "8e42253b-644f-4b19-9520-1224e7d1746d" (UID: "8e42253b-644f-4b19-9520-1224e7d1746d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:03:29.982294 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:29.982257 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e42253b-644f-4b19-9520-1224e7d1746d-maas-api-tls" pod "8e42253b-644f-4b19-9520-1224e7d1746d" (UID: "8e42253b-644f-4b19-9520-1224e7d1746d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:03:29.984117 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:29.984081 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e42253b-644f-4b19-9520-1224e7d1746d-kube-api-access-vxgqp" pod "8e42253b-644f-4b19-9520-1224e7d1746d" (UID: "8e42253b-644f-4b19-9520-1224e7d1746d"). InnerVolumeSpecName "kube-api-access-vxgqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:30.078204 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:30.078177 5196 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e42253b-644f-4b19-9520-1224e7d1746d-openshift-service-ca\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:30.078204 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:30.078203 5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e42253b-644f-4b19-9520-1224e7d1746d-maas-api-tls\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:30.078396 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:30.078213 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxgqp\" (UniqueName: \"kubernetes.io/projected/8e42253b-644f-4b19-9520-1224e7d1746d-kube-api-access-vxgqp\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:30.078396 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:30.078222 5196 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e42253b-644f-4b19-9520-1224e7d1746d-metrics-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:30.845707 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:30.845674 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr" Nov 03 23:03:31.014673 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:31.014645 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr"] Nov 03 23:03:31.018734 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:31.018711 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ea5c8f-65df879f45-flvqr"] Nov 03 23:03:31.164118 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:31.164087 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:31.485430 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:31.485340 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:31.636600 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:31.636572 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:32.170819 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:32.170781 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:32.815564 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:32.815530 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e42253b-644f-4b19-9520-1224e7d1746d" path="/var/lib/kubelet/pods/8e42253b-644f-4b19-9520-1224e7d1746d/volumes" Nov 03 23:03:34.894820 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:34.894784 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:35.039281 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:35.039251 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:36.675127 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:36.675098 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:37.975100 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:37.975066 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:38.128907 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:38.128872 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:38.320687 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:38.320594 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm"] Nov 03 23:03:38.320927 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:38.320900 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" podUID="c332f22b-f8b4-4b95-a2dc-bac68ef527d7" containerName="istio-proxy" containerID="cri-o://8a056dc2ca33477d802a1ccc1ac8d74e8b417b361bbfffee81d9b79a5a2c54ef" gracePeriod=30 Nov 03 23:03:38.332550 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:38.332495 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:40.139059 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:40.139009 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:41.248237 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:41.248203 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:41.394695 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:41.394665 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:41.836898 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:41.836852 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:42.687696 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:42.687659 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd"] Nov 03 23:03:42.688210 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:42.687962 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" podUID="aa304409-0e6c-4543-a6ae-3db66214e80d" containerName="istio-proxy" containerID="cri-o://fe7fc0df29dd3cc632af54d2c3ad99ea040dedd418576baaa425cf3ebf832a65" gracePeriod=30 Nov 03 23:03:43.584624 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:43.584600 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 23:03:43.643315 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:43.643290 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-istio-podinfo\") pod \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\" (UID: \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\") " Nov 03 23:03:43.643482 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:43.643341 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-istiod-ca-cert\") pod \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\" (UID: \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\") " Nov 03 23:03:43.643482 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:43.643449 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-istio-data\") pod \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\" (UID: \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\") " Nov 03 23:03:43.643482 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:43.643478 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-workload-certs\") pod \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\" (UID: \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\") " Nov 03 23:03:43.643650 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:43.643518 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-workload-socket\") pod \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\" (UID: \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\") " Nov 03 23:03:43.643650 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:43.643603 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-kube-api-access-qzkzw\" (UniqueName: \"kubernetes.io/projected/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-kube-api-access-qzkzw\") pod \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\" (UID: \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\") " Nov 03 23:03:43.643753 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:43.643704 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-credential-socket\") pod \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\" (UID: \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\") " Nov 03 23:03:43.643753 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:43.643739 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-istio-envoy\") pod \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\" (UID: \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\") " Nov 03 23:03:43.643892 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:43.643743 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-workload-certs" pod "c332f22b-f8b4-4b95-a2dc-bac68ef527d7" (UID: "c332f22b-f8b4-4b95-a2dc-bac68ef527d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:43.643892 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:43.643777 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-istio-token\" (UniqueName: \"kubernetes.io/projected/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-istio-token\") pod \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\" (UID: \"c332f22b-f8b4-4b95-a2dc-bac68ef527d7\") " Nov 03 23:03:43.643892 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:43.643786 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-istio-data" pod "c332f22b-f8b4-4b95-a2dc-bac68ef527d7" (UID: "c332f22b-f8b4-4b95-a2dc-bac68ef527d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:43.643892 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:43.643795 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-workload-socket" pod "c332f22b-f8b4-4b95-a2dc-bac68ef527d7" (UID: "c332f22b-f8b4-4b95-a2dc-bac68ef527d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:43.643892 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:43.643762 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-istiod-ca-cert" pod "c332f22b-f8b4-4b95-a2dc-bac68ef527d7" (UID: "c332f22b-f8b4-4b95-a2dc-bac68ef527d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:43.644140 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:43.644055 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-credential-socket" pod "c332f22b-f8b4-4b95-a2dc-bac68ef527d7" (UID: "c332f22b-f8b4-4b95-a2dc-bac68ef527d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:43.644555 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:43.644460 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:43.644555 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:43.644493 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:43.644555 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:43.644509 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:43.644555 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:43.644525 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:43.644555 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:43.644539 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:43.645793 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:43.645773 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-istio-podinfo" pod "c332f22b-f8b4-4b95-a2dc-bac68ef527d7" (UID: "c332f22b-f8b4-4b95-a2dc-bac68ef527d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:03:43.646007 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:43.645985 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-kube-api-access-qzkzw" pod "c332f22b-f8b4-4b95-a2dc-bac68ef527d7" (UID: "c332f22b-f8b4-4b95-a2dc-bac68ef527d7"). InnerVolumeSpecName "kube-api-access-qzkzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:43.646085 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:43.646013 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-istio-token" pod "c332f22b-f8b4-4b95-a2dc-bac68ef527d7" (UID: "c332f22b-f8b4-4b95-a2dc-bac68ef527d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:43.646085 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:43.646053 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-istio-envoy" pod "c332f22b-f8b4-4b95-a2dc-bac68ef527d7" (UID: "c332f22b-f8b4-4b95-a2dc-bac68ef527d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:43.745852 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:43.745827 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:43.745852 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:43.745867 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:43.746219 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:43.745878 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:43.746219 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:43.745886 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzkzw\" (UniqueName: \"kubernetes.io/projected/c332f22b-f8b4-4b95-a2dc-bac68ef527d7-kube-api-access-qzkzw\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:43.930383 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:43.930291 5196 generic.go:356] "Generic (PLEG): container finished" podID="c332f22b-f8b4-4b95-a2dc-bac68ef527d7" containerID="8a056dc2ca33477d802a1ccc1ac8d74e8b417b361bbfffee81d9b79a5a2c54ef" exitCode=0 Nov 03 23:03:43.930383 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:43.930342 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" Nov 03 23:03:43.930383 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:43.930357 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" event={"ID":"c332f22b-f8b4-4b95-a2dc-bac68ef527d7","Type":"ContainerDied","Data":"8a056dc2ca33477d802a1ccc1ac8d74e8b417b361bbfffee81d9b79a5a2c54ef"} Nov 03 23:03:43.930643 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:43.930412 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm" event={"ID":"c332f22b-f8b4-4b95-a2dc-bac68ef527d7","Type":"ContainerDied","Data":"aa6bfc3d1af80f8b2cb8d1223fc92f3aacfc7af1e045706cff328a62954c77a2"} Nov 03 23:03:43.930643 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:43.930440 5196 scope.go:122] "RemoveContainer" containerID="8a056dc2ca33477d802a1ccc1ac8d74e8b417b361bbfffee81d9b79a5a2c54ef" Nov 03 23:03:43.944380 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:43.944337 5196 scope.go:122] "RemoveContainer" containerID="bf358276393c80b5a63eefe869e46ca83776e0e6652062f8c9b31f4967180499" Nov 03 23:03:43.953417 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:43.953400 5196 scope.go:122] "RemoveContainer" containerID="8a056dc2ca33477d802a1ccc1ac8d74e8b417b361bbfffee81d9b79a5a2c54ef" Nov 03 23:03:43.953699 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:03:43.953681 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a056dc2ca33477d802a1ccc1ac8d74e8b417b361bbfffee81d9b79a5a2c54ef\": container with ID starting with 8a056dc2ca33477d802a1ccc1ac8d74e8b417b361bbfffee81d9b79a5a2c54ef not found: ID does not exist" containerID="8a056dc2ca33477d802a1ccc1ac8d74e8b417b361bbfffee81d9b79a5a2c54ef" Nov 03 23:03:43.953743 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:43.953709 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a056dc2ca33477d802a1ccc1ac8d74e8b417b361bbfffee81d9b79a5a2c54ef"} err="failed to get container status \"8a056dc2ca33477d802a1ccc1ac8d74e8b417b361bbfffee81d9b79a5a2c54ef\": rpc error: code = NotFound desc = could not find container \"8a056dc2ca33477d802a1ccc1ac8d74e8b417b361bbfffee81d9b79a5a2c54ef\": container with ID starting with 8a056dc2ca33477d802a1ccc1ac8d74e8b417b361bbfffee81d9b79a5a2c54ef not found: ID does not exist" Nov 03 23:03:43.953743 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:43.953727 5196 scope.go:122] "RemoveContainer" containerID="bf358276393c80b5a63eefe869e46ca83776e0e6652062f8c9b31f4967180499" Nov 03 23:03:43.954048 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:03:43.954026 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf358276393c80b5a63eefe869e46ca83776e0e6652062f8c9b31f4967180499\": container with ID starting with bf358276393c80b5a63eefe869e46ca83776e0e6652062f8c9b31f4967180499 not found: ID does not exist" containerID="bf358276393c80b5a63eefe869e46ca83776e0e6652062f8c9b31f4967180499" Nov 03 23:03:43.954090 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:43.954055 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf358276393c80b5a63eefe869e46ca83776e0e6652062f8c9b31f4967180499"} err="failed to get container status \"bf358276393c80b5a63eefe869e46ca83776e0e6652062f8c9b31f4967180499\": rpc error: code = NotFound desc = could not find container \"bf358276393c80b5a63eefe869e46ca83776e0e6652062f8c9b31f4967180499\": container with ID starting with bf358276393c80b5a63eefe869e46ca83776e0e6652062f8c9b31f4967180499 not found: ID does not exist" Nov 03 23:03:44.117767 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:44.117731 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm"] Nov 03 23:03:44.130155 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:44.130108 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ea5c8f-openshift-default-54496b9b7c-d2vcm"] Nov 03 23:03:44.815536 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:44.815502 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c332f22b-f8b4-4b95-a2dc-bac68ef527d7" path="/var/lib/kubelet/pods/c332f22b-f8b4-4b95-a2dc-bac68ef527d7/volumes" Nov 03 23:03:45.197910 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:45.197873 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:45.606569 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:45.606532 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:45.761527 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:45.761493 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:46.246536 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:46.246505 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:47.036036 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:47.035995 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:47.956620 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:47.956213 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 23:03:47.959828 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:47.959794 5196 generic.go:356] "Generic (PLEG): container finished" podID="aa304409-0e6c-4543-a6ae-3db66214e80d" containerID="fe7fc0df29dd3cc632af54d2c3ad99ea040dedd418576baaa425cf3ebf832a65" exitCode=0 Nov 03 23:03:47.960017 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:47.959908 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" Nov 03 23:03:47.960091 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:47.960036 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" event={"ID":"aa304409-0e6c-4543-a6ae-3db66214e80d","Type":"ContainerDied","Data":"fe7fc0df29dd3cc632af54d2c3ad99ea040dedd418576baaa425cf3ebf832a65"} Nov 03 23:03:47.960091 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:47.960085 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd" event={"ID":"aa304409-0e6c-4543-a6ae-3db66214e80d","Type":"ContainerDied","Data":"b2c29d11276b95c797b51b10c53742eb4fa69c1dbb4a18c4bc7b57dc58729e81"} Nov 03 23:03:47.960193 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:47.960114 5196 scope.go:122] "RemoveContainer" containerID="fe7fc0df29dd3cc632af54d2c3ad99ea040dedd418576baaa425cf3ebf832a65" Nov 03 23:03:47.979087 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:47.976723 5196 scope.go:122] "RemoveContainer" containerID="78d4402e5d9a4616c7d5fbb4fd1456c482bf0a981e8c4f2d101b043b89a7f391" Nov 03 23:03:47.992701 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:47.992658 5196 scope.go:122] "RemoveContainer" containerID="fe7fc0df29dd3cc632af54d2c3ad99ea040dedd418576baaa425cf3ebf832a65" Nov 03 23:03:47.992990 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:47.992968 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa304409-0e6c-4543-a6ae-3db66214e80d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa304409-0e6c-4543-a6ae-3db66214e80d-credential-socket\") pod \"aa304409-0e6c-4543-a6ae-3db66214e80d\" (UID: \"aa304409-0e6c-4543-a6ae-3db66214e80d\") " Nov 03 23:03:47.993140 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:47.993016 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa304409-0e6c-4543-a6ae-3db66214e80d-istio-token\" (UniqueName: \"kubernetes.io/projected/aa304409-0e6c-4543-a6ae-3db66214e80d-istio-token\") pod \"aa304409-0e6c-4543-a6ae-3db66214e80d\" (UID: \"aa304409-0e6c-4543-a6ae-3db66214e80d\") " Nov 03 23:03:47.993140 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:47.993091 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa304409-0e6c-4543-a6ae-3db66214e80d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa304409-0e6c-4543-a6ae-3db66214e80d-workload-socket\") pod \"aa304409-0e6c-4543-a6ae-3db66214e80d\" (UID: \"aa304409-0e6c-4543-a6ae-3db66214e80d\") " Nov 03 23:03:47.993140 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:47.993133 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa304409-0e6c-4543-a6ae-3db66214e80d-kube-api-access-4mg5k\" (UniqueName: \"kubernetes.io/projected/aa304409-0e6c-4543-a6ae-3db66214e80d-kube-api-access-4mg5k\") pod \"aa304409-0e6c-4543-a6ae-3db66214e80d\" (UID: \"aa304409-0e6c-4543-a6ae-3db66214e80d\") " Nov 03 23:03:47.993309 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:47.993166 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa304409-0e6c-4543-a6ae-3db66214e80d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa304409-0e6c-4543-a6ae-3db66214e80d-istio-envoy\") pod \"aa304409-0e6c-4543-a6ae-3db66214e80d\" (UID: \"aa304409-0e6c-4543-a6ae-3db66214e80d\") " Nov 03 23:03:47.993309 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:47.993197 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa304409-0e6c-4543-a6ae-3db66214e80d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa304409-0e6c-4543-a6ae-3db66214e80d-istio-data\") pod \"aa304409-0e6c-4543-a6ae-3db66214e80d\" (UID: \"aa304409-0e6c-4543-a6ae-3db66214e80d\") " Nov 03 23:03:47.993309 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:47.993276 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa304409-0e6c-4543-a6ae-3db66214e80d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa304409-0e6c-4543-a6ae-3db66214e80d-istio-podinfo\") pod \"aa304409-0e6c-4543-a6ae-3db66214e80d\" (UID: \"aa304409-0e6c-4543-a6ae-3db66214e80d\") " Nov 03 23:03:47.993461 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:47.993321 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa304409-0e6c-4543-a6ae-3db66214e80d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa304409-0e6c-4543-a6ae-3db66214e80d-istiod-ca-cert\") pod \"aa304409-0e6c-4543-a6ae-3db66214e80d\" (UID: \"aa304409-0e6c-4543-a6ae-3db66214e80d\") " Nov 03 23:03:47.993461 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:03:47.993320 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe7fc0df29dd3cc632af54d2c3ad99ea040dedd418576baaa425cf3ebf832a65\": container with ID starting with fe7fc0df29dd3cc632af54d2c3ad99ea040dedd418576baaa425cf3ebf832a65 not found: ID does not exist" containerID="fe7fc0df29dd3cc632af54d2c3ad99ea040dedd418576baaa425cf3ebf832a65" Nov 03 23:03:47.993461 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:47.993363 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe7fc0df29dd3cc632af54d2c3ad99ea040dedd418576baaa425cf3ebf832a65"} err="failed to get container status \"fe7fc0df29dd3cc632af54d2c3ad99ea040dedd418576baaa425cf3ebf832a65\": rpc error: code = NotFound desc = could not find container \"fe7fc0df29dd3cc632af54d2c3ad99ea040dedd418576baaa425cf3ebf832a65\": container with ID starting with fe7fc0df29dd3cc632af54d2c3ad99ea040dedd418576baaa425cf3ebf832a65 not found: ID does not exist" Nov 03 23:03:47.993461 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:47.993380 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa304409-0e6c-4543-a6ae-3db66214e80d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa304409-0e6c-4543-a6ae-3db66214e80d-workload-certs\") pod \"aa304409-0e6c-4543-a6ae-3db66214e80d\" (UID: \"aa304409-0e6c-4543-a6ae-3db66214e80d\") " Nov 03 23:03:47.993461 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:47.993389 5196 scope.go:122] "RemoveContainer" containerID="78d4402e5d9a4616c7d5fbb4fd1456c482bf0a981e8c4f2d101b043b89a7f391" Nov 03 23:03:47.994016 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:03:47.993987 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78d4402e5d9a4616c7d5fbb4fd1456c482bf0a981e8c4f2d101b043b89a7f391\": container with ID starting with 78d4402e5d9a4616c7d5fbb4fd1456c482bf0a981e8c4f2d101b043b89a7f391 not found: ID does not exist" containerID="78d4402e5d9a4616c7d5fbb4fd1456c482bf0a981e8c4f2d101b043b89a7f391" Nov 03 23:03:47.994127 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:47.994028 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78d4402e5d9a4616c7d5fbb4fd1456c482bf0a981e8c4f2d101b043b89a7f391"} err="failed to get container status \"78d4402e5d9a4616c7d5fbb4fd1456c482bf0a981e8c4f2d101b043b89a7f391\": rpc error: code = NotFound desc = could not find container \"78d4402e5d9a4616c7d5fbb4fd1456c482bf0a981e8c4f2d101b043b89a7f391\": container with ID starting with 78d4402e5d9a4616c7d5fbb4fd1456c482bf0a981e8c4f2d101b043b89a7f391 not found: ID does not exist" Nov 03 23:03:47.994258 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:47.994231 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa304409-0e6c-4543-a6ae-3db66214e80d-workload-certs" pod "aa304409-0e6c-4543-a6ae-3db66214e80d" (UID: "aa304409-0e6c-4543-a6ae-3db66214e80d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:47.994426 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:47.994406 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa304409-0e6c-4543-a6ae-3db66214e80d-istio-data" pod "aa304409-0e6c-4543-a6ae-3db66214e80d" (UID: "aa304409-0e6c-4543-a6ae-3db66214e80d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:47.994506 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:47.994453 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa304409-0e6c-4543-a6ae-3db66214e80d-istiod-ca-cert" pod "aa304409-0e6c-4543-a6ae-3db66214e80d" (UID: "aa304409-0e6c-4543-a6ae-3db66214e80d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:47.994506 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:47.994469 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa304409-0e6c-4543-a6ae-3db66214e80d-workload-socket" pod "aa304409-0e6c-4543-a6ae-3db66214e80d" (UID: "aa304409-0e6c-4543-a6ae-3db66214e80d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:47.995626 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:47.995602 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa304409-0e6c-4543-a6ae-3db66214e80d-istio-token" pod "aa304409-0e6c-4543-a6ae-3db66214e80d" (UID: "aa304409-0e6c-4543-a6ae-3db66214e80d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:47.996219 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:47.996185 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa304409-0e6c-4543-a6ae-3db66214e80d-istio-envoy" pod "aa304409-0e6c-4543-a6ae-3db66214e80d" (UID: "aa304409-0e6c-4543-a6ae-3db66214e80d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:47.996433 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:47.996408 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa304409-0e6c-4543-a6ae-3db66214e80d-istio-podinfo" pod "aa304409-0e6c-4543-a6ae-3db66214e80d" (UID: "aa304409-0e6c-4543-a6ae-3db66214e80d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:03:47.996559 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:47.996539 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa304409-0e6c-4543-a6ae-3db66214e80d-kube-api-access-4mg5k" pod "aa304409-0e6c-4543-a6ae-3db66214e80d" (UID: "aa304409-0e6c-4543-a6ae-3db66214e80d"). InnerVolumeSpecName "kube-api-access-4mg5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:47.996955 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:47.996935 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa304409-0e6c-4543-a6ae-3db66214e80d-credential-socket" pod "aa304409-0e6c-4543-a6ae-3db66214e80d" (UID: "aa304409-0e6c-4543-a6ae-3db66214e80d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:48.095343 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:48.095306 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa304409-0e6c-4543-a6ae-3db66214e80d-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:48.095343 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:48.095337 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa304409-0e6c-4543-a6ae-3db66214e80d-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:48.095343 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:48.095350 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa304409-0e6c-4543-a6ae-3db66214e80d-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:48.095583 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:48.095362 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa304409-0e6c-4543-a6ae-3db66214e80d-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:48.095583 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:48.095374 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mg5k\" (UniqueName: \"kubernetes.io/projected/aa304409-0e6c-4543-a6ae-3db66214e80d-kube-api-access-4mg5k\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:48.095583 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:48.095386 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa304409-0e6c-4543-a6ae-3db66214e80d-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:48.095583 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:48.095397 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa304409-0e6c-4543-a6ae-3db66214e80d-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:48.095583 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:48.095408 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa304409-0e6c-4543-a6ae-3db66214e80d-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:48.095583 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:48.095420 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa304409-0e6c-4543-a6ae-3db66214e80d-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:03:48.385896 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:48.385848 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd"] Nov 03 23:03:48.390698 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:48.390673 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-164c36-openshift-default-54b556b4c9-ldktd"] Nov 03 23:03:48.819101 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:48.819069 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa304409-0e6c-4543-a6ae-3db66214e80d" path="/var/lib/kubelet/pods/aa304409-0e6c-4543-a6ae-3db66214e80d/volumes" Nov 03 23:03:51.776784 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:51.776756 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:51.927639 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:51.927573 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:52.226900 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:52.226815 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:54.183076 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:54.183033 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:54.334774 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:54.334737 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:55.922047 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:03:55.922008 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:03:57.046659 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.046622 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh"] Nov 03 23:03:57.047439 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.047416 5196 cpu_manager.go:414] "containerMap: removing container" podUID="c332f22b-f8b4-4b95-a2dc-bac68ef527d7" containerName="istio-proxy" Nov 03 23:03:57.047496 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.047445 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c332f22b-f8b4-4b95-a2dc-bac68ef527d7" containerName="istio-proxy" Nov 03 23:03:57.047496 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.047479 5196 cpu_manager.go:414] "containerMap: removing container" podUID="c332f22b-f8b4-4b95-a2dc-bac68ef527d7" containerName="istio-proxy" Nov 03 23:03:57.047496 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.047486 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c332f22b-f8b4-4b95-a2dc-bac68ef527d7" containerName="istio-proxy" Nov 03 23:03:57.047596 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.047501 5196 cpu_manager.go:414] "containerMap: removing container" podUID="50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7" containerName="istio-proxy" Nov 03 23:03:57.047596 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.047510 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7" containerName="istio-proxy" Nov 03 23:03:57.047596 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.047522 5196 cpu_manager.go:414] "containerMap: removing container" podUID="aa304409-0e6c-4543-a6ae-3db66214e80d" containerName="istio-proxy" Nov 03 23:03:57.047596 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.047529 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa304409-0e6c-4543-a6ae-3db66214e80d" containerName="istio-proxy" Nov 03 23:03:57.047596 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.047543 5196 cpu_manager.go:414] "containerMap: removing container" podUID="abb461c4-9707-4b5c-aba0-793f7bb993e5" containerName="main" Nov 03 23:03:57.047596 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.047550 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abb461c4-9707-4b5c-aba0-793f7bb993e5" containerName="main" Nov 03 23:03:57.047596 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.047575 5196 cpu_manager.go:414] "containerMap: removing container" podUID="8e42253b-644f-4b19-9520-1224e7d1746d" containerName="maas-api" Nov 03 23:03:57.047596 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.047582 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e42253b-644f-4b19-9520-1224e7d1746d" containerName="maas-api" Nov 03 23:03:57.047596 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.047594 5196 cpu_manager.go:414] "containerMap: removing container" podUID="aa304409-0e6c-4543-a6ae-3db66214e80d" containerName="istio-proxy" Nov 03 23:03:57.047884 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.047601 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa304409-0e6c-4543-a6ae-3db66214e80d" containerName="istio-proxy" Nov 03 23:03:57.047884 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.047752 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="50bb7d6c-e09f-424f-8e80-8dbbbe4b7bf7" containerName="istio-proxy" Nov 03 23:03:57.047884 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.047765 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa304409-0e6c-4543-a6ae-3db66214e80d" containerName="istio-proxy" Nov 03 23:03:57.047884 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.047774 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="c332f22b-f8b4-4b95-a2dc-bac68ef527d7" containerName="istio-proxy" Nov 03 23:03:57.047884 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.047785 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="abb461c4-9707-4b5c-aba0-793f7bb993e5" containerName="main" Nov 03 23:03:57.047884 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.047796 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="c332f22b-f8b4-4b95-a2dc-bac68ef527d7" containerName="istio-proxy" Nov 03 23:03:57.047884 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.047807 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e42253b-644f-4b19-9520-1224e7d1746d" containerName="maas-api" Nov 03 23:03:57.053232 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.053208 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh" Nov 03 23:03:57.109062 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.106704 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-cfe1bd-serving-cert\"" Nov 03 23:03:57.110916 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.110884 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh"] Nov 03 23:03:57.209062 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.208996 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6540706-d21a-4432-8f80-a6d427a13153-openshift-service-ca\") pod \"maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh\" (UID: \"d6540706-d21a-4432-8f80-a6d427a13153\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh" Nov 03 23:03:57.209259 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.209115 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6540706-d21a-4432-8f80-a6d427a13153-maas-api-tls\") pod \"maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh\" (UID: \"d6540706-d21a-4432-8f80-a6d427a13153\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh" Nov 03 23:03:57.209259 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.209172 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmqbw\" (UniqueName: \"kubernetes.io/projected/d6540706-d21a-4432-8f80-a6d427a13153-kube-api-access-rmqbw\") pod \"maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh\" (UID: \"d6540706-d21a-4432-8f80-a6d427a13153\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh" Nov 03 23:03:57.209259 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.209218 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6540706-d21a-4432-8f80-a6d427a13153-metrics-certs\") pod \"maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh\" (UID: \"d6540706-d21a-4432-8f80-a6d427a13153\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh" Nov 03 23:03:57.288771 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.288734 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:57.311103 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.311007 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6540706-d21a-4432-8f80-a6d427a13153-openshift-service-ca\") pod \"maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh\" (UID: \"d6540706-d21a-4432-8f80-a6d427a13153\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh" Nov 03 23:03:57.311103 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.311067 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6540706-d21a-4432-8f80-a6d427a13153-maas-api-tls\") pod \"maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh\" (UID: \"d6540706-d21a-4432-8f80-a6d427a13153\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh" Nov 03 23:03:57.311103 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.311103 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmqbw\" (UniqueName: \"kubernetes.io/projected/d6540706-d21a-4432-8f80-a6d427a13153-kube-api-access-rmqbw\") pod \"maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh\" (UID: \"d6540706-d21a-4432-8f80-a6d427a13153\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh" Nov 03 23:03:57.311412 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.311130 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6540706-d21a-4432-8f80-a6d427a13153-metrics-certs\") pod \"maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh\" (UID: \"d6540706-d21a-4432-8f80-a6d427a13153\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh" Nov 03 23:03:57.311412 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:03:57.311242 5196 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-cfe1bd-serving-cert: secret "maas-api-e2e-api-b-cfe1bd-serving-cert" not found Nov 03 23:03:57.311412 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:03:57.311330 5196 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6540706-d21a-4432-8f80-a6d427a13153-maas-api-tls podName:d6540706-d21a-4432-8f80-a6d427a13153 nodeName:}" failed. No retries permitted until 2026-11-03 23:03:57.811310112 +0000 UTC m=+2467.120869397 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d6540706-d21a-4432-8f80-a6d427a13153-maas-api-tls") pod "maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh" (UID: "d6540706-d21a-4432-8f80-a6d427a13153") : secret "maas-api-e2e-api-b-cfe1bd-serving-cert" not found Nov 03 23:03:57.311721 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.311701 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6540706-d21a-4432-8f80-a6d427a13153-openshift-service-ca\") pod \"maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh\" (UID: \"d6540706-d21a-4432-8f80-a6d427a13153\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh" Nov 03 23:03:57.314251 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.314224 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6540706-d21a-4432-8f80-a6d427a13153-metrics-certs\") pod \"maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh\" (UID: \"d6540706-d21a-4432-8f80-a6d427a13153\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh" Nov 03 23:03:57.374058 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.374024 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmqbw\" (UniqueName: \"kubernetes.io/projected/d6540706-d21a-4432-8f80-a6d427a13153-kube-api-access-rmqbw\") pod \"maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh\" (UID: \"d6540706-d21a-4432-8f80-a6d427a13153\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh" Nov 03 23:03:57.541813 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.541782 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:57.687149 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.687113 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:03:57.817849 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.817807 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6540706-d21a-4432-8f80-a6d427a13153-maas-api-tls\") pod \"maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh\" (UID: \"d6540706-d21a-4432-8f80-a6d427a13153\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh" Nov 03 23:03:57.820367 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.820343 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6540706-d21a-4432-8f80-a6d427a13153-maas-api-tls\") pod \"maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh\" (UID: \"d6540706-d21a-4432-8f80-a6d427a13153\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh" Nov 03 23:03:57.967063 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:57.966975 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh" Nov 03 23:03:58.349732 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:58.349704 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh"] Nov 03 23:03:58.351879 ip-10-0-143-132 kubenswrapper[5196]: W1103 23:03:58.351833 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6540706_d21a_4432_8f80_a6d427a13153.slice/crio-f5e2eff4174da4a6c9ea3603e3b5d89f8649ba52216910e51873f9e9c1f59228 WatchSource:0}: Error finding container f5e2eff4174da4a6c9ea3603e3b5d89f8649ba52216910e51873f9e9c1f59228: Status 404 returned error can't find the container with id f5e2eff4174da4a6c9ea3603e3b5d89f8649ba52216910e51873f9e9c1f59228 Nov 03 23:03:59.034040 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:03:59.034003 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh" event={"ID":"d6540706-d21a-4432-8f80-a6d427a13153","Type":"ContainerStarted","Data":"f5e2eff4174da4a6c9ea3603e3b5d89f8649ba52216910e51873f9e9c1f59228"} Nov 03 23:04:00.041018 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:00.040973 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh" event={"ID":"d6540706-d21a-4432-8f80-a6d427a13153","Type":"ContainerStarted","Data":"9a0cb4ce4f35449acecc7a17e5b9ae1b4d3990114336ce870f420f4e4d389dc2"} Nov 03 23:04:00.041498 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:00.041121 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh" Nov 03 23:04:00.182546 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:00.182492 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh" podStartSLOduration=3.386060731 podStartE2EDuration="4.182478185s" totalImagesPullingTime="796.417454ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:03:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:04:00.181274011 +0000 UTC m=+2469.490833305" watchObservedRunningTime="2026-11-03 23:04:00.182478185 +0000 UTC m=+2469.492037478" Nov 03 23:04:02.760644 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:02.760605 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:06.052420 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:06.052383 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh" Nov 03 23:04:07.816472 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:07.816438 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:10.636082 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:10.636031 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:12.922237 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:12.922208 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:15.982392 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:15.982358 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:17.093948 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:17.093909 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5"] Nov 03 23:04:17.098213 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:17.098173 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5" podUID="994606f4-8a09-4d39-8310-d25dc898c140" containerName="maas-api" containerID="cri-o://cefea8cf76843c208e38dadff2c819a38c1c0e045d95399f356f7106eebf7b9e" gracePeriod=30 Nov 03 23:04:17.180685 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:17.180653 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:17.446165 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:17.446135 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5" Nov 03 23:04:17.538081 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:17.536967 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/994606f4-8a09-4d39-8310-d25dc898c140-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/994606f4-8a09-4d39-8310-d25dc898c140-openshift-service-ca\") pod \"994606f4-8a09-4d39-8310-d25dc898c140\" (UID: \"994606f4-8a09-4d39-8310-d25dc898c140\") " Nov 03 23:04:17.538081 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:17.537028 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/994606f4-8a09-4d39-8310-d25dc898c140-kube-api-access-v75j4\" (UniqueName: \"kubernetes.io/projected/994606f4-8a09-4d39-8310-d25dc898c140-kube-api-access-v75j4\") pod \"994606f4-8a09-4d39-8310-d25dc898c140\" (UID: \"994606f4-8a09-4d39-8310-d25dc898c140\") " Nov 03 23:04:17.538081 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:17.537075 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/994606f4-8a09-4d39-8310-d25dc898c140-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/994606f4-8a09-4d39-8310-d25dc898c140-maas-api-tls\") pod \"994606f4-8a09-4d39-8310-d25dc898c140\" (UID: \"994606f4-8a09-4d39-8310-d25dc898c140\") " Nov 03 23:04:17.538081 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:17.537226 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/994606f4-8a09-4d39-8310-d25dc898c140-metrics-certs\" (UniqueName: \"kubernetes.io/secret/994606f4-8a09-4d39-8310-d25dc898c140-metrics-certs\") pod \"994606f4-8a09-4d39-8310-d25dc898c140\" (UID: \"994606f4-8a09-4d39-8310-d25dc898c140\") " Nov 03 23:04:17.540244 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:17.540200 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/994606f4-8a09-4d39-8310-d25dc898c140-openshift-service-ca" pod "994606f4-8a09-4d39-8310-d25dc898c140" (UID: "994606f4-8a09-4d39-8310-d25dc898c140"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:04:17.540750 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:17.540727 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/994606f4-8a09-4d39-8310-d25dc898c140-metrics-certs" pod "994606f4-8a09-4d39-8310-d25dc898c140" (UID: "994606f4-8a09-4d39-8310-d25dc898c140"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:04:17.541589 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:17.541540 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/994606f4-8a09-4d39-8310-d25dc898c140-kube-api-access-v75j4" pod "994606f4-8a09-4d39-8310-d25dc898c140" (UID: "994606f4-8a09-4d39-8310-d25dc898c140"). InnerVolumeSpecName "kube-api-access-v75j4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:17.553021 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:17.552974 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/994606f4-8a09-4d39-8310-d25dc898c140-maas-api-tls" pod "994606f4-8a09-4d39-8310-d25dc898c140" (UID: "994606f4-8a09-4d39-8310-d25dc898c140"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:04:17.638890 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:17.638834 5196 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/994606f4-8a09-4d39-8310-d25dc898c140-metrics-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:17.638890 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:17.638886 5196 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/994606f4-8a09-4d39-8310-d25dc898c140-openshift-service-ca\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:17.639096 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:17.638899 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v75j4\" (UniqueName: \"kubernetes.io/projected/994606f4-8a09-4d39-8310-d25dc898c140-kube-api-access-v75j4\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:17.639096 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:17.638910 5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/994606f4-8a09-4d39-8310-d25dc898c140-maas-api-tls\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:18.154583 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:18.154548 5196 generic.go:356] "Generic (PLEG): container finished" podID="994606f4-8a09-4d39-8310-d25dc898c140" containerID="cefea8cf76843c208e38dadff2c819a38c1c0e045d95399f356f7106eebf7b9e" exitCode=0 Nov 03 23:04:18.155153 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:18.154619 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5" Nov 03 23:04:18.155153 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:18.154632 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5" event={"ID":"994606f4-8a09-4d39-8310-d25dc898c140","Type":"ContainerDied","Data":"cefea8cf76843c208e38dadff2c819a38c1c0e045d95399f356f7106eebf7b9e"} Nov 03 23:04:18.155153 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:18.154698 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5" event={"ID":"994606f4-8a09-4d39-8310-d25dc898c140","Type":"ContainerDied","Data":"7cc1cb87cf0827a7c2d1ec508620404a72d62939328289d333e4914a336f9126"} Nov 03 23:04:18.155153 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:18.154712 5196 scope.go:122] "RemoveContainer" containerID="cefea8cf76843c208e38dadff2c819a38c1c0e045d95399f356f7106eebf7b9e" Nov 03 23:04:18.167836 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:18.167798 5196 scope.go:122] "RemoveContainer" containerID="cefea8cf76843c208e38dadff2c819a38c1c0e045d95399f356f7106eebf7b9e" Nov 03 23:04:18.168168 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:04:18.168150 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cefea8cf76843c208e38dadff2c819a38c1c0e045d95399f356f7106eebf7b9e\": container with ID starting with cefea8cf76843c208e38dadff2c819a38c1c0e045d95399f356f7106eebf7b9e not found: ID does not exist" containerID="cefea8cf76843c208e38dadff2c819a38c1c0e045d95399f356f7106eebf7b9e" Nov 03 23:04:18.168224 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:18.168179 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cefea8cf76843c208e38dadff2c819a38c1c0e045d95399f356f7106eebf7b9e"} err="failed to get container status \"cefea8cf76843c208e38dadff2c819a38c1c0e045d95399f356f7106eebf7b9e\": rpc error: code = NotFound desc = could not find container \"cefea8cf76843c208e38dadff2c819a38c1c0e045d95399f356f7106eebf7b9e\": container with ID starting with cefea8cf76843c208e38dadff2c819a38c1c0e045d95399f356f7106eebf7b9e not found: ID does not exist" Nov 03 23:04:18.322348 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:18.322312 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5"] Nov 03 23:04:18.373036 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:18.373005 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-efd39a-85c768fdfb-qhsz5"] Nov 03 23:04:18.815905 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:18.815875 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="994606f4-8a09-4d39-8310-d25dc898c140" path="/var/lib/kubelet/pods/994606f4-8a09-4d39-8310-d25dc898c140/volumes" Nov 03 23:04:18.884489 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:18.884460 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:19.267756 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:19.267724 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:19.422277 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:19.422212 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:19.903225 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:19.903192 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:20.057518 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:20.057469 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:20.211499 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:20.211401 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:22.029696 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:22.029661 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:22.175564 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:22.175534 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:23.400842 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:23.400805 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:24.479039 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:24.478989 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:25.918878 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:04:25.918819 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:04:26.013416 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:26.013385 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:26.160699 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:26.160662 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:26.372420 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:26.372384 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:28.048529 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.048489 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk"] Nov 03 23:04:28.049288 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.049268 5196 cpu_manager.go:414] "containerMap: removing container" podUID="994606f4-8a09-4d39-8310-d25dc898c140" containerName="maas-api" Nov 03 23:04:28.049346 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.049292 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="994606f4-8a09-4d39-8310-d25dc898c140" containerName="maas-api" Nov 03 23:04:28.049426 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.049415 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="994606f4-8a09-4d39-8310-d25dc898c140" containerName="maas-api" Nov 03 23:04:28.049468 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.049435 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa304409-0e6c-4543-a6ae-3db66214e80d" containerName="istio-proxy" Nov 03 23:04:28.054523 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.054502 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" Nov 03 23:04:28.108188 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.108150 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-7afb1c-openshift-default-dockercfg-nlmnh\"" Nov 03 23:04:28.161961 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.161927 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk"] Nov 03 23:04:28.189006 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.188976 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eec3a9d-0eea-44e8-92e4-2d19e4508148-istio-envoy\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk\" (UID: \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" Nov 03 23:04:28.189173 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.189011 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eec3a9d-0eea-44e8-92e4-2d19e4508148-istio-token\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk\" (UID: \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" Nov 03 23:04:28.189173 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.189094 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eec3a9d-0eea-44e8-92e4-2d19e4508148-istiod-ca-cert\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk\" (UID: \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" Nov 03 23:04:28.189340 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.189311 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eec3a9d-0eea-44e8-92e4-2d19e4508148-istio-data\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk\" (UID: \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" Nov 03 23:04:28.189477 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.189353 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eec3a9d-0eea-44e8-92e4-2d19e4508148-credential-socket\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk\" (UID: \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" Nov 03 23:04:28.189477 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.189460 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eec3a9d-0eea-44e8-92e4-2d19e4508148-workload-socket\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk\" (UID: \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" Nov 03 23:04:28.189609 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.189520 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eec3a9d-0eea-44e8-92e4-2d19e4508148-workload-certs\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk\" (UID: \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" Nov 03 23:04:28.189609 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.189550 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvxgd\" (UniqueName: \"kubernetes.io/projected/9eec3a9d-0eea-44e8-92e4-2d19e4508148-kube-api-access-cvxgd\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk\" (UID: \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" Nov 03 23:04:28.189733 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.189685 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eec3a9d-0eea-44e8-92e4-2d19e4508148-istio-podinfo\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk\" (UID: \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" Nov 03 23:04:28.290392 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.290357 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eec3a9d-0eea-44e8-92e4-2d19e4508148-istio-envoy\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk\" (UID: \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" Nov 03 23:04:28.290392 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.290397 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eec3a9d-0eea-44e8-92e4-2d19e4508148-istio-token\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk\" (UID: \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" Nov 03 23:04:28.290631 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.290450 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eec3a9d-0eea-44e8-92e4-2d19e4508148-istiod-ca-cert\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk\" (UID: \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" Nov 03 23:04:28.290631 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.290517 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eec3a9d-0eea-44e8-92e4-2d19e4508148-istio-data\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk\" (UID: \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" Nov 03 23:04:28.290631 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.290547 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eec3a9d-0eea-44e8-92e4-2d19e4508148-credential-socket\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk\" (UID: \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" Nov 03 23:04:28.290631 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.290580 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eec3a9d-0eea-44e8-92e4-2d19e4508148-workload-socket\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk\" (UID: \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" Nov 03 23:04:28.290631 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.290622 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eec3a9d-0eea-44e8-92e4-2d19e4508148-workload-certs\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk\" (UID: \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" Nov 03 23:04:28.290939 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.290650 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvxgd\" (UniqueName: \"kubernetes.io/projected/9eec3a9d-0eea-44e8-92e4-2d19e4508148-kube-api-access-cvxgd\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk\" (UID: \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" Nov 03 23:04:28.290939 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.290735 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eec3a9d-0eea-44e8-92e4-2d19e4508148-istio-podinfo\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk\" (UID: \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" Nov 03 23:04:28.291056 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.291019 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eec3a9d-0eea-44e8-92e4-2d19e4508148-istio-data\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk\" (UID: \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" Nov 03 23:04:28.291231 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.291205 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eec3a9d-0eea-44e8-92e4-2d19e4508148-workload-socket\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk\" (UID: \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" Nov 03 23:04:28.291313 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.291288 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eec3a9d-0eea-44e8-92e4-2d19e4508148-credential-socket\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk\" (UID: \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" Nov 03 23:04:28.291313 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.291295 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eec3a9d-0eea-44e8-92e4-2d19e4508148-istiod-ca-cert\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk\" (UID: \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" Nov 03 23:04:28.291461 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.291442 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eec3a9d-0eea-44e8-92e4-2d19e4508148-workload-certs\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk\" (UID: \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" Nov 03 23:04:28.293521 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.293496 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eec3a9d-0eea-44e8-92e4-2d19e4508148-istio-envoy\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk\" (UID: \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" Nov 03 23:04:28.293740 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.293724 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eec3a9d-0eea-44e8-92e4-2d19e4508148-istio-podinfo\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk\" (UID: \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" Nov 03 23:04:28.352603 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.352568 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eec3a9d-0eea-44e8-92e4-2d19e4508148-istio-token\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk\" (UID: \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" Nov 03 23:04:28.352772 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.352702 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvxgd\" (UniqueName: \"kubernetes.io/projected/9eec3a9d-0eea-44e8-92e4-2d19e4508148-kube-api-access-cvxgd\") pod \"e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk\" (UID: \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\") " pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" Nov 03 23:04:28.372231 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.372200 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" Nov 03 23:04:28.816328 ip-10-0-143-132 kubenswrapper[5196]: W1103 23:04:28.816297 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eec3a9d_0eea_44e8_92e4_2d19e4508148.slice/crio-ac896323491c5e9e531c5907aca06866d019c40b34e9ec0f0648f6d5c142163e WatchSource:0}: Error finding container ac896323491c5e9e531c5907aca06866d019c40b34e9ec0f0648f6d5c142163e: Status 404 returned error can't find the container with id ac896323491c5e9e531c5907aca06866d019c40b34e9ec0f0648f6d5c142163e Nov 03 23:04:28.816974 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.816954 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk"] Nov 03 23:04:28.818780 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.818751 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:04:28.818849 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.818813 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:04:28.818921 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:28.818851 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:04:29.258262 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:29.258229 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" event={"ID":"9eec3a9d-0eea-44e8-92e4-2d19e4508148","Type":"ContainerStarted","Data":"294973e9950cb83e2c9be79bfc632e8119ca46c8982a4a8d67f8fbab70ddef71"} Nov 03 23:04:29.258656 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:29.258286 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" event={"ID":"9eec3a9d-0eea-44e8-92e4-2d19e4508148","Type":"ContainerStarted","Data":"ac896323491c5e9e531c5907aca06866d019c40b34e9ec0f0648f6d5c142163e"} Nov 03 23:04:29.291296 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:29.291263 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:29.373389 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:29.373361 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" Nov 03 23:04:29.374897 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:29.374876 5196 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 23:04:29.374972 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:29.374920 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" podUID="9eec3a9d-0eea-44e8-92e4-2d19e4508148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 23:04:29.382933 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:29.382878 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" podStartSLOduration=2.382836176 podStartE2EDuration="2.382836176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:04:29.380061315 +0000 UTC m=+2498.689620607" watchObservedRunningTime="2026-11-03 23:04:29.382836176 +0000 UTC m=+2498.692395470" Nov 03 23:04:29.603027 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:29.602990 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:30.373067 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:30.373034 5196 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 23:04:30.373467 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:30.373085 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" podUID="9eec3a9d-0eea-44e8-92e4-2d19e4508148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 23:04:31.373112 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:31.373080 5196 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 23:04:31.373484 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:31.373128 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" podUID="9eec3a9d-0eea-44e8-92e4-2d19e4508148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 23:04:32.372534 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:32.372503 5196 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 23:04:32.372692 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:32.372553 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" podUID="9eec3a9d-0eea-44e8-92e4-2d19e4508148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 23:04:33.373176 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:33.373140 5196 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 23:04:33.373698 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:33.373192 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" podUID="9eec3a9d-0eea-44e8-92e4-2d19e4508148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 23:04:34.373402 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:34.373365 5196 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 23:04:34.373881 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:34.373416 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" podUID="9eec3a9d-0eea-44e8-92e4-2d19e4508148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 23:04:34.878025 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:34.877985 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:35.372902 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:35.372847 5196 patch_prober.go:29] interesting pod/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 03 23:04:35.373094 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:35.372959 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" podUID="9eec3a9d-0eea-44e8-92e4-2d19e4508148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 03 23:04:36.941851 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:36.941803 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" Nov 03 23:04:36.942432 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:36.942407 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" Nov 03 23:04:36.943785 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:36.943767 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" Nov 03 23:04:39.040916 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:39.040884 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:39.190242 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:39.190211 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:39.353362 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:39.353334 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr"] Nov 03 23:04:39.353562 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:39.353529 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr" podUID="d4042310-83e8-44e6-b8e4-73ecb70403b3" containerName="maas-api" containerID="cri-o://f31f9fafc261f77015412837af9f54e9b660ca7379393f4953e1fc7bc6e96bd0" gracePeriod=30 Nov 03 23:04:39.477473 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:39.477437 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:39.644349 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:39.644319 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr" Nov 03 23:04:39.737189 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:39.734417 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4042310-83e8-44e6-b8e4-73ecb70403b3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4042310-83e8-44e6-b8e4-73ecb70403b3-metrics-certs\") pod \"d4042310-83e8-44e6-b8e4-73ecb70403b3\" (UID: \"d4042310-83e8-44e6-b8e4-73ecb70403b3\") " Nov 03 23:04:39.737189 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:39.734490 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4042310-83e8-44e6-b8e4-73ecb70403b3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4042310-83e8-44e6-b8e4-73ecb70403b3-maas-api-tls\") pod \"d4042310-83e8-44e6-b8e4-73ecb70403b3\" (UID: \"d4042310-83e8-44e6-b8e4-73ecb70403b3\") " Nov 03 23:04:39.737189 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:39.734560 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4042310-83e8-44e6-b8e4-73ecb70403b3-kube-api-access-svkzw\" (UniqueName: \"kubernetes.io/projected/d4042310-83e8-44e6-b8e4-73ecb70403b3-kube-api-access-svkzw\") pod \"d4042310-83e8-44e6-b8e4-73ecb70403b3\" (UID: \"d4042310-83e8-44e6-b8e4-73ecb70403b3\") " Nov 03 23:04:39.737189 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:39.734594 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4042310-83e8-44e6-b8e4-73ecb70403b3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4042310-83e8-44e6-b8e4-73ecb70403b3-openshift-service-ca\") pod \"d4042310-83e8-44e6-b8e4-73ecb70403b3\" (UID: \"d4042310-83e8-44e6-b8e4-73ecb70403b3\") " Nov 03 23:04:39.737189 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:39.735616 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4042310-83e8-44e6-b8e4-73ecb70403b3-openshift-service-ca" pod "d4042310-83e8-44e6-b8e4-73ecb70403b3" (UID: "d4042310-83e8-44e6-b8e4-73ecb70403b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:04:39.756037 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:39.755984 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4042310-83e8-44e6-b8e4-73ecb70403b3-maas-api-tls" pod "d4042310-83e8-44e6-b8e4-73ecb70403b3" (UID: "d4042310-83e8-44e6-b8e4-73ecb70403b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:04:39.767261 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:39.767225 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4042310-83e8-44e6-b8e4-73ecb70403b3-metrics-certs" pod "d4042310-83e8-44e6-b8e4-73ecb70403b3" (UID: "d4042310-83e8-44e6-b8e4-73ecb70403b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:04:39.776115 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:39.776045 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4042310-83e8-44e6-b8e4-73ecb70403b3-kube-api-access-svkzw" pod "d4042310-83e8-44e6-b8e4-73ecb70403b3" (UID: "d4042310-83e8-44e6-b8e4-73ecb70403b3"). InnerVolumeSpecName "kube-api-access-svkzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:39.836565 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:39.836528 5196 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4042310-83e8-44e6-b8e4-73ecb70403b3-metrics-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:39.836565 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:39.836552 5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4042310-83e8-44e6-b8e4-73ecb70403b3-maas-api-tls\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:39.836565 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:39.836564 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svkzw\" (UniqueName: \"kubernetes.io/projected/d4042310-83e8-44e6-b8e4-73ecb70403b3-kube-api-access-svkzw\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:39.836819 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:39.836574 5196 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4042310-83e8-44e6-b8e4-73ecb70403b3-openshift-service-ca\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:40.347910 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:40.344121 5196 generic.go:356] "Generic (PLEG): container finished" podID="d4042310-83e8-44e6-b8e4-73ecb70403b3" containerID="f31f9fafc261f77015412837af9f54e9b660ca7379393f4953e1fc7bc6e96bd0" exitCode=0 Nov 03 23:04:40.347910 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:40.344220 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr" event={"ID":"d4042310-83e8-44e6-b8e4-73ecb70403b3","Type":"ContainerDied","Data":"f31f9fafc261f77015412837af9f54e9b660ca7379393f4953e1fc7bc6e96bd0"} Nov 03 23:04:40.347910 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:40.344271 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr" event={"ID":"d4042310-83e8-44e6-b8e4-73ecb70403b3","Type":"ContainerDied","Data":"c79b365a0a319b9f3afd64098ffcbe694a9d2e11b8c0c549f92b2a898ee7ebbd"} Nov 03 23:04:40.347910 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:40.344301 5196 scope.go:122] "RemoveContainer" containerID="f31f9fafc261f77015412837af9f54e9b660ca7379393f4953e1fc7bc6e96bd0" Nov 03 23:04:40.347910 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:40.345612 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr" Nov 03 23:04:40.389905 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:40.388071 5196 scope.go:122] "RemoveContainer" containerID="f31f9fafc261f77015412837af9f54e9b660ca7379393f4953e1fc7bc6e96bd0" Nov 03 23:04:40.389905 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:04:40.388569 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f31f9fafc261f77015412837af9f54e9b660ca7379393f4953e1fc7bc6e96bd0\": container with ID starting with f31f9fafc261f77015412837af9f54e9b660ca7379393f4953e1fc7bc6e96bd0 not found: ID does not exist" containerID="f31f9fafc261f77015412837af9f54e9b660ca7379393f4953e1fc7bc6e96bd0" Nov 03 23:04:40.389905 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:40.388606 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f31f9fafc261f77015412837af9f54e9b660ca7379393f4953e1fc7bc6e96bd0"} err="failed to get container status \"f31f9fafc261f77015412837af9f54e9b660ca7379393f4953e1fc7bc6e96bd0\": rpc error: code = NotFound desc = could not find container \"f31f9fafc261f77015412837af9f54e9b660ca7379393f4953e1fc7bc6e96bd0\": container with ID starting with f31f9fafc261f77015412837af9f54e9b660ca7379393f4953e1fc7bc6e96bd0 not found: ID does not exist" Nov 03 23:04:40.515017 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:40.514979 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr"] Nov 03 23:04:40.567218 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:40.567183 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f8cd0c-6b9c7f89dd-8nkbr"] Nov 03 23:04:40.815479 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:40.815405 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4042310-83e8-44e6-b8e4-73ecb70403b3" path="/var/lib/kubelet/pods/d4042310-83e8-44e6-b8e4-73ecb70403b3/volumes" Nov 03 23:04:41.002679 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:41.002642 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:41.340667 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:41.340607 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:41.499370 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:41.499338 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:41.948138 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:41.948101 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:42.319978 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:42.319894 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:44.784158 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:44.784124 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:44.934123 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:44.934091 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:45.664213 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:45.664180 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl"] Nov 03 23:04:45.664438 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:45.664408 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl" podUID="8f7ce2e3-e5ee-45b8-99e6-985011216f8b" containerName="maas-api" containerID="cri-o://53cec113a2bf82ab6da87eee3e35e57187b114db27fd18f7c1ab7b3db51b7a99" gracePeriod=30 Nov 03 23:04:45.772309 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:45.772274 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:45.957127 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:45.957098 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl" Nov 03 23:04:46.021866 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:46.021821 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f7ce2e3-e5ee-45b8-99e6-985011216f8b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f7ce2e3-e5ee-45b8-99e6-985011216f8b-maas-api-tls\") pod \"8f7ce2e3-e5ee-45b8-99e6-985011216f8b\" (UID: \"8f7ce2e3-e5ee-45b8-99e6-985011216f8b\") " Nov 03 23:04:46.022026 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:46.021972 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f7ce2e3-e5ee-45b8-99e6-985011216f8b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f7ce2e3-e5ee-45b8-99e6-985011216f8b-metrics-certs\") pod \"8f7ce2e3-e5ee-45b8-99e6-985011216f8b\" (UID: \"8f7ce2e3-e5ee-45b8-99e6-985011216f8b\") " Nov 03 23:04:46.022098 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:46.022083 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f7ce2e3-e5ee-45b8-99e6-985011216f8b-kube-api-access-qbd46\" (UniqueName: \"kubernetes.io/projected/8f7ce2e3-e5ee-45b8-99e6-985011216f8b-kube-api-access-qbd46\") pod \"8f7ce2e3-e5ee-45b8-99e6-985011216f8b\" (UID: \"8f7ce2e3-e5ee-45b8-99e6-985011216f8b\") " Nov 03 23:04:46.022155 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:46.022139 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f7ce2e3-e5ee-45b8-99e6-985011216f8b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f7ce2e3-e5ee-45b8-99e6-985011216f8b-openshift-service-ca\") pod \"8f7ce2e3-e5ee-45b8-99e6-985011216f8b\" (UID: \"8f7ce2e3-e5ee-45b8-99e6-985011216f8b\") " Nov 03 23:04:46.022806 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:46.022775 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f7ce2e3-e5ee-45b8-99e6-985011216f8b-openshift-service-ca" pod "8f7ce2e3-e5ee-45b8-99e6-985011216f8b" (UID: "8f7ce2e3-e5ee-45b8-99e6-985011216f8b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:04:46.024337 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:46.024307 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f7ce2e3-e5ee-45b8-99e6-985011216f8b-maas-api-tls" pod "8f7ce2e3-e5ee-45b8-99e6-985011216f8b" (UID: "8f7ce2e3-e5ee-45b8-99e6-985011216f8b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:04:46.024760 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:46.024744 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f7ce2e3-e5ee-45b8-99e6-985011216f8b-metrics-certs" pod "8f7ce2e3-e5ee-45b8-99e6-985011216f8b" (UID: "8f7ce2e3-e5ee-45b8-99e6-985011216f8b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:04:46.024847 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:46.024787 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f7ce2e3-e5ee-45b8-99e6-985011216f8b-kube-api-access-qbd46" pod "8f7ce2e3-e5ee-45b8-99e6-985011216f8b" (UID: "8f7ce2e3-e5ee-45b8-99e6-985011216f8b"). InnerVolumeSpecName "kube-api-access-qbd46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:46.124028 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:46.123988 5196 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f7ce2e3-e5ee-45b8-99e6-985011216f8b-metrics-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:46.124028 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:46.124024 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbd46\" (UniqueName: \"kubernetes.io/projected/8f7ce2e3-e5ee-45b8-99e6-985011216f8b-kube-api-access-qbd46\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:46.124294 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:46.124040 5196 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f7ce2e3-e5ee-45b8-99e6-985011216f8b-openshift-service-ca\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:46.124294 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:46.124054 5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f7ce2e3-e5ee-45b8-99e6-985011216f8b-maas-api-tls\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:04:46.382332 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:46.382302 5196 generic.go:356] "Generic (PLEG): container finished" podID="8f7ce2e3-e5ee-45b8-99e6-985011216f8b" containerID="53cec113a2bf82ab6da87eee3e35e57187b114db27fd18f7c1ab7b3db51b7a99" exitCode=0 Nov 03 23:04:46.382504 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:46.382364 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl" Nov 03 23:04:46.382504 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:46.382385 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl" event={"ID":"8f7ce2e3-e5ee-45b8-99e6-985011216f8b","Type":"ContainerDied","Data":"53cec113a2bf82ab6da87eee3e35e57187b114db27fd18f7c1ab7b3db51b7a99"} Nov 03 23:04:46.382504 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:46.382441 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl" event={"ID":"8f7ce2e3-e5ee-45b8-99e6-985011216f8b","Type":"ContainerDied","Data":"b4c50aef55a605c1fda99c19919b914bbd1c38dcbfc2aff1c78e393b834af0d5"} Nov 03 23:04:46.382504 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:46.382473 5196 scope.go:122] "RemoveContainer" containerID="53cec113a2bf82ab6da87eee3e35e57187b114db27fd18f7c1ab7b3db51b7a99" Nov 03 23:04:46.393147 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:46.393123 5196 scope.go:122] "RemoveContainer" containerID="53cec113a2bf82ab6da87eee3e35e57187b114db27fd18f7c1ab7b3db51b7a99" Nov 03 23:04:46.393417 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:04:46.393399 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53cec113a2bf82ab6da87eee3e35e57187b114db27fd18f7c1ab7b3db51b7a99\": container with ID starting with 53cec113a2bf82ab6da87eee3e35e57187b114db27fd18f7c1ab7b3db51b7a99 not found: ID does not exist" containerID="53cec113a2bf82ab6da87eee3e35e57187b114db27fd18f7c1ab7b3db51b7a99" Nov 03 23:04:46.393472 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:46.393427 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53cec113a2bf82ab6da87eee3e35e57187b114db27fd18f7c1ab7b3db51b7a99"} err="failed to get container status \"53cec113a2bf82ab6da87eee3e35e57187b114db27fd18f7c1ab7b3db51b7a99\": rpc error: code = NotFound desc = could not find container \"53cec113a2bf82ab6da87eee3e35e57187b114db27fd18f7c1ab7b3db51b7a99\": container with ID starting with 53cec113a2bf82ab6da87eee3e35e57187b114db27fd18f7c1ab7b3db51b7a99 not found: ID does not exist" Nov 03 23:04:46.555894 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:46.554987 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl"] Nov 03 23:04:46.565910 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:46.565884 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-7afb1c-85c4ddc956-bn8xl"] Nov 03 23:04:46.815260 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:46.815181 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f7ce2e3-e5ee-45b8-99e6-985011216f8b" path="/var/lib/kubelet/pods/8f7ce2e3-e5ee-45b8-99e6-985011216f8b/volumes" Nov 03 23:04:47.389976 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:47.389940 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:47.598074 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:47.598037 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:47.745199 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:47.745121 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:48.340771 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:48.340712 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:48.619459 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:48.619357 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:48.811371 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:48.811341 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:48.972638 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:48.972565 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:52.738636 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:52.738595 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:54.553548 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:54.553516 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:54.700198 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:54.700169 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:54.887421 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:54.887387 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:04:54.892218 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:54.892189 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk"] Nov 03 23:04:54.892440 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:54.892420 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" podUID="9eec3a9d-0eea-44e8-92e4-2d19e4508148" containerName="istio-proxy" containerID="cri-o://294973e9950cb83e2c9be79bfc632e8119ca46c8982a4a8d67f8fbab70ddef71" gracePeriod=30 Nov 03 23:04:55.918469 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:04:55.918427 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:04:57.621535 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:04:57.621501 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:00.159837 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:00.159781 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" Nov 03 23:05:00.176040 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:00.176015 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eec3a9d-0eea-44e8-92e4-2d19e4508148-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eec3a9d-0eea-44e8-92e4-2d19e4508148-istio-envoy\") pod \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\" (UID: \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\") " Nov 03 23:05:00.176155 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:00.176054 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eec3a9d-0eea-44e8-92e4-2d19e4508148-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eec3a9d-0eea-44e8-92e4-2d19e4508148-credential-socket\") pod \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\" (UID: \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\") " Nov 03 23:05:00.176155 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:00.176078 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9eec3a9d-0eea-44e8-92e4-2d19e4508148-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eec3a9d-0eea-44e8-92e4-2d19e4508148-istio-podinfo\") pod \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\" (UID: \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\") " Nov 03 23:05:00.176325 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:00.176169 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eec3a9d-0eea-44e8-92e4-2d19e4508148-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eec3a9d-0eea-44e8-92e4-2d19e4508148-workload-certs\") pod \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\" (UID: \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\") " Nov 03 23:05:00.176325 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:00.176205 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eec3a9d-0eea-44e8-92e4-2d19e4508148-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eec3a9d-0eea-44e8-92e4-2d19e4508148-workload-socket\") pod \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\" (UID: \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\") " Nov 03 23:05:00.176325 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:00.176240 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eec3a9d-0eea-44e8-92e4-2d19e4508148-istio-token\" (UniqueName: \"kubernetes.io/projected/9eec3a9d-0eea-44e8-92e4-2d19e4508148-istio-token\") pod \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\" (UID: \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\") " Nov 03 23:05:00.176325 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:00.176260 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eec3a9d-0eea-44e8-92e4-2d19e4508148-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eec3a9d-0eea-44e8-92e4-2d19e4508148-istio-data\") pod \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\" (UID: \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\") " Nov 03 23:05:00.176325 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:00.176313 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eec3a9d-0eea-44e8-92e4-2d19e4508148-kube-api-access-cvxgd\" (UniqueName: \"kubernetes.io/projected/9eec3a9d-0eea-44e8-92e4-2d19e4508148-kube-api-access-cvxgd\") pod \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\" (UID: \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\") " Nov 03 23:05:00.176599 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:00.176332 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eec3a9d-0eea-44e8-92e4-2d19e4508148-credential-socket" pod "9eec3a9d-0eea-44e8-92e4-2d19e4508148" (UID: "9eec3a9d-0eea-44e8-92e4-2d19e4508148"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:00.176599 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:00.176406 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9eec3a9d-0eea-44e8-92e4-2d19e4508148-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eec3a9d-0eea-44e8-92e4-2d19e4508148-istiod-ca-cert\") pod \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\" (UID: \"9eec3a9d-0eea-44e8-92e4-2d19e4508148\") " Nov 03 23:05:00.176599 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:00.176448 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eec3a9d-0eea-44e8-92e4-2d19e4508148-workload-certs" pod "9eec3a9d-0eea-44e8-92e4-2d19e4508148" (UID: "9eec3a9d-0eea-44e8-92e4-2d19e4508148"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:00.176599 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:00.176480 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eec3a9d-0eea-44e8-92e4-2d19e4508148-workload-socket" pod "9eec3a9d-0eea-44e8-92e4-2d19e4508148" (UID: "9eec3a9d-0eea-44e8-92e4-2d19e4508148"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:00.176812 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:00.176665 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eec3a9d-0eea-44e8-92e4-2d19e4508148-istio-data" pod "9eec3a9d-0eea-44e8-92e4-2d19e4508148" (UID: "9eec3a9d-0eea-44e8-92e4-2d19e4508148"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:00.177065 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:00.177042 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eec3a9d-0eea-44e8-92e4-2d19e4508148-istiod-ca-cert" pod "9eec3a9d-0eea-44e8-92e4-2d19e4508148" (UID: "9eec3a9d-0eea-44e8-92e4-2d19e4508148"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:00.177152 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:00.177133 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eec3a9d-0eea-44e8-92e4-2d19e4508148-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:00.177200 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:00.177161 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eec3a9d-0eea-44e8-92e4-2d19e4508148-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:00.177200 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:00.177179 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eec3a9d-0eea-44e8-92e4-2d19e4508148-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:00.177200 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:00.177196 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eec3a9d-0eea-44e8-92e4-2d19e4508148-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:00.178383 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:00.178363 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eec3a9d-0eea-44e8-92e4-2d19e4508148-istio-envoy" pod "9eec3a9d-0eea-44e8-92e4-2d19e4508148" (UID: "9eec3a9d-0eea-44e8-92e4-2d19e4508148"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:00.178477 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:00.178394 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9eec3a9d-0eea-44e8-92e4-2d19e4508148-istio-podinfo" pod "9eec3a9d-0eea-44e8-92e4-2d19e4508148" (UID: "9eec3a9d-0eea-44e8-92e4-2d19e4508148"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:05:00.178999 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:00.178981 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eec3a9d-0eea-44e8-92e4-2d19e4508148-kube-api-access-cvxgd" pod "9eec3a9d-0eea-44e8-92e4-2d19e4508148" (UID: "9eec3a9d-0eea-44e8-92e4-2d19e4508148"). InnerVolumeSpecName "kube-api-access-cvxgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:00.178999 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:00.178989 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eec3a9d-0eea-44e8-92e4-2d19e4508148-istio-token" pod "9eec3a9d-0eea-44e8-92e4-2d19e4508148" (UID: "9eec3a9d-0eea-44e8-92e4-2d19e4508148"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:00.278577 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:00.278554 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eec3a9d-0eea-44e8-92e4-2d19e4508148-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:00.278577 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:00.278578 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvxgd\" (UniqueName: \"kubernetes.io/projected/9eec3a9d-0eea-44e8-92e4-2d19e4508148-kube-api-access-cvxgd\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:00.278734 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:00.278590 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eec3a9d-0eea-44e8-92e4-2d19e4508148-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:00.278734 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:00.278598 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eec3a9d-0eea-44e8-92e4-2d19e4508148-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:00.278734 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:00.278607 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eec3a9d-0eea-44e8-92e4-2d19e4508148-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:00.474394 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:00.474315 5196 generic.go:356] "Generic (PLEG): container finished" podID="9eec3a9d-0eea-44e8-92e4-2d19e4508148" containerID="294973e9950cb83e2c9be79bfc632e8119ca46c8982a4a8d67f8fbab70ddef71" exitCode=0 Nov 03 23:05:00.474394 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:00.474379 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" Nov 03 23:05:00.474569 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:00.474400 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" event={"ID":"9eec3a9d-0eea-44e8-92e4-2d19e4508148","Type":"ContainerDied","Data":"294973e9950cb83e2c9be79bfc632e8119ca46c8982a4a8d67f8fbab70ddef71"} Nov 03 23:05:00.474569 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:00.474476 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk" event={"ID":"9eec3a9d-0eea-44e8-92e4-2d19e4508148","Type":"ContainerDied","Data":"ac896323491c5e9e531c5907aca06866d019c40b34e9ec0f0648f6d5c142163e"} Nov 03 23:05:00.474569 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:00.474498 5196 scope.go:122] "RemoveContainer" containerID="294973e9950cb83e2c9be79bfc632e8119ca46c8982a4a8d67f8fbab70ddef71" Nov 03 23:05:00.485114 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:00.485097 5196 scope.go:122] "RemoveContainer" containerID="294973e9950cb83e2c9be79bfc632e8119ca46c8982a4a8d67f8fbab70ddef71" Nov 03 23:05:00.485427 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:05:00.485401 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"294973e9950cb83e2c9be79bfc632e8119ca46c8982a4a8d67f8fbab70ddef71\": container with ID starting with 294973e9950cb83e2c9be79bfc632e8119ca46c8982a4a8d67f8fbab70ddef71 not found: ID does not exist" containerID="294973e9950cb83e2c9be79bfc632e8119ca46c8982a4a8d67f8fbab70ddef71" Nov 03 23:05:00.485501 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:00.485439 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"294973e9950cb83e2c9be79bfc632e8119ca46c8982a4a8d67f8fbab70ddef71"} err="failed to get container status \"294973e9950cb83e2c9be79bfc632e8119ca46c8982a4a8d67f8fbab70ddef71\": rpc error: code = NotFound desc = could not find container \"294973e9950cb83e2c9be79bfc632e8119ca46c8982a4a8d67f8fbab70ddef71\": container with ID starting with 294973e9950cb83e2c9be79bfc632e8119ca46c8982a4a8d67f8fbab70ddef71 not found: ID does not exist" Nov 03 23:05:00.660619 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:00.660589 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk"] Nov 03 23:05:00.665910 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:00.665882 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7afb1c-openshift-default-6d7b749b87-5xrfk"] Nov 03 23:05:00.815769 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:00.815692 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eec3a9d-0eea-44e8-92e4-2d19e4508148" path="/var/lib/kubelet/pods/9eec3a9d-0eea-44e8-92e4-2d19e4508148/volumes" Nov 03 23:05:01.648663 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:01.648633 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:01.795475 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:01.795443 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:02.412113 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:02.412085 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:07.197558 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:07.197526 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:09.529960 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:09.529923 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:09.682784 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:09.682758 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:10.987555 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:10.987522 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:11.241463 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:11.241383 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp"] Nov 03 23:05:11.241623 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:11.241564 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp" podUID="47d17627-fece-4154-a00b-bfa9838574ca" containerName="maas-api" containerID="cri-o://58baef1b4d5b44c0810712da07bdb568f8f9f2b63a5be603c0a04c8e5fab8d61" gracePeriod=30 Nov 03 23:05:11.345292 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:11.345264 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:11.522690 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:11.522657 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp" Nov 03 23:05:11.552516 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:11.552473 5196 generic.go:356] "Generic (PLEG): container finished" podID="47d17627-fece-4154-a00b-bfa9838574ca" containerID="58baef1b4d5b44c0810712da07bdb568f8f9f2b63a5be603c0a04c8e5fab8d61" exitCode=0 Nov 03 23:05:11.552675 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:11.552512 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp" event={"ID":"47d17627-fece-4154-a00b-bfa9838574ca","Type":"ContainerDied","Data":"58baef1b4d5b44c0810712da07bdb568f8f9f2b63a5be603c0a04c8e5fab8d61"} Nov 03 23:05:11.552675 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:11.552567 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp" event={"ID":"47d17627-fece-4154-a00b-bfa9838574ca","Type":"ContainerDied","Data":"0a36e521c5ce43d364922234e9ad8b31e37f699e0d573a38f5cab8c8ee330365"} Nov 03 23:05:11.552675 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:11.552590 5196 scope.go:122] "RemoveContainer" containerID="58baef1b4d5b44c0810712da07bdb568f8f9f2b63a5be603c0a04c8e5fab8d61" Nov 03 23:05:11.552675 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:11.552526 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp" Nov 03 23:05:11.564265 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:11.564244 5196 scope.go:122] "RemoveContainer" containerID="58baef1b4d5b44c0810712da07bdb568f8f9f2b63a5be603c0a04c8e5fab8d61" Nov 03 23:05:11.564573 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:05:11.564548 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58baef1b4d5b44c0810712da07bdb568f8f9f2b63a5be603c0a04c8e5fab8d61\": container with ID starting with 58baef1b4d5b44c0810712da07bdb568f8f9f2b63a5be603c0a04c8e5fab8d61 not found: ID does not exist" containerID="58baef1b4d5b44c0810712da07bdb568f8f9f2b63a5be603c0a04c8e5fab8d61" Nov 03 23:05:11.564627 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:11.564586 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58baef1b4d5b44c0810712da07bdb568f8f9f2b63a5be603c0a04c8e5fab8d61"} err="failed to get container status \"58baef1b4d5b44c0810712da07bdb568f8f9f2b63a5be603c0a04c8e5fab8d61\": rpc error: code = NotFound desc = could not find container \"58baef1b4d5b44c0810712da07bdb568f8f9f2b63a5be603c0a04c8e5fab8d61\": container with ID starting with 58baef1b4d5b44c0810712da07bdb568f8f9f2b63a5be603c0a04c8e5fab8d61 not found: ID does not exist" Nov 03 23:05:11.609365 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:11.609334 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47d17627-fece-4154-a00b-bfa9838574ca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/47d17627-fece-4154-a00b-bfa9838574ca-metrics-certs\") pod \"47d17627-fece-4154-a00b-bfa9838574ca\" (UID: \"47d17627-fece-4154-a00b-bfa9838574ca\") " Nov 03 23:05:11.609365 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:11.609378 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47d17627-fece-4154-a00b-bfa9838574ca-kube-api-access-tljwg\" (UniqueName: \"kubernetes.io/projected/47d17627-fece-4154-a00b-bfa9838574ca-kube-api-access-tljwg\") pod \"47d17627-fece-4154-a00b-bfa9838574ca\" (UID: \"47d17627-fece-4154-a00b-bfa9838574ca\") " Nov 03 23:05:11.609618 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:11.609474 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47d17627-fece-4154-a00b-bfa9838574ca-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47d17627-fece-4154-a00b-bfa9838574ca-maas-api-tls\") pod \"47d17627-fece-4154-a00b-bfa9838574ca\" (UID: \"47d17627-fece-4154-a00b-bfa9838574ca\") " Nov 03 23:05:11.609691 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:11.609657 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47d17627-fece-4154-a00b-bfa9838574ca-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47d17627-fece-4154-a00b-bfa9838574ca-openshift-service-ca\") pod \"47d17627-fece-4154-a00b-bfa9838574ca\" (UID: \"47d17627-fece-4154-a00b-bfa9838574ca\") " Nov 03 23:05:11.610094 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:11.610064 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47d17627-fece-4154-a00b-bfa9838574ca-openshift-service-ca" pod "47d17627-fece-4154-a00b-bfa9838574ca" (UID: "47d17627-fece-4154-a00b-bfa9838574ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:11.610458 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:11.610433 5196 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47d17627-fece-4154-a00b-bfa9838574ca-openshift-service-ca\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:11.612165 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:11.612113 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47d17627-fece-4154-a00b-bfa9838574ca-maas-api-tls" pod "47d17627-fece-4154-a00b-bfa9838574ca" (UID: "47d17627-fece-4154-a00b-bfa9838574ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:05:11.612285 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:11.612257 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47d17627-fece-4154-a00b-bfa9838574ca-metrics-certs" pod "47d17627-fece-4154-a00b-bfa9838574ca" (UID: "47d17627-fece-4154-a00b-bfa9838574ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:05:11.613322 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:11.613297 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d17627-fece-4154-a00b-bfa9838574ca-kube-api-access-tljwg" pod "47d17627-fece-4154-a00b-bfa9838574ca" (UID: "47d17627-fece-4154-a00b-bfa9838574ca"). InnerVolumeSpecName "kube-api-access-tljwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:11.711870 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:11.711823 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tljwg\" (UniqueName: \"kubernetes.io/projected/47d17627-fece-4154-a00b-bfa9838574ca-kube-api-access-tljwg\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:11.711870 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:11.711849 5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47d17627-fece-4154-a00b-bfa9838574ca-maas-api-tls\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:11.712068 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:11.711888 5196 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47d17627-fece-4154-a00b-bfa9838574ca-metrics-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:11.974923 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:11.974894 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp"] Nov 03 23:05:11.979112 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:11.979082 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-924fba-768fd9f44d-q78wp"] Nov 03 23:05:12.815242 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:12.815204 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47d17627-fece-4154-a00b-bfa9838574ca" path="/var/lib/kubelet/pods/47d17627-fece-4154-a00b-bfa9838574ca/volumes" Nov 03 23:05:12.854711 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:12.854685 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:13.062658 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:13.062605 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:13.217101 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:13.217069 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:13.782128 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:13.782095 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:16.327571 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:16.327537 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:17.993267 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:17.993237 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:19.801139 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:19.801103 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:19.951155 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:19.951116 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:20.106208 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:20.106170 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v"] Nov 03 23:05:20.106518 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:20.106473 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" podUID="cef4b051-45b0-4db8-8ab5-afcfe3b9149d" containerName="istio-proxy" containerID="cri-o://0e4e30b30aad4a7021f4e93782143e182169bab3591086fdf720c750a1efacfd" gracePeriod=30 Nov 03 23:05:20.150711 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:20.150684 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:21.110822 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:21.110791 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:23.018155 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:23.018127 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:25.367593 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.367570 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 23:05:25.458332 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.458261 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-istio-token\" (UniqueName: \"kubernetes.io/projected/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-istio-token\") pod \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\" (UID: \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\") " Nov 03 23:05:25.458332 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.458309 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-istio-envoy\") pod \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\" (UID: \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\") " Nov 03 23:05:25.458521 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.458339 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-istiod-ca-cert\") pod \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\" (UID: \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\") " Nov 03 23:05:25.458521 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.458432 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-credential-socket\") pod \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\" (UID: \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\") " Nov 03 23:05:25.458521 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.458489 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-workload-socket\") pod \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\" (UID: \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\") " Nov 03 23:05:25.458669 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.458551 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-istio-podinfo\") pod \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\" (UID: \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\") " Nov 03 23:05:25.458669 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.458583 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-kube-api-access-nct7k\" (UniqueName: \"kubernetes.io/projected/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-kube-api-access-nct7k\") pod \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\" (UID: \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\") " Nov 03 23:05:25.458669 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.458620 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-istio-data\") pod \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\" (UID: \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\") " Nov 03 23:05:25.458830 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.458675 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-workload-certs\") pod \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\" (UID: \"cef4b051-45b0-4db8-8ab5-afcfe3b9149d\") " Nov 03 23:05:25.458830 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.458748 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-credential-socket" pod "cef4b051-45b0-4db8-8ab5-afcfe3b9149d" (UID: "cef4b051-45b0-4db8-8ab5-afcfe3b9149d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:25.458996 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.458953 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-workload-socket" pod "cef4b051-45b0-4db8-8ab5-afcfe3b9149d" (UID: "cef4b051-45b0-4db8-8ab5-afcfe3b9149d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:25.458996 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.458826 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-istiod-ca-cert" pod "cef4b051-45b0-4db8-8ab5-afcfe3b9149d" (UID: "cef4b051-45b0-4db8-8ab5-afcfe3b9149d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:25.459100 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.459038 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-istio-data" pod "cef4b051-45b0-4db8-8ab5-afcfe3b9149d" (UID: "cef4b051-45b0-4db8-8ab5-afcfe3b9149d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:25.459191 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.459164 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-workload-certs" pod "cef4b051-45b0-4db8-8ab5-afcfe3b9149d" (UID: "cef4b051-45b0-4db8-8ab5-afcfe3b9149d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:25.459630 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.459520 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:25.459630 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.459547 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:25.459630 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.459565 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:25.459630 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.459579 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:25.459630 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.459594 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:25.460696 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.460673 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-istio-token" pod "cef4b051-45b0-4db8-8ab5-afcfe3b9149d" (UID: "cef4b051-45b0-4db8-8ab5-afcfe3b9149d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:25.460909 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.460886 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-istio-podinfo" pod "cef4b051-45b0-4db8-8ab5-afcfe3b9149d" (UID: "cef4b051-45b0-4db8-8ab5-afcfe3b9149d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:05:25.461151 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.461134 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-istio-envoy" pod "cef4b051-45b0-4db8-8ab5-afcfe3b9149d" (UID: "cef4b051-45b0-4db8-8ab5-afcfe3b9149d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:25.461215 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.461133 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-kube-api-access-nct7k" pod "cef4b051-45b0-4db8-8ab5-afcfe3b9149d" (UID: "cef4b051-45b0-4db8-8ab5-afcfe3b9149d"). InnerVolumeSpecName "kube-api-access-nct7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:25.560663 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.560634 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:25.560663 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.560661 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nct7k\" (UniqueName: \"kubernetes.io/projected/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-kube-api-access-nct7k\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:25.560845 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.560673 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:25.560845 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.560682 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cef4b051-45b0-4db8-8ab5-afcfe3b9149d-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:25.639258 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.639228 5196 generic.go:356] "Generic (PLEG): container finished" podID="cef4b051-45b0-4db8-8ab5-afcfe3b9149d" containerID="0e4e30b30aad4a7021f4e93782143e182169bab3591086fdf720c750a1efacfd" exitCode=0 Nov 03 23:05:25.639429 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.639275 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" event={"ID":"cef4b051-45b0-4db8-8ab5-afcfe3b9149d","Type":"ContainerDied","Data":"0e4e30b30aad4a7021f4e93782143e182169bab3591086fdf720c750a1efacfd"} Nov 03 23:05:25.639429 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.639334 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" event={"ID":"cef4b051-45b0-4db8-8ab5-afcfe3b9149d","Type":"ContainerDied","Data":"874af0e53973cd7adeb2bd912e133318560c9b0e8e661e9904e21053410981cb"} Nov 03 23:05:25.639429 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.639355 5196 scope.go:122] "RemoveContainer" containerID="0e4e30b30aad4a7021f4e93782143e182169bab3591086fdf720c750a1efacfd" Nov 03 23:05:25.639429 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.639283 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v" Nov 03 23:05:25.650173 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.650156 5196 scope.go:122] "RemoveContainer" containerID="154f65d83db979208f117a3eda40f081f98a9631b46bee2a47df32867f5c6966" Nov 03 23:05:25.668031 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.662015 5196 scope.go:122] "RemoveContainer" containerID="0e4e30b30aad4a7021f4e93782143e182169bab3591086fdf720c750a1efacfd" Nov 03 23:05:25.668031 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:05:25.664168 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e4e30b30aad4a7021f4e93782143e182169bab3591086fdf720c750a1efacfd\": container with ID starting with 0e4e30b30aad4a7021f4e93782143e182169bab3591086fdf720c750a1efacfd not found: ID does not exist" containerID="0e4e30b30aad4a7021f4e93782143e182169bab3591086fdf720c750a1efacfd" Nov 03 23:05:25.668031 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.664208 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e4e30b30aad4a7021f4e93782143e182169bab3591086fdf720c750a1efacfd"} err="failed to get container status \"0e4e30b30aad4a7021f4e93782143e182169bab3591086fdf720c750a1efacfd\": rpc error: code = NotFound desc = could not find container \"0e4e30b30aad4a7021f4e93782143e182169bab3591086fdf720c750a1efacfd\": container with ID starting with 0e4e30b30aad4a7021f4e93782143e182169bab3591086fdf720c750a1efacfd not found: ID does not exist" Nov 03 23:05:25.668031 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.664237 5196 scope.go:122] "RemoveContainer" containerID="154f65d83db979208f117a3eda40f081f98a9631b46bee2a47df32867f5c6966" Nov 03 23:05:25.668350 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:05:25.668126 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"154f65d83db979208f117a3eda40f081f98a9631b46bee2a47df32867f5c6966\": container with ID starting with 154f65d83db979208f117a3eda40f081f98a9631b46bee2a47df32867f5c6966 not found: ID does not exist" containerID="154f65d83db979208f117a3eda40f081f98a9631b46bee2a47df32867f5c6966" Nov 03 23:05:25.668350 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.668163 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"154f65d83db979208f117a3eda40f081f98a9631b46bee2a47df32867f5c6966"} err="failed to get container status \"154f65d83db979208f117a3eda40f081f98a9631b46bee2a47df32867f5c6966\": rpc error: code = NotFound desc = could not find container \"154f65d83db979208f117a3eda40f081f98a9631b46bee2a47df32867f5c6966\": container with ID starting with 154f65d83db979208f117a3eda40f081f98a9631b46bee2a47df32867f5c6966 not found: ID does not exist" Nov 03 23:05:25.807240 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.807166 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v"] Nov 03 23:05:25.813054 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:25.813027 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-924fba-openshift-default-dd49687c5-phg8v"] Nov 03 23:05:25.918997 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:05:25.918961 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:05:26.814419 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:26.814386 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cef4b051-45b0-4db8-8ab5-afcfe3b9149d" path="/var/lib/kubelet/pods/cef4b051-45b0-4db8-8ab5-afcfe3b9149d/volumes" Nov 03 23:05:28.057250 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:28.057210 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:32.382850 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:32.382808 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:33.897308 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:33.897269 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:34.047007 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:34.046975 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:34.194604 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:34.194528 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:34.343015 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:34.342982 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:34.799006 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:34.798975 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:36.383849 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:36.383816 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:36.530285 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:36.530256 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:38.599686 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:38.599654 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s"] Nov 03 23:05:38.600102 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:38.599877 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s" podUID="41e5ef76-ed91-4c59-a3e5-36a288fbc075" containerName="maas-api" containerID="cri-o://d28feca9800963b68e65bc185de8fb38358d45436094b77d00af0679f2181800" gracePeriod=30 Nov 03 23:05:38.633517 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:38.633490 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s" podUID="41e5ef76-ed91-4c59-a3e5-36a288fbc075" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.124:8443/health\": dial tcp 10.133.0.124:8443: connect: connection refused" Nov 03 23:05:38.732462 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:38.732421 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:38.737976 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:38.737952 5196 generic.go:356] "Generic (PLEG): container finished" podID="41e5ef76-ed91-4c59-a3e5-36a288fbc075" containerID="d28feca9800963b68e65bc185de8fb38358d45436094b77d00af0679f2181800" exitCode=0 Nov 03 23:05:38.738088 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:38.738029 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s" event={"ID":"41e5ef76-ed91-4c59-a3e5-36a288fbc075","Type":"ContainerDied","Data":"d28feca9800963b68e65bc185de8fb38358d45436094b77d00af0679f2181800"} Nov 03 23:05:38.890744 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:38.890710 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s" Nov 03 23:05:38.928025 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:38.927994 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41e5ef76-ed91-4c59-a3e5-36a288fbc075-metrics-certs\" (UniqueName: \"kubernetes.io/secret/41e5ef76-ed91-4c59-a3e5-36a288fbc075-metrics-certs\") pod \"41e5ef76-ed91-4c59-a3e5-36a288fbc075\" (UID: \"41e5ef76-ed91-4c59-a3e5-36a288fbc075\") " Nov 03 23:05:38.928189 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:38.928145 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41e5ef76-ed91-4c59-a3e5-36a288fbc075-kube-api-access-p24lq\" (UniqueName: \"kubernetes.io/projected/41e5ef76-ed91-4c59-a3e5-36a288fbc075-kube-api-access-p24lq\") pod \"41e5ef76-ed91-4c59-a3e5-36a288fbc075\" (UID: \"41e5ef76-ed91-4c59-a3e5-36a288fbc075\") " Nov 03 23:05:38.928243 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:38.928216 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41e5ef76-ed91-4c59-a3e5-36a288fbc075-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e5ef76-ed91-4c59-a3e5-36a288fbc075-openshift-service-ca\") pod \"41e5ef76-ed91-4c59-a3e5-36a288fbc075\" (UID: \"41e5ef76-ed91-4c59-a3e5-36a288fbc075\") " Nov 03 23:05:38.928319 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:38.928302 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41e5ef76-ed91-4c59-a3e5-36a288fbc075-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41e5ef76-ed91-4c59-a3e5-36a288fbc075-maas-api-tls\") pod \"41e5ef76-ed91-4c59-a3e5-36a288fbc075\" (UID: \"41e5ef76-ed91-4c59-a3e5-36a288fbc075\") " Nov 03 23:05:38.928637 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:38.928605 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41e5ef76-ed91-4c59-a3e5-36a288fbc075-openshift-service-ca" pod "41e5ef76-ed91-4c59-a3e5-36a288fbc075" (UID: "41e5ef76-ed91-4c59-a3e5-36a288fbc075"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:38.930638 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:38.930589 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41e5ef76-ed91-4c59-a3e5-36a288fbc075-kube-api-access-p24lq" pod "41e5ef76-ed91-4c59-a3e5-36a288fbc075" (UID: "41e5ef76-ed91-4c59-a3e5-36a288fbc075"). InnerVolumeSpecName "kube-api-access-p24lq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:38.930937 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:38.930916 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41e5ef76-ed91-4c59-a3e5-36a288fbc075-metrics-certs" pod "41e5ef76-ed91-4c59-a3e5-36a288fbc075" (UID: "41e5ef76-ed91-4c59-a3e5-36a288fbc075"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:05:38.931032 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:38.930949 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41e5ef76-ed91-4c59-a3e5-36a288fbc075-maas-api-tls" pod "41e5ef76-ed91-4c59-a3e5-36a288fbc075" (UID: "41e5ef76-ed91-4c59-a3e5-36a288fbc075"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:05:39.029763 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:39.029725 5196 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e5ef76-ed91-4c59-a3e5-36a288fbc075-openshift-service-ca\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:39.029763 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:39.029765 5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41e5ef76-ed91-4c59-a3e5-36a288fbc075-maas-api-tls\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:39.030033 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:39.029782 5196 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41e5ef76-ed91-4c59-a3e5-36a288fbc075-metrics-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:39.030033 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:39.029798 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p24lq\" (UniqueName: \"kubernetes.io/projected/41e5ef76-ed91-4c59-a3e5-36a288fbc075-kube-api-access-p24lq\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:39.746097 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:39.746048 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s" event={"ID":"41e5ef76-ed91-4c59-a3e5-36a288fbc075","Type":"ContainerDied","Data":"1d63df83cb701fad6c63dc871562c06843bfe7c768b900aa07cf439ca5611f36"} Nov 03 23:05:39.746593 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:39.746132 5196 scope.go:122] "RemoveContainer" containerID="d28feca9800963b68e65bc185de8fb38358d45436094b77d00af0679f2181800" Nov 03 23:05:39.746593 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:39.746067 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s" Nov 03 23:05:39.913435 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:39.913397 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s"] Nov 03 23:05:39.963915 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:39.963870 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e66d59-5685f55ffb-q2f6s"] Nov 03 23:05:40.330605 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:40.330572 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:40.604185 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:40.604153 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:40.749654 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:40.749624 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:40.814130 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:40.814102 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41e5ef76-ed91-4c59-a3e5-36a288fbc075" path="/var/lib/kubelet/pods/41e5ef76-ed91-4c59-a3e5-36a288fbc075/volumes" Nov 03 23:05:41.001978 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:41.001898 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:41.159042 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:41.159009 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:41.310185 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:41.310111 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:41.327968 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:41.327929 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg"] Nov 03 23:05:41.328158 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:41.328136 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" podUID="b8d72750-08ae-4995-ac23-6304ada9b53f" containerName="istio-proxy" containerID="cri-o://0145bcb8d4e420d9fc45e443abe0fa5436144a62a0135c5dfcb7c81ef54ffd69" gracePeriod=30 Nov 03 23:05:41.474461 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:41.474433 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:45.701117 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:45.701079 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:46.578700 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:46.578675 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" Nov 03 23:05:46.713939 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:46.713828 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8d72750-08ae-4995-ac23-6304ada9b53f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8d72750-08ae-4995-ac23-6304ada9b53f-workload-certs\") pod \"b8d72750-08ae-4995-ac23-6304ada9b53f\" (UID: \"b8d72750-08ae-4995-ac23-6304ada9b53f\") " Nov 03 23:05:46.713939 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:46.713899 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8d72750-08ae-4995-ac23-6304ada9b53f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8d72750-08ae-4995-ac23-6304ada9b53f-istio-envoy\") pod \"b8d72750-08ae-4995-ac23-6304ada9b53f\" (UID: \"b8d72750-08ae-4995-ac23-6304ada9b53f\") " Nov 03 23:05:46.713939 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:46.713935 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8d72750-08ae-4995-ac23-6304ada9b53f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8d72750-08ae-4995-ac23-6304ada9b53f-istio-data\") pod \"b8d72750-08ae-4995-ac23-6304ada9b53f\" (UID: \"b8d72750-08ae-4995-ac23-6304ada9b53f\") " Nov 03 23:05:46.714511 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:46.713991 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8d72750-08ae-4995-ac23-6304ada9b53f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8d72750-08ae-4995-ac23-6304ada9b53f-istiod-ca-cert\") pod \"b8d72750-08ae-4995-ac23-6304ada9b53f\" (UID: \"b8d72750-08ae-4995-ac23-6304ada9b53f\") " Nov 03 23:05:46.714511 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:46.714046 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8d72750-08ae-4995-ac23-6304ada9b53f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8d72750-08ae-4995-ac23-6304ada9b53f-istio-podinfo\") pod \"b8d72750-08ae-4995-ac23-6304ada9b53f\" (UID: \"b8d72750-08ae-4995-ac23-6304ada9b53f\") " Nov 03 23:05:46.714511 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:46.714081 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8d72750-08ae-4995-ac23-6304ada9b53f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d72750-08ae-4995-ac23-6304ada9b53f-workload-socket\") pod \"b8d72750-08ae-4995-ac23-6304ada9b53f\" (UID: \"b8d72750-08ae-4995-ac23-6304ada9b53f\") " Nov 03 23:05:46.714511 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:46.714101 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d72750-08ae-4995-ac23-6304ada9b53f-workload-certs" pod "b8d72750-08ae-4995-ac23-6304ada9b53f" (UID: "b8d72750-08ae-4995-ac23-6304ada9b53f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:46.714511 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:46.714186 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8d72750-08ae-4995-ac23-6304ada9b53f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d72750-08ae-4995-ac23-6304ada9b53f-credential-socket\") pod \"b8d72750-08ae-4995-ac23-6304ada9b53f\" (UID: \"b8d72750-08ae-4995-ac23-6304ada9b53f\") " Nov 03 23:05:46.714511 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:46.714220 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8d72750-08ae-4995-ac23-6304ada9b53f-istio-token\" (UniqueName: \"kubernetes.io/projected/b8d72750-08ae-4995-ac23-6304ada9b53f-istio-token\") pod \"b8d72750-08ae-4995-ac23-6304ada9b53f\" (UID: \"b8d72750-08ae-4995-ac23-6304ada9b53f\") " Nov 03 23:05:46.714511 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:46.714234 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d72750-08ae-4995-ac23-6304ada9b53f-istio-data" pod "b8d72750-08ae-4995-ac23-6304ada9b53f" (UID: "b8d72750-08ae-4995-ac23-6304ada9b53f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:46.714511 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:46.714292 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8d72750-08ae-4995-ac23-6304ada9b53f-kube-api-access-tcsl7\" (UniqueName: \"kubernetes.io/projected/b8d72750-08ae-4995-ac23-6304ada9b53f-kube-api-access-tcsl7\") pod \"b8d72750-08ae-4995-ac23-6304ada9b53f\" (UID: \"b8d72750-08ae-4995-ac23-6304ada9b53f\") " Nov 03 23:05:46.714511 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:46.714389 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d72750-08ae-4995-ac23-6304ada9b53f-workload-socket" pod "b8d72750-08ae-4995-ac23-6304ada9b53f" (UID: "b8d72750-08ae-4995-ac23-6304ada9b53f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:46.714511 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:46.714403 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d72750-08ae-4995-ac23-6304ada9b53f-credential-socket" pod "b8d72750-08ae-4995-ac23-6304ada9b53f" (UID: "b8d72750-08ae-4995-ac23-6304ada9b53f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:46.714511 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:46.714423 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8d72750-08ae-4995-ac23-6304ada9b53f-istiod-ca-cert" pod "b8d72750-08ae-4995-ac23-6304ada9b53f" (UID: "b8d72750-08ae-4995-ac23-6304ada9b53f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:46.715074 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:46.714725 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8d72750-08ae-4995-ac23-6304ada9b53f-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:46.715074 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:46.714747 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d72750-08ae-4995-ac23-6304ada9b53f-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:46.715074 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:46.714765 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d72750-08ae-4995-ac23-6304ada9b53f-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:46.715074 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:46.714779 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8d72750-08ae-4995-ac23-6304ada9b53f-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:46.715074 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:46.714794 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8d72750-08ae-4995-ac23-6304ada9b53f-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:46.716403 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:46.716373 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8d72750-08ae-4995-ac23-6304ada9b53f-istio-podinfo" pod "b8d72750-08ae-4995-ac23-6304ada9b53f" (UID: "b8d72750-08ae-4995-ac23-6304ada9b53f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:05:46.716542 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:46.716483 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d72750-08ae-4995-ac23-6304ada9b53f-istio-token" pod "b8d72750-08ae-4995-ac23-6304ada9b53f" (UID: "b8d72750-08ae-4995-ac23-6304ada9b53f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:46.716725 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:46.716701 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d72750-08ae-4995-ac23-6304ada9b53f-istio-envoy" pod "b8d72750-08ae-4995-ac23-6304ada9b53f" (UID: "b8d72750-08ae-4995-ac23-6304ada9b53f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:46.716835 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:46.716786 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d72750-08ae-4995-ac23-6304ada9b53f-kube-api-access-tcsl7" pod "b8d72750-08ae-4995-ac23-6304ada9b53f" (UID: "b8d72750-08ae-4995-ac23-6304ada9b53f"). InnerVolumeSpecName "kube-api-access-tcsl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:46.791324 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:46.791292 5196 generic.go:356] "Generic (PLEG): container finished" podID="b8d72750-08ae-4995-ac23-6304ada9b53f" containerID="0145bcb8d4e420d9fc45e443abe0fa5436144a62a0135c5dfcb7c81ef54ffd69" exitCode=0 Nov 03 23:05:46.791489 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:46.791353 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" Nov 03 23:05:46.791489 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:46.791374 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" event={"ID":"b8d72750-08ae-4995-ac23-6304ada9b53f","Type":"ContainerDied","Data":"0145bcb8d4e420d9fc45e443abe0fa5436144a62a0135c5dfcb7c81ef54ffd69"} Nov 03 23:05:46.791489 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:46.791430 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg" event={"ID":"b8d72750-08ae-4995-ac23-6304ada9b53f","Type":"ContainerDied","Data":"3ede1c71069cd497aef25919fe0d3b6baa2385f9fc0ef669448ccc9ea3ab27b2"} Nov 03 23:05:46.791489 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:46.791451 5196 scope.go:122] "RemoveContainer" containerID="0145bcb8d4e420d9fc45e443abe0fa5436144a62a0135c5dfcb7c81ef54ffd69" Nov 03 23:05:46.801800 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:46.801784 5196 scope.go:122] "RemoveContainer" containerID="0145bcb8d4e420d9fc45e443abe0fa5436144a62a0135c5dfcb7c81ef54ffd69" Nov 03 23:05:46.802130 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:05:46.802111 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0145bcb8d4e420d9fc45e443abe0fa5436144a62a0135c5dfcb7c81ef54ffd69\": container with ID starting with 0145bcb8d4e420d9fc45e443abe0fa5436144a62a0135c5dfcb7c81ef54ffd69 not found: ID does not exist" containerID="0145bcb8d4e420d9fc45e443abe0fa5436144a62a0135c5dfcb7c81ef54ffd69" Nov 03 23:05:46.802223 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:46.802144 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0145bcb8d4e420d9fc45e443abe0fa5436144a62a0135c5dfcb7c81ef54ffd69"} err="failed to get container status \"0145bcb8d4e420d9fc45e443abe0fa5436144a62a0135c5dfcb7c81ef54ffd69\": rpc error: code = NotFound desc = could not find container \"0145bcb8d4e420d9fc45e443abe0fa5436144a62a0135c5dfcb7c81ef54ffd69\": container with ID starting with 0145bcb8d4e420d9fc45e443abe0fa5436144a62a0135c5dfcb7c81ef54ffd69 not found: ID does not exist" Nov 03 23:05:46.815240 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:46.815217 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8d72750-08ae-4995-ac23-6304ada9b53f-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:46.815240 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:46.815238 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8d72750-08ae-4995-ac23-6304ada9b53f-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:46.815370 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:46.815249 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8d72750-08ae-4995-ac23-6304ada9b53f-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:46.815370 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:46.815258 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcsl7\" (UniqueName: \"kubernetes.io/projected/b8d72750-08ae-4995-ac23-6304ada9b53f-kube-api-access-tcsl7\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:46.958706 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:46.958673 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg"] Nov 03 23:05:46.963520 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:46.963493 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b64e2e-openshift-default-64c944c45c-h59rg"] Nov 03 23:05:47.419831 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:47.419798 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:47.566611 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:47.566577 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:47.717188 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:47.717105 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4"] Nov 03 23:05:47.717976 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:47.717943 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" podUID="59200876-e179-4170-832d-d85cba111281" containerName="istio-proxy" containerID="cri-o://f9b6ad04597760bd9788d8c14229e49341b2467d81c993c53bd42fb1ddf7bb7d" gracePeriod=30 Nov 03 23:05:47.747473 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:47.747447 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:48.283512 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:48.283475 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:48.814269 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:48.814235 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8d72750-08ae-4995-ac23-6304ada9b53f" path="/var/lib/kubelet/pods/b8d72750-08ae-4995-ac23-6304ada9b53f/volumes" Nov 03 23:05:50.745039 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:50.745007 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:51.036752 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:51.036666 5196 patch_prober.go:29] interesting pod/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 23:05:51.036752 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:51.036732 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" podUID="59200876-e179-4170-832d-d85cba111281" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 23:05:52.976535 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:52.976511 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 23:05:53.083578 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:53.083501 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59200876-e179-4170-832d-d85cba111281-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59200876-e179-4170-832d-d85cba111281-credential-socket\") pod \"59200876-e179-4170-832d-d85cba111281\" (UID: \"59200876-e179-4170-832d-d85cba111281\") " Nov 03 23:05:53.083578 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:53.083551 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59200876-e179-4170-832d-d85cba111281-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59200876-e179-4170-832d-d85cba111281-istiod-ca-cert\") pod \"59200876-e179-4170-832d-d85cba111281\" (UID: \"59200876-e179-4170-832d-d85cba111281\") " Nov 03 23:05:53.083809 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:53.083582 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59200876-e179-4170-832d-d85cba111281-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59200876-e179-4170-832d-d85cba111281-workload-socket\") pod \"59200876-e179-4170-832d-d85cba111281\" (UID: \"59200876-e179-4170-832d-d85cba111281\") " Nov 03 23:05:53.083809 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:53.083599 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59200876-e179-4170-832d-d85cba111281-istio-token\" (UniqueName: \"kubernetes.io/projected/59200876-e179-4170-832d-d85cba111281-istio-token\") pod \"59200876-e179-4170-832d-d85cba111281\" (UID: \"59200876-e179-4170-832d-d85cba111281\") " Nov 03 23:05:53.083809 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:53.083654 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59200876-e179-4170-832d-d85cba111281-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59200876-e179-4170-832d-d85cba111281-istio-data\") pod \"59200876-e179-4170-832d-d85cba111281\" (UID: \"59200876-e179-4170-832d-d85cba111281\") " Nov 03 23:05:53.083809 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:53.083702 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59200876-e179-4170-832d-d85cba111281-kube-api-access-tzgxs\" (UniqueName: \"kubernetes.io/projected/59200876-e179-4170-832d-d85cba111281-kube-api-access-tzgxs\") pod \"59200876-e179-4170-832d-d85cba111281\" (UID: \"59200876-e179-4170-832d-d85cba111281\") " Nov 03 23:05:53.083809 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:53.083726 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59200876-e179-4170-832d-d85cba111281-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59200876-e179-4170-832d-d85cba111281-istio-envoy\") pod \"59200876-e179-4170-832d-d85cba111281\" (UID: \"59200876-e179-4170-832d-d85cba111281\") " Nov 03 23:05:53.083809 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:53.083753 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59200876-e179-4170-832d-d85cba111281-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59200876-e179-4170-832d-d85cba111281-istio-podinfo\") pod \"59200876-e179-4170-832d-d85cba111281\" (UID: \"59200876-e179-4170-832d-d85cba111281\") " Nov 03 23:05:53.084168 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:53.083827 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59200876-e179-4170-832d-d85cba111281-credential-socket" pod "59200876-e179-4170-832d-d85cba111281" (UID: "59200876-e179-4170-832d-d85cba111281"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:53.084168 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:53.083991 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59200876-e179-4170-832d-d85cba111281-workload-socket" pod "59200876-e179-4170-832d-d85cba111281" (UID: "59200876-e179-4170-832d-d85cba111281"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:53.084168 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:53.084012 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59200876-e179-4170-832d-d85cba111281-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59200876-e179-4170-832d-d85cba111281-workload-certs\") pod \"59200876-e179-4170-832d-d85cba111281\" (UID: \"59200876-e179-4170-832d-d85cba111281\") " Nov 03 23:05:53.084168 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:53.084085 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59200876-e179-4170-832d-d85cba111281-istio-data" pod "59200876-e179-4170-832d-d85cba111281" (UID: "59200876-e179-4170-832d-d85cba111281"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:53.084408 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:53.084176 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59200876-e179-4170-832d-d85cba111281-istiod-ca-cert" pod "59200876-e179-4170-832d-d85cba111281" (UID: "59200876-e179-4170-832d-d85cba111281"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:53.084408 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:53.084190 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59200876-e179-4170-832d-d85cba111281-workload-certs" pod "59200876-e179-4170-832d-d85cba111281" (UID: "59200876-e179-4170-832d-d85cba111281"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:53.084807 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:53.084742 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59200876-e179-4170-832d-d85cba111281-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:53.084807 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:53.084776 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59200876-e179-4170-832d-d85cba111281-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:53.084807 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:53.084793 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59200876-e179-4170-832d-d85cba111281-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:53.084807 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:53.084808 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59200876-e179-4170-832d-d85cba111281-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:53.085098 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:53.084823 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59200876-e179-4170-832d-d85cba111281-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:53.086108 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:53.086075 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59200876-e179-4170-832d-d85cba111281-istio-envoy" pod "59200876-e179-4170-832d-d85cba111281" (UID: "59200876-e179-4170-832d-d85cba111281"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:53.086108 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:53.086102 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59200876-e179-4170-832d-d85cba111281-kube-api-access-tzgxs" pod "59200876-e179-4170-832d-d85cba111281" (UID: "59200876-e179-4170-832d-d85cba111281"). InnerVolumeSpecName "kube-api-access-tzgxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:53.086284 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:53.086129 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59200876-e179-4170-832d-d85cba111281-istio-token" pod "59200876-e179-4170-832d-d85cba111281" (UID: "59200876-e179-4170-832d-d85cba111281"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:53.086371 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:53.086355 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59200876-e179-4170-832d-d85cba111281-istio-podinfo" pod "59200876-e179-4170-832d-d85cba111281" (UID: "59200876-e179-4170-832d-d85cba111281"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:05:53.185792 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:53.185762 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59200876-e179-4170-832d-d85cba111281-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:53.185792 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:53.185790 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzgxs\" (UniqueName: \"kubernetes.io/projected/59200876-e179-4170-832d-d85cba111281-kube-api-access-tzgxs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:53.185993 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:53.185805 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59200876-e179-4170-832d-d85cba111281-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:53.185993 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:53.185817 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59200876-e179-4170-832d-d85cba111281-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:53.336594 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:53.336508 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:53.835287 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:53.835246 5196 generic.go:356] "Generic (PLEG): container finished" podID="59200876-e179-4170-832d-d85cba111281" containerID="f9b6ad04597760bd9788d8c14229e49341b2467d81c993c53bd42fb1ddf7bb7d" exitCode=0 Nov 03 23:05:53.835463 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:53.835298 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" Nov 03 23:05:53.835463 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:53.835326 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" event={"ID":"59200876-e179-4170-832d-d85cba111281","Type":"ContainerDied","Data":"f9b6ad04597760bd9788d8c14229e49341b2467d81c993c53bd42fb1ddf7bb7d"} Nov 03 23:05:53.835463 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:53.835384 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4" event={"ID":"59200876-e179-4170-832d-d85cba111281","Type":"ContainerDied","Data":"55baed265dc0d69df9d7fc5e107098d99b00682b7571d2d3ee44dd3ee4ba2aa7"} Nov 03 23:05:53.835463 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:53.835409 5196 scope.go:122] "RemoveContainer" containerID="f9b6ad04597760bd9788d8c14229e49341b2467d81c993c53bd42fb1ddf7bb7d" Nov 03 23:05:53.846432 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:53.846411 5196 scope.go:122] "RemoveContainer" containerID="159fb8a18f6a38943292cbe931ada2d53535627cd6be0c70cda2d95bb733c204" Nov 03 23:05:53.855613 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:53.855594 5196 scope.go:122] "RemoveContainer" containerID="f9b6ad04597760bd9788d8c14229e49341b2467d81c993c53bd42fb1ddf7bb7d" Nov 03 23:05:53.855907 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:05:53.855884 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9b6ad04597760bd9788d8c14229e49341b2467d81c993c53bd42fb1ddf7bb7d\": container with ID starting with f9b6ad04597760bd9788d8c14229e49341b2467d81c993c53bd42fb1ddf7bb7d not found: ID does not exist" containerID="f9b6ad04597760bd9788d8c14229e49341b2467d81c993c53bd42fb1ddf7bb7d" Nov 03 23:05:53.855986 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:53.855917 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9b6ad04597760bd9788d8c14229e49341b2467d81c993c53bd42fb1ddf7bb7d"} err="failed to get container status \"f9b6ad04597760bd9788d8c14229e49341b2467d81c993c53bd42fb1ddf7bb7d\": rpc error: code = NotFound desc = could not find container \"f9b6ad04597760bd9788d8c14229e49341b2467d81c993c53bd42fb1ddf7bb7d\": container with ID starting with f9b6ad04597760bd9788d8c14229e49341b2467d81c993c53bd42fb1ddf7bb7d not found: ID does not exist" Nov 03 23:05:53.855986 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:53.855937 5196 scope.go:122] "RemoveContainer" containerID="159fb8a18f6a38943292cbe931ada2d53535627cd6be0c70cda2d95bb733c204" Nov 03 23:05:53.856188 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:05:53.856170 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"159fb8a18f6a38943292cbe931ada2d53535627cd6be0c70cda2d95bb733c204\": container with ID starting with 159fb8a18f6a38943292cbe931ada2d53535627cd6be0c70cda2d95bb733c204 not found: ID does not exist" containerID="159fb8a18f6a38943292cbe931ada2d53535627cd6be0c70cda2d95bb733c204" Nov 03 23:05:53.856244 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:53.856198 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"159fb8a18f6a38943292cbe931ada2d53535627cd6be0c70cda2d95bb733c204"} err="failed to get container status \"159fb8a18f6a38943292cbe931ada2d53535627cd6be0c70cda2d95bb733c204\": rpc error: code = NotFound desc = could not find container \"159fb8a18f6a38943292cbe931ada2d53535627cd6be0c70cda2d95bb733c204\": container with ID starting with 159fb8a18f6a38943292cbe931ada2d53535627cd6be0c70cda2d95bb733c204 not found: ID does not exist" Nov 03 23:05:54.055494 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:54.055459 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4"] Nov 03 23:05:54.105554 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:54.105521 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e66d59-openshift-default-6b654555-8jvm4"] Nov 03 23:05:54.814055 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:54.814025 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59200876-e179-4170-832d-d85cba111281" path="/var/lib/kubelet/pods/59200876-e179-4170-832d-d85cba111281/volumes" Nov 03 23:05:55.066110 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:55.066035 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5"] Nov 03 23:05:55.066485 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:55.066243 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5" podUID="7914cbce-00fd-4d6b-84c9-4519c50af650" containerName="maas-api" containerID="cri-o://7d0bdc6396e6d713d6b27723c120a88fff35255a4d85edb98eadf14947921e6e" gracePeriod=30 Nov 03 23:05:55.235224 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:55.235182 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:55.342063 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:55.342039 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5" Nov 03 23:05:55.410412 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:55.410378 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7914cbce-00fd-4d6b-84c9-4519c50af650-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7914cbce-00fd-4d6b-84c9-4519c50af650-maas-api-tls\") pod \"7914cbce-00fd-4d6b-84c9-4519c50af650\" (UID: \"7914cbce-00fd-4d6b-84c9-4519c50af650\") " Nov 03 23:05:55.410607 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:55.410432 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7914cbce-00fd-4d6b-84c9-4519c50af650-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7914cbce-00fd-4d6b-84c9-4519c50af650-metrics-certs\") pod \"7914cbce-00fd-4d6b-84c9-4519c50af650\" (UID: \"7914cbce-00fd-4d6b-84c9-4519c50af650\") " Nov 03 23:05:55.410607 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:55.410558 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7914cbce-00fd-4d6b-84c9-4519c50af650-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7914cbce-00fd-4d6b-84c9-4519c50af650-openshift-service-ca\") pod \"7914cbce-00fd-4d6b-84c9-4519c50af650\" (UID: \"7914cbce-00fd-4d6b-84c9-4519c50af650\") " Nov 03 23:05:55.410607 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:55.410574 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7914cbce-00fd-4d6b-84c9-4519c50af650-kube-api-access-srtb5\" (UniqueName: \"kubernetes.io/projected/7914cbce-00fd-4d6b-84c9-4519c50af650-kube-api-access-srtb5\") pod \"7914cbce-00fd-4d6b-84c9-4519c50af650\" (UID: \"7914cbce-00fd-4d6b-84c9-4519c50af650\") " Nov 03 23:05:55.411082 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:55.411054 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7914cbce-00fd-4d6b-84c9-4519c50af650-openshift-service-ca" pod "7914cbce-00fd-4d6b-84c9-4519c50af650" (UID: "7914cbce-00fd-4d6b-84c9-4519c50af650"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:55.412789 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:55.412768 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7914cbce-00fd-4d6b-84c9-4519c50af650-maas-api-tls" pod "7914cbce-00fd-4d6b-84c9-4519c50af650" (UID: "7914cbce-00fd-4d6b-84c9-4519c50af650"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:05:55.413259 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:55.413242 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7914cbce-00fd-4d6b-84c9-4519c50af650-metrics-certs" pod "7914cbce-00fd-4d6b-84c9-4519c50af650" (UID: "7914cbce-00fd-4d6b-84c9-4519c50af650"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:05:55.413354 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:55.413297 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7914cbce-00fd-4d6b-84c9-4519c50af650-kube-api-access-srtb5" pod "7914cbce-00fd-4d6b-84c9-4519c50af650" (UID: "7914cbce-00fd-4d6b-84c9-4519c50af650"). InnerVolumeSpecName "kube-api-access-srtb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:55.512330 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:55.512299 5196 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7914cbce-00fd-4d6b-84c9-4519c50af650-openshift-service-ca\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:55.512330 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:55.512324 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srtb5\" (UniqueName: \"kubernetes.io/projected/7914cbce-00fd-4d6b-84c9-4519c50af650-kube-api-access-srtb5\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:55.512330 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:55.512335 5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7914cbce-00fd-4d6b-84c9-4519c50af650-maas-api-tls\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:55.512555 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:55.512343 5196 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7914cbce-00fd-4d6b-84c9-4519c50af650-metrics-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:05:55.849070 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:55.849040 5196 generic.go:356] "Generic (PLEG): container finished" podID="7914cbce-00fd-4d6b-84c9-4519c50af650" containerID="7d0bdc6396e6d713d6b27723c120a88fff35255a4d85edb98eadf14947921e6e" exitCode=0 Nov 03 23:05:55.849271 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:55.849089 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5" Nov 03 23:05:55.849271 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:55.849125 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5" event={"ID":"7914cbce-00fd-4d6b-84c9-4519c50af650","Type":"ContainerDied","Data":"7d0bdc6396e6d713d6b27723c120a88fff35255a4d85edb98eadf14947921e6e"} Nov 03 23:05:55.849271 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:55.849193 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5" event={"ID":"7914cbce-00fd-4d6b-84c9-4519c50af650","Type":"ContainerDied","Data":"93b255f4c3e987a320b57272652e65cdbdc59d947f4f6470677624374f5e00b2"} Nov 03 23:05:55.849271 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:55.849223 5196 scope.go:122] "RemoveContainer" containerID="7d0bdc6396e6d713d6b27723c120a88fff35255a4d85edb98eadf14947921e6e" Nov 03 23:05:55.859077 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:55.859063 5196 scope.go:122] "RemoveContainer" containerID="7d0bdc6396e6d713d6b27723c120a88fff35255a4d85edb98eadf14947921e6e" Nov 03 23:05:55.859334 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:05:55.859316 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d0bdc6396e6d713d6b27723c120a88fff35255a4d85edb98eadf14947921e6e\": container with ID starting with 7d0bdc6396e6d713d6b27723c120a88fff35255a4d85edb98eadf14947921e6e not found: ID does not exist" containerID="7d0bdc6396e6d713d6b27723c120a88fff35255a4d85edb98eadf14947921e6e" Nov 03 23:05:55.859385 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:55.859342 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d0bdc6396e6d713d6b27723c120a88fff35255a4d85edb98eadf14947921e6e"} err="failed to get container status \"7d0bdc6396e6d713d6b27723c120a88fff35255a4d85edb98eadf14947921e6e\": rpc error: code = NotFound desc = could not find container \"7d0bdc6396e6d713d6b27723c120a88fff35255a4d85edb98eadf14947921e6e\": container with ID starting with 7d0bdc6396e6d713d6b27723c120a88fff35255a4d85edb98eadf14947921e6e not found: ID does not exist" Nov 03 23:05:55.918351 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:05:55.918319 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:05:56.015372 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:56.015343 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5"] Nov 03 23:05:56.019233 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:56.019208 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dcf5a5-9bbf7495c-jc6m5"] Nov 03 23:05:56.784042 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:56.784003 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:56.814776 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:56.814745 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7914cbce-00fd-4d6b-84c9-4519c50af650" path="/var/lib/kubelet/pods/7914cbce-00fd-4d6b-84c9-4519c50af650/volumes" Nov 03 23:05:57.043313 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:57.043228 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:57.199700 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:57.199670 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:05:57.788115 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:05:57.788077 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:00.636963 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:00.636931 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:00.781189 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:00.781149 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:02.184150 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:02.184104 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:03.779027 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:03.778990 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:03.935261 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:03.935223 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:04.109483 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:04.109452 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn"] Nov 03 23:06:04.109700 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:04.109678 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" podUID="4b847dde-7257-44e0-b25c-9ebe4cdad697" containerName="istio-proxy" containerID="cri-o://5a8d4cb035b182d99b74f47f82b198b0a4c1df99743dff6adc724a937ab08ce8" gracePeriod=30 Nov 03 23:06:04.129710 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:04.129682 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:07.516967 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:07.516939 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:09.370998 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:09.370938 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:06:09.454991 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:09.454961 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b847dde-7257-44e0-b25c-9ebe4cdad697-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b847dde-7257-44e0-b25c-9ebe4cdad697-credential-socket\") pod \"4b847dde-7257-44e0-b25c-9ebe4cdad697\" (UID: \"4b847dde-7257-44e0-b25c-9ebe4cdad697\") " Nov 03 23:06:09.455125 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:09.455013 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b847dde-7257-44e0-b25c-9ebe4cdad697-istio-token\" (UniqueName: \"kubernetes.io/projected/4b847dde-7257-44e0-b25c-9ebe4cdad697-istio-token\") pod \"4b847dde-7257-44e0-b25c-9ebe4cdad697\" (UID: \"4b847dde-7257-44e0-b25c-9ebe4cdad697\") " Nov 03 23:06:09.455125 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:09.455056 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b847dde-7257-44e0-b25c-9ebe4cdad697-kube-api-access-rq8kl\" (UniqueName: \"kubernetes.io/projected/4b847dde-7257-44e0-b25c-9ebe4cdad697-kube-api-access-rq8kl\") pod \"4b847dde-7257-44e0-b25c-9ebe4cdad697\" (UID: \"4b847dde-7257-44e0-b25c-9ebe4cdad697\") " Nov 03 23:06:09.455125 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:09.455098 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b847dde-7257-44e0-b25c-9ebe4cdad697-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b847dde-7257-44e0-b25c-9ebe4cdad697-workload-socket\") pod \"4b847dde-7257-44e0-b25c-9ebe4cdad697\" (UID: \"4b847dde-7257-44e0-b25c-9ebe4cdad697\") " Nov 03 23:06:09.455125 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:09.455118 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b847dde-7257-44e0-b25c-9ebe4cdad697-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b847dde-7257-44e0-b25c-9ebe4cdad697-istio-data\") pod \"4b847dde-7257-44e0-b25c-9ebe4cdad697\" (UID: \"4b847dde-7257-44e0-b25c-9ebe4cdad697\") " Nov 03 23:06:09.455321 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:09.455138 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b847dde-7257-44e0-b25c-9ebe4cdad697-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b847dde-7257-44e0-b25c-9ebe4cdad697-istio-envoy\") pod \"4b847dde-7257-44e0-b25c-9ebe4cdad697\" (UID: \"4b847dde-7257-44e0-b25c-9ebe4cdad697\") " Nov 03 23:06:09.455321 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:09.455171 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b847dde-7257-44e0-b25c-9ebe4cdad697-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b847dde-7257-44e0-b25c-9ebe4cdad697-istiod-ca-cert\") pod \"4b847dde-7257-44e0-b25c-9ebe4cdad697\" (UID: \"4b847dde-7257-44e0-b25c-9ebe4cdad697\") " Nov 03 23:06:09.455321 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:09.455237 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b847dde-7257-44e0-b25c-9ebe4cdad697-credential-socket" pod "4b847dde-7257-44e0-b25c-9ebe4cdad697" (UID: "4b847dde-7257-44e0-b25c-9ebe4cdad697"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:09.455321 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:09.455271 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b847dde-7257-44e0-b25c-9ebe4cdad697-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b847dde-7257-44e0-b25c-9ebe4cdad697-istio-podinfo\") pod \"4b847dde-7257-44e0-b25c-9ebe4cdad697\" (UID: \"4b847dde-7257-44e0-b25c-9ebe4cdad697\") " Nov 03 23:06:09.455321 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:09.455316 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b847dde-7257-44e0-b25c-9ebe4cdad697-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b847dde-7257-44e0-b25c-9ebe4cdad697-workload-certs\") pod \"4b847dde-7257-44e0-b25c-9ebe4cdad697\" (UID: \"4b847dde-7257-44e0-b25c-9ebe4cdad697\") " Nov 03 23:06:09.455590 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:09.455398 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b847dde-7257-44e0-b25c-9ebe4cdad697-workload-socket" pod "4b847dde-7257-44e0-b25c-9ebe4cdad697" (UID: "4b847dde-7257-44e0-b25c-9ebe4cdad697"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:09.455590 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:09.455415 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b847dde-7257-44e0-b25c-9ebe4cdad697-istio-data" pod "4b847dde-7257-44e0-b25c-9ebe4cdad697" (UID: "4b847dde-7257-44e0-b25c-9ebe4cdad697"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:09.455714 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:09.455613 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b847dde-7257-44e0-b25c-9ebe4cdad697-workload-certs" pod "4b847dde-7257-44e0-b25c-9ebe4cdad697" (UID: "4b847dde-7257-44e0-b25c-9ebe4cdad697"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:09.455714 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:09.455654 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b847dde-7257-44e0-b25c-9ebe4cdad697-istiod-ca-cert" pod "4b847dde-7257-44e0-b25c-9ebe4cdad697" (UID: "4b847dde-7257-44e0-b25c-9ebe4cdad697"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:09.455714 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:09.455668 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b847dde-7257-44e0-b25c-9ebe4cdad697-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:09.455714 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:09.455688 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b847dde-7257-44e0-b25c-9ebe4cdad697-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:09.455714 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:09.455703 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b847dde-7257-44e0-b25c-9ebe4cdad697-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:09.457423 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:09.457397 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b847dde-7257-44e0-b25c-9ebe4cdad697-istio-token" pod "4b847dde-7257-44e0-b25c-9ebe4cdad697" (UID: "4b847dde-7257-44e0-b25c-9ebe4cdad697"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:09.457519 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:09.457456 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b847dde-7257-44e0-b25c-9ebe4cdad697-istio-envoy" pod "4b847dde-7257-44e0-b25c-9ebe4cdad697" (UID: "4b847dde-7257-44e0-b25c-9ebe4cdad697"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:09.457850 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:09.457835 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b847dde-7257-44e0-b25c-9ebe4cdad697-kube-api-access-rq8kl" pod "4b847dde-7257-44e0-b25c-9ebe4cdad697" (UID: "4b847dde-7257-44e0-b25c-9ebe4cdad697"). InnerVolumeSpecName "kube-api-access-rq8kl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:09.457933 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:09.457915 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b847dde-7257-44e0-b25c-9ebe4cdad697-istio-podinfo" pod "4b847dde-7257-44e0-b25c-9ebe4cdad697" (UID: "4b847dde-7257-44e0-b25c-9ebe4cdad697"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:06:09.557091 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:09.557062 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq8kl\" (UniqueName: \"kubernetes.io/projected/4b847dde-7257-44e0-b25c-9ebe4cdad697-kube-api-access-rq8kl\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:09.557091 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:09.557088 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b847dde-7257-44e0-b25c-9ebe4cdad697-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:09.557091 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:09.557098 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b847dde-7257-44e0-b25c-9ebe4cdad697-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:09.557300 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:09.557106 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b847dde-7257-44e0-b25c-9ebe4cdad697-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:09.557300 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:09.557115 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b847dde-7257-44e0-b25c-9ebe4cdad697-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:09.557300 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:09.557123 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b847dde-7257-44e0-b25c-9ebe4cdad697-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:09.929988 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:09.929955 5196 generic.go:356] "Generic (PLEG): container finished" podID="4b847dde-7257-44e0-b25c-9ebe4cdad697" containerID="5a8d4cb035b182d99b74f47f82b198b0a4c1df99743dff6adc724a937ab08ce8" exitCode=0 Nov 03 23:06:09.930177 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:09.930011 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" Nov 03 23:06:09.930177 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:09.930043 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" event={"ID":"4b847dde-7257-44e0-b25c-9ebe4cdad697","Type":"ContainerDied","Data":"5a8d4cb035b182d99b74f47f82b198b0a4c1df99743dff6adc724a937ab08ce8"} Nov 03 23:06:09.930177 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:09.930100 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn" event={"ID":"4b847dde-7257-44e0-b25c-9ebe4cdad697","Type":"ContainerDied","Data":"8b1f5a79eb4f31269c5c2895d8168abd9dcfcd4666ee61e0fb4f8aacefece256"} Nov 03 23:06:09.930177 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:09.930123 5196 scope.go:122] "RemoveContainer" containerID="5a8d4cb035b182d99b74f47f82b198b0a4c1df99743dff6adc724a937ab08ce8" Nov 03 23:06:09.940397 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:09.940377 5196 scope.go:122] "RemoveContainer" containerID="031184b6b8125e3f772103763af015ba5f2dff52af37f785d5fbdac5b41e5c00" Nov 03 23:06:09.948966 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:09.948946 5196 scope.go:122] "RemoveContainer" containerID="5a8d4cb035b182d99b74f47f82b198b0a4c1df99743dff6adc724a937ab08ce8" Nov 03 23:06:09.949247 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:06:09.949228 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a8d4cb035b182d99b74f47f82b198b0a4c1df99743dff6adc724a937ab08ce8\": container with ID starting with 5a8d4cb035b182d99b74f47f82b198b0a4c1df99743dff6adc724a937ab08ce8 not found: ID does not exist" containerID="5a8d4cb035b182d99b74f47f82b198b0a4c1df99743dff6adc724a937ab08ce8" Nov 03 23:06:09.949292 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:09.949255 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a8d4cb035b182d99b74f47f82b198b0a4c1df99743dff6adc724a937ab08ce8"} err="failed to get container status \"5a8d4cb035b182d99b74f47f82b198b0a4c1df99743dff6adc724a937ab08ce8\": rpc error: code = NotFound desc = could not find container \"5a8d4cb035b182d99b74f47f82b198b0a4c1df99743dff6adc724a937ab08ce8\": container with ID starting with 5a8d4cb035b182d99b74f47f82b198b0a4c1df99743dff6adc724a937ab08ce8 not found: ID does not exist" Nov 03 23:06:09.949292 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:09.949273 5196 scope.go:122] "RemoveContainer" containerID="031184b6b8125e3f772103763af015ba5f2dff52af37f785d5fbdac5b41e5c00" Nov 03 23:06:09.949502 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:06:09.949485 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"031184b6b8125e3f772103763af015ba5f2dff52af37f785d5fbdac5b41e5c00\": container with ID starting with 031184b6b8125e3f772103763af015ba5f2dff52af37f785d5fbdac5b41e5c00 not found: ID does not exist" containerID="031184b6b8125e3f772103763af015ba5f2dff52af37f785d5fbdac5b41e5c00" Nov 03 23:06:09.949540 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:09.949510 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"031184b6b8125e3f772103763af015ba5f2dff52af37f785d5fbdac5b41e5c00"} err="failed to get container status \"031184b6b8125e3f772103763af015ba5f2dff52af37f785d5fbdac5b41e5c00\": rpc error: code = NotFound desc = could not find container \"031184b6b8125e3f772103763af015ba5f2dff52af37f785d5fbdac5b41e5c00\": container with ID starting with 031184b6b8125e3f772103763af015ba5f2dff52af37f785d5fbdac5b41e5c00 not found: ID does not exist" Nov 03 23:06:10.097892 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:10.097847 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn"] Nov 03 23:06:10.100839 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:10.100817 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-dcf5a5-openshift-default-68bf4b7597-qtcnn"] Nov 03 23:06:10.814677 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:10.814645 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b847dde-7257-44e0-b25c-9ebe4cdad697" path="/var/lib/kubelet/pods/4b847dde-7257-44e0-b25c-9ebe4cdad697/volumes" Nov 03 23:06:11.335358 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:11.335310 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:12.307365 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:12.307337 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:16.680056 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:16.680019 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:16.916270 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:16.916239 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:17.070811 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:17.070738 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:21.792580 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:21.792545 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:25.876031 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:25.875985 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2"] Nov 03 23:06:25.876521 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:25.876291 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2" podUID="5f4dad96-aaa9-467a-b599-6b45196c5eaa" containerName="maas-api" containerID="cri-o://e8ee163f30526671fc382c4ab7368088664d7caed8e0b46c4f7bfdbc6901c5db" gracePeriod=30 Nov 03 23:06:25.920904 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:06:25.920833 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:06:25.971971 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:25.971941 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:26.030172 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:26.030137 5196 generic.go:356] "Generic (PLEG): container finished" podID="5f4dad96-aaa9-467a-b599-6b45196c5eaa" containerID="e8ee163f30526671fc382c4ab7368088664d7caed8e0b46c4f7bfdbc6901c5db" exitCode=0 Nov 03 23:06:26.030337 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:26.030186 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2" event={"ID":"5f4dad96-aaa9-467a-b599-6b45196c5eaa","Type":"ContainerDied","Data":"e8ee163f30526671fc382c4ab7368088664d7caed8e0b46c4f7bfdbc6901c5db"} Nov 03 23:06:26.126274 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:26.126212 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2" Nov 03 23:06:26.220667 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:26.220635 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f4dad96-aaa9-467a-b599-6b45196c5eaa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f4dad96-aaa9-467a-b599-6b45196c5eaa-openshift-service-ca\") pod \"5f4dad96-aaa9-467a-b599-6b45196c5eaa\" (UID: \"5f4dad96-aaa9-467a-b599-6b45196c5eaa\") " Nov 03 23:06:26.220886 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:26.220703 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f4dad96-aaa9-467a-b599-6b45196c5eaa-kube-api-access-cjplx\" (UniqueName: \"kubernetes.io/projected/5f4dad96-aaa9-467a-b599-6b45196c5eaa-kube-api-access-cjplx\") pod \"5f4dad96-aaa9-467a-b599-6b45196c5eaa\" (UID: \"5f4dad96-aaa9-467a-b599-6b45196c5eaa\") " Nov 03 23:06:26.220886 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:26.220746 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f4dad96-aaa9-467a-b599-6b45196c5eaa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f4dad96-aaa9-467a-b599-6b45196c5eaa-metrics-certs\") pod \"5f4dad96-aaa9-467a-b599-6b45196c5eaa\" (UID: \"5f4dad96-aaa9-467a-b599-6b45196c5eaa\") " Nov 03 23:06:26.220886 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:26.220835 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f4dad96-aaa9-467a-b599-6b45196c5eaa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f4dad96-aaa9-467a-b599-6b45196c5eaa-maas-api-tls\") pod \"5f4dad96-aaa9-467a-b599-6b45196c5eaa\" (UID: \"5f4dad96-aaa9-467a-b599-6b45196c5eaa\") " Nov 03 23:06:26.221076 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:26.221050 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f4dad96-aaa9-467a-b599-6b45196c5eaa-openshift-service-ca" pod "5f4dad96-aaa9-467a-b599-6b45196c5eaa" (UID: "5f4dad96-aaa9-467a-b599-6b45196c5eaa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:26.221242 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:26.221219 5196 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f4dad96-aaa9-467a-b599-6b45196c5eaa-openshift-service-ca\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:26.223090 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:26.223066 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f4dad96-aaa9-467a-b599-6b45196c5eaa-metrics-certs" pod "5f4dad96-aaa9-467a-b599-6b45196c5eaa" (UID: "5f4dad96-aaa9-467a-b599-6b45196c5eaa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:26.223498 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:26.223483 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f4dad96-aaa9-467a-b599-6b45196c5eaa-maas-api-tls" pod "5f4dad96-aaa9-467a-b599-6b45196c5eaa" (UID: "5f4dad96-aaa9-467a-b599-6b45196c5eaa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:26.223595 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:26.223576 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f4dad96-aaa9-467a-b599-6b45196c5eaa-kube-api-access-cjplx" pod "5f4dad96-aaa9-467a-b599-6b45196c5eaa" (UID: "5f4dad96-aaa9-467a-b599-6b45196c5eaa"). InnerVolumeSpecName "kube-api-access-cjplx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:26.322129 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:26.322098 5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f4dad96-aaa9-467a-b599-6b45196c5eaa-maas-api-tls\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:26.322129 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:26.322126 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjplx\" (UniqueName: \"kubernetes.io/projected/5f4dad96-aaa9-467a-b599-6b45196c5eaa-kube-api-access-cjplx\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:26.322129 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:26.322136 5196 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f4dad96-aaa9-467a-b599-6b45196c5eaa-metrics-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:27.037388 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:27.037359 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2" Nov 03 23:06:27.037835 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:27.037388 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2" event={"ID":"5f4dad96-aaa9-467a-b599-6b45196c5eaa","Type":"ContainerDied","Data":"294ba27ecd0bbed9aec336f6ac158835d02f5e7112328d7df9f2e644ea051076"} Nov 03 23:06:27.037835 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:27.037456 5196 scope.go:122] "RemoveContainer" containerID="e8ee163f30526671fc382c4ab7368088664d7caed8e0b46c4f7bfdbc6901c5db" Nov 03 23:06:27.205011 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:27.204979 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2"] Nov 03 23:06:27.255905 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:27.255872 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-78be2c-b96b8766b-wmjt2"] Nov 03 23:06:27.476714 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:27.476638 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:27.727512 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:27.727433 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:27.873840 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:27.873809 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:28.426174 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:28.426140 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:28.815391 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:28.815307 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f4dad96-aaa9-467a-b599-6b45196c5eaa" path="/var/lib/kubelet/pods/5f4dad96-aaa9-467a-b599-6b45196c5eaa/volumes" Nov 03 23:06:29.767551 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:29.767399 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:29.924436 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:29.924404 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:31.709320 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:31.709281 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:32.923833 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:32.923778 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:34.872176 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:34.872146 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:35.148934 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:35.148805 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:35.303538 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:35.303507 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:37.682429 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:37.682400 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:37.828042 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:37.828002 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:37.980017 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:37.979940 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:42.632354 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:42.632317 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq"] Nov 03 23:06:42.632786 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:42.632555 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq" podUID="c0801535-901a-46f0-a882-e9634537f0fc" containerName="maas-api" containerID="cri-o://572d15d69c928ab800f55c1991f6d9ad1ab421ceb8dc8b3e742bb41a65dd52ce" gracePeriod=30 Nov 03 23:06:42.734560 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:42.734530 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:42.897113 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:42.897045 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq" Nov 03 23:06:42.994413 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:42.994385 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0801535-901a-46f0-a882-e9634537f0fc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0801535-901a-46f0-a882-e9634537f0fc-maas-api-tls\") pod \"c0801535-901a-46f0-a882-e9634537f0fc\" (UID: \"c0801535-901a-46f0-a882-e9634537f0fc\") " Nov 03 23:06:42.994630 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:42.994422 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0801535-901a-46f0-a882-e9634537f0fc-kube-api-access-jsgwh\" (UniqueName: \"kubernetes.io/projected/c0801535-901a-46f0-a882-e9634537f0fc-kube-api-access-jsgwh\") pod \"c0801535-901a-46f0-a882-e9634537f0fc\" (UID: \"c0801535-901a-46f0-a882-e9634537f0fc\") " Nov 03 23:06:42.994630 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:42.994450 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0801535-901a-46f0-a882-e9634537f0fc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0801535-901a-46f0-a882-e9634537f0fc-metrics-certs\") pod \"c0801535-901a-46f0-a882-e9634537f0fc\" (UID: \"c0801535-901a-46f0-a882-e9634537f0fc\") " Nov 03 23:06:42.994630 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:42.994535 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0801535-901a-46f0-a882-e9634537f0fc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0801535-901a-46f0-a882-e9634537f0fc-openshift-service-ca\") pod \"c0801535-901a-46f0-a882-e9634537f0fc\" (UID: \"c0801535-901a-46f0-a882-e9634537f0fc\") " Nov 03 23:06:42.994977 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:42.994952 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0801535-901a-46f0-a882-e9634537f0fc-openshift-service-ca" pod "c0801535-901a-46f0-a882-e9634537f0fc" (UID: "c0801535-901a-46f0-a882-e9634537f0fc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:42.996745 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:42.996716 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0801535-901a-46f0-a882-e9634537f0fc-maas-api-tls" pod "c0801535-901a-46f0-a882-e9634537f0fc" (UID: "c0801535-901a-46f0-a882-e9634537f0fc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:42.996875 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:42.996758 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0801535-901a-46f0-a882-e9634537f0fc-metrics-certs" pod "c0801535-901a-46f0-a882-e9634537f0fc" (UID: "c0801535-901a-46f0-a882-e9634537f0fc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:42.996875 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:42.996789 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0801535-901a-46f0-a882-e9634537f0fc-kube-api-access-jsgwh" pod "c0801535-901a-46f0-a882-e9634537f0fc" (UID: "c0801535-901a-46f0-a882-e9634537f0fc"). InnerVolumeSpecName "kube-api-access-jsgwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:43.095425 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:43.095393 5196 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0801535-901a-46f0-a882-e9634537f0fc-openshift-service-ca\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:43.095425 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:43.095426 5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0801535-901a-46f0-a882-e9634537f0fc-maas-api-tls\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:43.095634 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:43.095439 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsgwh\" (UniqueName: \"kubernetes.io/projected/c0801535-901a-46f0-a882-e9634537f0fc-kube-api-access-jsgwh\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:43.095634 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:43.095448 5196 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0801535-901a-46f0-a882-e9634537f0fc-metrics-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:06:43.136895 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:43.136852 5196 generic.go:356] "Generic (PLEG): container finished" podID="c0801535-901a-46f0-a882-e9634537f0fc" containerID="572d15d69c928ab800f55c1991f6d9ad1ab421ceb8dc8b3e742bb41a65dd52ce" exitCode=0 Nov 03 23:06:43.137063 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:43.136903 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq" event={"ID":"c0801535-901a-46f0-a882-e9634537f0fc","Type":"ContainerDied","Data":"572d15d69c928ab800f55c1991f6d9ad1ab421ceb8dc8b3e742bb41a65dd52ce"} Nov 03 23:06:43.137063 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:43.136976 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq" event={"ID":"c0801535-901a-46f0-a882-e9634537f0fc","Type":"ContainerDied","Data":"93236eb763cfe2bc9a4cfb43307574d45d143fe0e29598088e5c121180aa93fb"} Nov 03 23:06:43.137063 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:43.136918 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq" Nov 03 23:06:43.137063 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:43.137006 5196 scope.go:122] "RemoveContainer" containerID="572d15d69c928ab800f55c1991f6d9ad1ab421ceb8dc8b3e742bb41a65dd52ce" Nov 03 23:06:43.147462 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:43.147442 5196 scope.go:122] "RemoveContainer" containerID="572d15d69c928ab800f55c1991f6d9ad1ab421ceb8dc8b3e742bb41a65dd52ce" Nov 03 23:06:43.147756 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:06:43.147737 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"572d15d69c928ab800f55c1991f6d9ad1ab421ceb8dc8b3e742bb41a65dd52ce\": container with ID starting with 572d15d69c928ab800f55c1991f6d9ad1ab421ceb8dc8b3e742bb41a65dd52ce not found: ID does not exist" containerID="572d15d69c928ab800f55c1991f6d9ad1ab421ceb8dc8b3e742bb41a65dd52ce" Nov 03 23:06:43.147820 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:43.147765 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"572d15d69c928ab800f55c1991f6d9ad1ab421ceb8dc8b3e742bb41a65dd52ce"} err="failed to get container status \"572d15d69c928ab800f55c1991f6d9ad1ab421ceb8dc8b3e742bb41a65dd52ce\": rpc error: code = NotFound desc = could not find container \"572d15d69c928ab800f55c1991f6d9ad1ab421ceb8dc8b3e742bb41a65dd52ce\": container with ID starting with 572d15d69c928ab800f55c1991f6d9ad1ab421ceb8dc8b3e742bb41a65dd52ce not found: ID does not exist" Nov 03 23:06:43.303464 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:43.303432 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq"] Nov 03 23:06:43.307247 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:43.307221 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c93e70-664db949c7-7vdpq"] Nov 03 23:06:44.427732 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:44.427689 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:44.801143 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:44.801055 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:44.814142 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:44.814114 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0801535-901a-46f0-a882-e9634537f0fc" path="/var/lib/kubelet/pods/c0801535-901a-46f0-a882-e9634537f0fc/volumes" Nov 03 23:06:44.977733 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:44.977705 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:45.420786 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:45.420756 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:46.908110 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:46.908045 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:47.059103 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:47.059069 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:48.029426 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:48.029383 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:48.178260 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:48.178233 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:49.219258 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:49.219228 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:50.824587 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:50.824556 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:51.110157 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:51.110131 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:51.259584 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:51.259550 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:51.596182 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:51.596143 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:51.743688 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:51.743660 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:51.851995 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:51.851905 5196 scope.go:122] "RemoveContainer" containerID="b5b8619a973adca607afdf1f0ce3ccf67721e6eda1b2a41d772aa8407339fe29" Nov 03 23:06:51.961211 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:51.961178 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:55.918248 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:06:55.918205 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:06:56.517197 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:56.517169 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:57.822488 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:57.822455 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:57.973036 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:57.973002 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:06:58.140753 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:58.140726 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz"] Nov 03 23:06:58.140979 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:58.140958 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" podUID="0d375a3f-2f84-4495-a683-42a98477ff52" containerName="istio-proxy" containerID="cri-o://ffbd4f361bac15f9b185f79737d3e7b7fa9dc6cabe4e44eefdcb8c2b61c3b03c" gracePeriod=30 Nov 03 23:06:58.147600 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:06:58.147574 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:01.717902 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:01.717838 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:03.407630 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:03.407569 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 23:07:03.499322 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:03.499282 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d375a3f-2f84-4495-a683-42a98477ff52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d375a3f-2f84-4495-a683-42a98477ff52-credential-socket\") pod \"0d375a3f-2f84-4495-a683-42a98477ff52\" (UID: \"0d375a3f-2f84-4495-a683-42a98477ff52\") " Nov 03 23:07:03.499322 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:03.499323 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d375a3f-2f84-4495-a683-42a98477ff52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d375a3f-2f84-4495-a683-42a98477ff52-istio-podinfo\") pod \"0d375a3f-2f84-4495-a683-42a98477ff52\" (UID: \"0d375a3f-2f84-4495-a683-42a98477ff52\") " Nov 03 23:07:03.499545 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:03.499372 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d375a3f-2f84-4495-a683-42a98477ff52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d375a3f-2f84-4495-a683-42a98477ff52-workload-socket\") pod \"0d375a3f-2f84-4495-a683-42a98477ff52\" (UID: \"0d375a3f-2f84-4495-a683-42a98477ff52\") " Nov 03 23:07:03.499545 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:03.499421 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d375a3f-2f84-4495-a683-42a98477ff52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d375a3f-2f84-4495-a683-42a98477ff52-istio-data\") pod \"0d375a3f-2f84-4495-a683-42a98477ff52\" (UID: \"0d375a3f-2f84-4495-a683-42a98477ff52\") " Nov 03 23:07:03.499545 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:03.499471 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d375a3f-2f84-4495-a683-42a98477ff52-kube-api-access-qwg2l\" (UniqueName: \"kubernetes.io/projected/0d375a3f-2f84-4495-a683-42a98477ff52-kube-api-access-qwg2l\") pod \"0d375a3f-2f84-4495-a683-42a98477ff52\" (UID: \"0d375a3f-2f84-4495-a683-42a98477ff52\") " Nov 03 23:07:03.499545 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:03.499537 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d375a3f-2f84-4495-a683-42a98477ff52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d375a3f-2f84-4495-a683-42a98477ff52-istiod-ca-cert\") pod \"0d375a3f-2f84-4495-a683-42a98477ff52\" (UID: \"0d375a3f-2f84-4495-a683-42a98477ff52\") " Nov 03 23:07:03.499757 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:03.499563 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d375a3f-2f84-4495-a683-42a98477ff52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d375a3f-2f84-4495-a683-42a98477ff52-workload-certs\") pod \"0d375a3f-2f84-4495-a683-42a98477ff52\" (UID: \"0d375a3f-2f84-4495-a683-42a98477ff52\") " Nov 03 23:07:03.499757 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:03.499585 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d375a3f-2f84-4495-a683-42a98477ff52-istio-token\" (UniqueName: \"kubernetes.io/projected/0d375a3f-2f84-4495-a683-42a98477ff52-istio-token\") pod \"0d375a3f-2f84-4495-a683-42a98477ff52\" (UID: \"0d375a3f-2f84-4495-a683-42a98477ff52\") " Nov 03 23:07:03.499757 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:03.499615 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d375a3f-2f84-4495-a683-42a98477ff52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d375a3f-2f84-4495-a683-42a98477ff52-istio-envoy\") pod \"0d375a3f-2f84-4495-a683-42a98477ff52\" (UID: \"0d375a3f-2f84-4495-a683-42a98477ff52\") " Nov 03 23:07:03.499757 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:03.499616 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d375a3f-2f84-4495-a683-42a98477ff52-credential-socket" pod "0d375a3f-2f84-4495-a683-42a98477ff52" (UID: "0d375a3f-2f84-4495-a683-42a98477ff52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:03.500008 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:03.499890 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d375a3f-2f84-4495-a683-42a98477ff52-istio-data" pod "0d375a3f-2f84-4495-a683-42a98477ff52" (UID: "0d375a3f-2f84-4495-a683-42a98477ff52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:03.500008 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:03.499917 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d375a3f-2f84-4495-a683-42a98477ff52-workload-certs" pod "0d375a3f-2f84-4495-a683-42a98477ff52" (UID: "0d375a3f-2f84-4495-a683-42a98477ff52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:03.500008 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:03.499954 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d375a3f-2f84-4495-a683-42a98477ff52-istiod-ca-cert" pod "0d375a3f-2f84-4495-a683-42a98477ff52" (UID: "0d375a3f-2f84-4495-a683-42a98477ff52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:03.500232 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:03.500044 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d375a3f-2f84-4495-a683-42a98477ff52-workload-socket" pod "0d375a3f-2f84-4495-a683-42a98477ff52" (UID: "0d375a3f-2f84-4495-a683-42a98477ff52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:03.500232 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:03.500065 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d375a3f-2f84-4495-a683-42a98477ff52-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:03.500232 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:03.500087 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d375a3f-2f84-4495-a683-42a98477ff52-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:03.500232 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:03.500104 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d375a3f-2f84-4495-a683-42a98477ff52-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:03.500232 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:03.500118 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d375a3f-2f84-4495-a683-42a98477ff52-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:03.502166 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:03.502128 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d375a3f-2f84-4495-a683-42a98477ff52-istio-envoy" pod "0d375a3f-2f84-4495-a683-42a98477ff52" (UID: "0d375a3f-2f84-4495-a683-42a98477ff52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:03.502300 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:03.502244 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d375a3f-2f84-4495-a683-42a98477ff52-istio-token" pod "0d375a3f-2f84-4495-a683-42a98477ff52" (UID: "0d375a3f-2f84-4495-a683-42a98477ff52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:03.502430 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:03.502406 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d375a3f-2f84-4495-a683-42a98477ff52-kube-api-access-qwg2l" pod "0d375a3f-2f84-4495-a683-42a98477ff52" (UID: "0d375a3f-2f84-4495-a683-42a98477ff52"). InnerVolumeSpecName "kube-api-access-qwg2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:03.502528 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:03.502472 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d375a3f-2f84-4495-a683-42a98477ff52-istio-podinfo" pod "0d375a3f-2f84-4495-a683-42a98477ff52" (UID: "0d375a3f-2f84-4495-a683-42a98477ff52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:07:03.601551 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:03.601515 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwg2l\" (UniqueName: \"kubernetes.io/projected/0d375a3f-2f84-4495-a683-42a98477ff52-kube-api-access-qwg2l\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:03.601551 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:03.601550 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d375a3f-2f84-4495-a683-42a98477ff52-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:03.601551 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:03.601560 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d375a3f-2f84-4495-a683-42a98477ff52-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:03.601797 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:03.601569 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d375a3f-2f84-4495-a683-42a98477ff52-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:03.601797 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:03.601578 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d375a3f-2f84-4495-a683-42a98477ff52-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:03.604007 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:03.603979 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:03.750281 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:03.750187 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:04.267433 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:04.267393 5196 generic.go:356] "Generic (PLEG): container finished" podID="0d375a3f-2f84-4495-a683-42a98477ff52" containerID="ffbd4f361bac15f9b185f79737d3e7b7fa9dc6cabe4e44eefdcb8c2b61c3b03c" exitCode=0 Nov 03 23:07:04.267631 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:04.267462 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" Nov 03 23:07:04.267631 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:04.267487 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" event={"ID":"0d375a3f-2f84-4495-a683-42a98477ff52","Type":"ContainerDied","Data":"ffbd4f361bac15f9b185f79737d3e7b7fa9dc6cabe4e44eefdcb8c2b61c3b03c"} Nov 03 23:07:04.267631 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:04.267578 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz" event={"ID":"0d375a3f-2f84-4495-a683-42a98477ff52","Type":"ContainerDied","Data":"3414d0eee6b63461b1168ca371619e6dbf37cee6b4d7e8f92e46a06f08233698"} Nov 03 23:07:04.267631 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:04.267606 5196 scope.go:122] "RemoveContainer" containerID="ffbd4f361bac15f9b185f79737d3e7b7fa9dc6cabe4e44eefdcb8c2b61c3b03c" Nov 03 23:07:04.277705 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:04.277689 5196 scope.go:122] "RemoveContainer" containerID="162270799d34e633401238397fd27799cf07fc10e603e751e08527bd0178b50a" Nov 03 23:07:04.286829 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:04.286813 5196 scope.go:122] "RemoveContainer" containerID="ffbd4f361bac15f9b185f79737d3e7b7fa9dc6cabe4e44eefdcb8c2b61c3b03c" Nov 03 23:07:04.287147 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:07:04.287123 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffbd4f361bac15f9b185f79737d3e7b7fa9dc6cabe4e44eefdcb8c2b61c3b03c\": container with ID starting with ffbd4f361bac15f9b185f79737d3e7b7fa9dc6cabe4e44eefdcb8c2b61c3b03c not found: ID does not exist" containerID="ffbd4f361bac15f9b185f79737d3e7b7fa9dc6cabe4e44eefdcb8c2b61c3b03c" Nov 03 23:07:04.287202 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:04.287161 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffbd4f361bac15f9b185f79737d3e7b7fa9dc6cabe4e44eefdcb8c2b61c3b03c"} err="failed to get container status \"ffbd4f361bac15f9b185f79737d3e7b7fa9dc6cabe4e44eefdcb8c2b61c3b03c\": rpc error: code = NotFound desc = could not find container \"ffbd4f361bac15f9b185f79737d3e7b7fa9dc6cabe4e44eefdcb8c2b61c3b03c\": container with ID starting with ffbd4f361bac15f9b185f79737d3e7b7fa9dc6cabe4e44eefdcb8c2b61c3b03c not found: ID does not exist" Nov 03 23:07:04.287202 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:04.287185 5196 scope.go:122] "RemoveContainer" containerID="162270799d34e633401238397fd27799cf07fc10e603e751e08527bd0178b50a" Nov 03 23:07:04.287447 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:07:04.287428 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"162270799d34e633401238397fd27799cf07fc10e603e751e08527bd0178b50a\": container with ID starting with 162270799d34e633401238397fd27799cf07fc10e603e751e08527bd0178b50a not found: ID does not exist" containerID="162270799d34e633401238397fd27799cf07fc10e603e751e08527bd0178b50a" Nov 03 23:07:04.287485 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:04.287454 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"162270799d34e633401238397fd27799cf07fc10e603e751e08527bd0178b50a"} err="failed to get container status \"162270799d34e633401238397fd27799cf07fc10e603e751e08527bd0178b50a\": rpc error: code = NotFound desc = could not find container \"162270799d34e633401238397fd27799cf07fc10e603e751e08527bd0178b50a\": container with ID starting with 162270799d34e633401238397fd27799cf07fc10e603e751e08527bd0178b50a not found: ID does not exist" Nov 03 23:07:04.434616 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:04.434581 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz"] Nov 03 23:07:04.485240 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:04.485208 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-1048fc-openshift-default-887fc8d4f-lq7gz"] Nov 03 23:07:04.815640 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:04.815608 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d375a3f-2f84-4495-a683-42a98477ff52" path="/var/lib/kubelet/pods/0d375a3f-2f84-4495-a683-42a98477ff52/volumes" Nov 03 23:07:06.230118 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:06.230082 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl"] Nov 03 23:07:06.230602 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:06.230327 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl" podUID="ff5d39d8-8b0c-4dc2-ab16-7626474d04de" containerName="maas-api" containerID="cri-o://dabc62bc0af004394bec0614005191fbeb7487e2d474ce0fa95aafa7c0c1f559" gracePeriod=30 Nov 03 23:07:06.336151 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:06.336103 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:06.504573 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:06.504542 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl" Nov 03 23:07:06.630356 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:06.630322 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff5d39d8-8b0c-4dc2-ab16-7626474d04de-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff5d39d8-8b0c-4dc2-ab16-7626474d04de-maas-api-tls\") pod \"ff5d39d8-8b0c-4dc2-ab16-7626474d04de\" (UID: \"ff5d39d8-8b0c-4dc2-ab16-7626474d04de\") " Nov 03 23:07:06.630547 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:06.630401 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff5d39d8-8b0c-4dc2-ab16-7626474d04de-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff5d39d8-8b0c-4dc2-ab16-7626474d04de-metrics-certs\") pod \"ff5d39d8-8b0c-4dc2-ab16-7626474d04de\" (UID: \"ff5d39d8-8b0c-4dc2-ab16-7626474d04de\") " Nov 03 23:07:06.630547 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:06.630505 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff5d39d8-8b0c-4dc2-ab16-7626474d04de-kube-api-access-4gfxr\" (UniqueName: \"kubernetes.io/projected/ff5d39d8-8b0c-4dc2-ab16-7626474d04de-kube-api-access-4gfxr\") pod \"ff5d39d8-8b0c-4dc2-ab16-7626474d04de\" (UID: \"ff5d39d8-8b0c-4dc2-ab16-7626474d04de\") " Nov 03 23:07:06.630547 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:06.630534 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff5d39d8-8b0c-4dc2-ab16-7626474d04de-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff5d39d8-8b0c-4dc2-ab16-7626474d04de-openshift-service-ca\") pod \"ff5d39d8-8b0c-4dc2-ab16-7626474d04de\" (UID: \"ff5d39d8-8b0c-4dc2-ab16-7626474d04de\") " Nov 03 23:07:06.631202 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:06.631177 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff5d39d8-8b0c-4dc2-ab16-7626474d04de-openshift-service-ca" pod "ff5d39d8-8b0c-4dc2-ab16-7626474d04de" (UID: "ff5d39d8-8b0c-4dc2-ab16-7626474d04de"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:06.633345 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:06.633319 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff5d39d8-8b0c-4dc2-ab16-7626474d04de-maas-api-tls" pod "ff5d39d8-8b0c-4dc2-ab16-7626474d04de" (UID: "ff5d39d8-8b0c-4dc2-ab16-7626474d04de"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:07:06.633590 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:06.633533 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff5d39d8-8b0c-4dc2-ab16-7626474d04de-kube-api-access-4gfxr" pod "ff5d39d8-8b0c-4dc2-ab16-7626474d04de" (UID: "ff5d39d8-8b0c-4dc2-ab16-7626474d04de"). InnerVolumeSpecName "kube-api-access-4gfxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:06.633715 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:06.633633 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff5d39d8-8b0c-4dc2-ab16-7626474d04de-metrics-certs" pod "ff5d39d8-8b0c-4dc2-ab16-7626474d04de" (UID: "ff5d39d8-8b0c-4dc2-ab16-7626474d04de"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:07:06.732419 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:06.732339 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gfxr\" (UniqueName: \"kubernetes.io/projected/ff5d39d8-8b0c-4dc2-ab16-7626474d04de-kube-api-access-4gfxr\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:06.732419 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:06.732369 5196 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff5d39d8-8b0c-4dc2-ab16-7626474d04de-openshift-service-ca\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:06.732419 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:06.732380 5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff5d39d8-8b0c-4dc2-ab16-7626474d04de-maas-api-tls\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:06.732419 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:06.732388 5196 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff5d39d8-8b0c-4dc2-ab16-7626474d04de-metrics-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:07.286942 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:07.286905 5196 generic.go:356] "Generic (PLEG): container finished" podID="ff5d39d8-8b0c-4dc2-ab16-7626474d04de" containerID="dabc62bc0af004394bec0614005191fbeb7487e2d474ce0fa95aafa7c0c1f559" exitCode=0 Nov 03 23:07:07.287794 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:07.286967 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl" event={"ID":"ff5d39d8-8b0c-4dc2-ab16-7626474d04de","Type":"ContainerDied","Data":"dabc62bc0af004394bec0614005191fbeb7487e2d474ce0fa95aafa7c0c1f559"} Nov 03 23:07:07.287794 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:07.286969 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl" Nov 03 23:07:07.287794 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:07.287007 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl" event={"ID":"ff5d39d8-8b0c-4dc2-ab16-7626474d04de","Type":"ContainerDied","Data":"1b5698880d69bbc9f4d2eb8d54889ecec5276d20a38809c15d0a9c341844d73f"} Nov 03 23:07:07.287794 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:07.287029 5196 scope.go:122] "RemoveContainer" containerID="dabc62bc0af004394bec0614005191fbeb7487e2d474ce0fa95aafa7c0c1f559" Nov 03 23:07:07.296765 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:07.296745 5196 scope.go:122] "RemoveContainer" containerID="dabc62bc0af004394bec0614005191fbeb7487e2d474ce0fa95aafa7c0c1f559" Nov 03 23:07:07.297062 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:07:07.297043 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dabc62bc0af004394bec0614005191fbeb7487e2d474ce0fa95aafa7c0c1f559\": container with ID starting with dabc62bc0af004394bec0614005191fbeb7487e2d474ce0fa95aafa7c0c1f559 not found: ID does not exist" containerID="dabc62bc0af004394bec0614005191fbeb7487e2d474ce0fa95aafa7c0c1f559" Nov 03 23:07:07.297130 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:07.297071 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dabc62bc0af004394bec0614005191fbeb7487e2d474ce0fa95aafa7c0c1f559"} err="failed to get container status \"dabc62bc0af004394bec0614005191fbeb7487e2d474ce0fa95aafa7c0c1f559\": rpc error: code = NotFound desc = could not find container \"dabc62bc0af004394bec0614005191fbeb7487e2d474ce0fa95aafa7c0c1f559\": container with ID starting with dabc62bc0af004394bec0614005191fbeb7487e2d474ce0fa95aafa7c0c1f559 not found: ID does not exist" Nov 03 23:07:07.455573 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:07.455539 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl"] Nov 03 23:07:07.506422 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:07.506393 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eb3c33-5bf9fbb77b-vkzhl"] Nov 03 23:07:07.887811 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:07.887751 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:08.124215 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:08.124181 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:08.282198 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:08.282099 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:08.813657 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:08.813626 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:08.814577 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:08.814555 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff5d39d8-8b0c-4dc2-ab16-7626474d04de" path="/var/lib/kubelet/pods/ff5d39d8-8b0c-4dc2-ab16-7626474d04de/volumes" Nov 03 23:07:10.972560 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:10.972528 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:11.118399 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:11.118359 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:11.677093 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:11.677054 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:13.311027 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:13.310993 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:14.910037 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:14.909973 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:15.056210 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:15.056178 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:15.206447 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:15.206354 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:15.362085 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:15.362049 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:17.055245 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:17.055208 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:18.446469 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:18.446435 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:22.841078 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:22.841041 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:24.394675 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:24.394646 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:24.611841 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:24.611809 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:24.760248 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:24.760163 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:25.303752 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:25.303715 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:25.919371 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:07:25.919337 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:07:27.934032 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:27.933998 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:28.079236 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:28.079198 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:29.125756 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:29.125724 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:30.689384 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:30.689350 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:30.966849 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:30.966756 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:31.113745 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:31.113715 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:31.466770 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:31.466738 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:31.615047 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:31.614994 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:31.791962 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:31.791869 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:36.155634 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:36.155577 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:37.776483 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:37.776445 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:37.926798 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:37.926762 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:38.151517 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:38.151489 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:38.637198 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:38.637164 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:40.676802 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:40.676769 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k"] Nov 03 23:07:40.677245 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:40.676990 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k" podUID="3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b" containerName="maas-api" containerID="cri-o://cb823ccc60b573dfae944eef3c494c36099bb1d19cb2f5464bf06ad700efff7c" gracePeriod=30 Nov 03 23:07:40.817637 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:40.817601 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:40.956158 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:40.956133 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k" Nov 03 23:07:40.966797 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:40.966772 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:41.065379 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:41.065346 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b-metrics-certs\") pod \"3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b\" (UID: \"3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b\") " Nov 03 23:07:41.065550 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:41.065418 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b-maas-api-tls\") pod \"3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b\" (UID: \"3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b\") " Nov 03 23:07:41.065550 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:41.065454 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b-openshift-service-ca\") pod \"3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b\" (UID: \"3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b\") " Nov 03 23:07:41.065646 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:41.065565 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b-kube-api-access-64ntn\" (UniqueName: \"kubernetes.io/projected/3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b-kube-api-access-64ntn\") pod \"3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b\" (UID: \"3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b\") " Nov 03 23:07:41.065881 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:41.065833 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b-openshift-service-ca" pod "3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b" (UID: "3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:41.067683 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:41.067654 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b-maas-api-tls" pod "3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b" (UID: "3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:07:41.067788 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:41.067725 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b-metrics-certs" pod "3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b" (UID: "3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:07:41.067788 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:41.067763 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b-kube-api-access-64ntn" pod "3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b" (UID: "3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b"). InnerVolumeSpecName "kube-api-access-64ntn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:41.166603 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:41.166566 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64ntn\" (UniqueName: \"kubernetes.io/projected/3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b-kube-api-access-64ntn\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:41.166603 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:41.166592 5196 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b-metrics-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:41.166603 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:41.166603 5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b-maas-api-tls\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:41.166879 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:41.166611 5196 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b-openshift-service-ca\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:41.466827 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:41.466793 5196 generic.go:356] "Generic (PLEG): container finished" podID="3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b" containerID="cb823ccc60b573dfae944eef3c494c36099bb1d19cb2f5464bf06ad700efff7c" exitCode=0 Nov 03 23:07:41.467047 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:41.466848 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k" event={"ID":"3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b","Type":"ContainerDied","Data":"cb823ccc60b573dfae944eef3c494c36099bb1d19cb2f5464bf06ad700efff7c"} Nov 03 23:07:41.467047 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:41.466886 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k" Nov 03 23:07:41.467047 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:41.466909 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k" event={"ID":"3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b","Type":"ContainerDied","Data":"6669e7e8f6da3d9b687df28f3ddf35923c2b866b48f5e6caab105e963e84addb"} Nov 03 23:07:41.467047 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:41.466930 5196 scope.go:122] "RemoveContainer" containerID="cb823ccc60b573dfae944eef3c494c36099bb1d19cb2f5464bf06ad700efff7c" Nov 03 23:07:41.477001 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:41.476936 5196 scope.go:122] "RemoveContainer" containerID="cb823ccc60b573dfae944eef3c494c36099bb1d19cb2f5464bf06ad700efff7c" Nov 03 23:07:41.477224 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:07:41.477204 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb823ccc60b573dfae944eef3c494c36099bb1d19cb2f5464bf06ad700efff7c\": container with ID starting with cb823ccc60b573dfae944eef3c494c36099bb1d19cb2f5464bf06ad700efff7c not found: ID does not exist" containerID="cb823ccc60b573dfae944eef3c494c36099bb1d19cb2f5464bf06ad700efff7c" Nov 03 23:07:41.477272 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:41.477234 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb823ccc60b573dfae944eef3c494c36099bb1d19cb2f5464bf06ad700efff7c"} err="failed to get container status \"cb823ccc60b573dfae944eef3c494c36099bb1d19cb2f5464bf06ad700efff7c\": rpc error: code = NotFound desc = could not find container \"cb823ccc60b573dfae944eef3c494c36099bb1d19cb2f5464bf06ad700efff7c\": container with ID starting with cb823ccc60b573dfae944eef3c494c36099bb1d19cb2f5464bf06ad700efff7c not found: ID does not exist" Nov 03 23:07:41.633512 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:41.633473 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k"] Nov 03 23:07:41.637801 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:41.637775 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-433db8-cdbbb9bfd-ntr9k"] Nov 03 23:07:42.383449 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:42.383415 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:42.617130 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:42.617095 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:42.771728 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:42.771640 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:42.814334 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:42.814299 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b" path="/var/lib/kubelet/pods/3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b/volumes" Nov 03 23:07:43.345457 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:43.345421 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:43.594379 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:43.594346 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc"] Nov 03 23:07:43.747395 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:43.747307 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:47.602464 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:47.602428 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:49.893827 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:49.893787 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:50.039639 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:50.039612 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:50.195801 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:50.195703 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:50.471086 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:50.471008 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:51.021442 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:51.021407 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:51.709538 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:51.709503 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh"] Nov 03 23:07:51.709729 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:51.709704 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh" podUID="d6540706-d21a-4432-8f80-a6d427a13153" containerName="maas-api" containerID="cri-o://9a0cb4ce4f35449acecc7a17e5b9ae1b4d3990114336ce870f420f4e4d389dc2" gracePeriod=30 Nov 03 23:07:51.808194 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:51.808166 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:51.987722 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:51.987690 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh" Nov 03 23:07:52.075617 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:52.075589 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6540706-d21a-4432-8f80-a6d427a13153-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6540706-d21a-4432-8f80-a6d427a13153-maas-api-tls\") pod \"d6540706-d21a-4432-8f80-a6d427a13153\" (UID: \"d6540706-d21a-4432-8f80-a6d427a13153\") " Nov 03 23:07:52.076053 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:52.075653 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6540706-d21a-4432-8f80-a6d427a13153-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6540706-d21a-4432-8f80-a6d427a13153-openshift-service-ca\") pod \"d6540706-d21a-4432-8f80-a6d427a13153\" (UID: \"d6540706-d21a-4432-8f80-a6d427a13153\") " Nov 03 23:07:52.076053 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:52.075719 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6540706-d21a-4432-8f80-a6d427a13153-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6540706-d21a-4432-8f80-a6d427a13153-metrics-certs\") pod \"d6540706-d21a-4432-8f80-a6d427a13153\" (UID: \"d6540706-d21a-4432-8f80-a6d427a13153\") " Nov 03 23:07:52.076053 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:52.075756 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6540706-d21a-4432-8f80-a6d427a13153-kube-api-access-rmqbw\" (UniqueName: \"kubernetes.io/projected/d6540706-d21a-4432-8f80-a6d427a13153-kube-api-access-rmqbw\") pod \"d6540706-d21a-4432-8f80-a6d427a13153\" (UID: \"d6540706-d21a-4432-8f80-a6d427a13153\") " Nov 03 23:07:52.076053 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:52.076037 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6540706-d21a-4432-8f80-a6d427a13153-openshift-service-ca" pod "d6540706-d21a-4432-8f80-a6d427a13153" (UID: "d6540706-d21a-4432-8f80-a6d427a13153"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:52.076394 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:52.076138 5196 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6540706-d21a-4432-8f80-a6d427a13153-openshift-service-ca\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:52.077917 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:52.077894 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6540706-d21a-4432-8f80-a6d427a13153-metrics-certs" pod "d6540706-d21a-4432-8f80-a6d427a13153" (UID: "d6540706-d21a-4432-8f80-a6d427a13153"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:07:52.078114 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:52.078098 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6540706-d21a-4432-8f80-a6d427a13153-maas-api-tls" pod "d6540706-d21a-4432-8f80-a6d427a13153" (UID: "d6540706-d21a-4432-8f80-a6d427a13153"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:07:52.078220 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:52.078198 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6540706-d21a-4432-8f80-a6d427a13153-kube-api-access-rmqbw" pod "d6540706-d21a-4432-8f80-a6d427a13153" (UID: "d6540706-d21a-4432-8f80-a6d427a13153"). InnerVolumeSpecName "kube-api-access-rmqbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:52.176768 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:52.176734 5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6540706-d21a-4432-8f80-a6d427a13153-maas-api-tls\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:52.176768 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:52.176761 5196 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6540706-d21a-4432-8f80-a6d427a13153-metrics-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:52.176768 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:52.176772 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmqbw\" (UniqueName: \"kubernetes.io/projected/d6540706-d21a-4432-8f80-a6d427a13153-kube-api-access-rmqbw\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:52.533769 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:52.533729 5196 generic.go:356] "Generic (PLEG): container finished" podID="d6540706-d21a-4432-8f80-a6d427a13153" containerID="9a0cb4ce4f35449acecc7a17e5b9ae1b4d3990114336ce870f420f4e4d389dc2" exitCode=0 Nov 03 23:07:52.533965 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:52.533799 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh" Nov 03 23:07:52.533965 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:52.533811 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh" event={"ID":"d6540706-d21a-4432-8f80-a6d427a13153","Type":"ContainerDied","Data":"9a0cb4ce4f35449acecc7a17e5b9ae1b4d3990114336ce870f420f4e4d389dc2"} Nov 03 23:07:52.533965 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:52.533915 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh" event={"ID":"d6540706-d21a-4432-8f80-a6d427a13153","Type":"ContainerDied","Data":"f5e2eff4174da4a6c9ea3603e3b5d89f8649ba52216910e51873f9e9c1f59228"} Nov 03 23:07:52.533965 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:52.533948 5196 scope.go:122] "RemoveContainer" containerID="9a0cb4ce4f35449acecc7a17e5b9ae1b4d3990114336ce870f420f4e4d389dc2" Nov 03 23:07:52.544080 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:52.544064 5196 scope.go:122] "RemoveContainer" containerID="9a0cb4ce4f35449acecc7a17e5b9ae1b4d3990114336ce870f420f4e4d389dc2" Nov 03 23:07:52.544357 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:07:52.544338 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a0cb4ce4f35449acecc7a17e5b9ae1b4d3990114336ce870f420f4e4d389dc2\": container with ID starting with 9a0cb4ce4f35449acecc7a17e5b9ae1b4d3990114336ce870f420f4e4d389dc2 not found: ID does not exist" containerID="9a0cb4ce4f35449acecc7a17e5b9ae1b4d3990114336ce870f420f4e4d389dc2" Nov 03 23:07:52.544426 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:52.544366 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a0cb4ce4f35449acecc7a17e5b9ae1b4d3990114336ce870f420f4e4d389dc2"} err="failed to get container status \"9a0cb4ce4f35449acecc7a17e5b9ae1b4d3990114336ce870f420f4e4d389dc2\": rpc error: code = NotFound desc = could not find container \"9a0cb4ce4f35449acecc7a17e5b9ae1b4d3990114336ce870f420f4e4d389dc2\": container with ID starting with 9a0cb4ce4f35449acecc7a17e5b9ae1b4d3990114336ce870f420f4e4d389dc2 not found: ID does not exist" Nov 03 23:07:52.703651 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:52.703608 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh"] Nov 03 23:07:52.706768 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:52.706743 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cfe1bd-84f794b6b7-rhmvh"] Nov 03 23:07:52.814044 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:52.813971 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6540706-d21a-4432-8f80-a6d427a13153" path="/var/lib/kubelet/pods/d6540706-d21a-4432-8f80-a6d427a13153/volumes" Nov 03 23:07:53.414515 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:53.414483 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:53.678105 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:53.678023 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:53.832713 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:53.832679 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:54.347682 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:54.347654 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:55.919578 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:07:55.919543 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:07:55.972779 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:55.972751 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd"] Nov 03 23:07:56.023546 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:56.023511 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p"] Nov 03 23:07:56.119069 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:56.119031 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:58.615436 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:58.615400 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" podUID="34e9cf3a-3456-4835-8933-102ce735cccc" containerName="main" containerID="cri-o://e05cefab23c6656ebcf697f93755ee7a8e941637acf10f2594a65d1f99625170" gracePeriod=45 Nov 03 23:07:58.870148 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:58.870092 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" Nov 03 23:07:58.955175 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:58.955141 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:07:59.045648 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:59.045612 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34e9cf3a-3456-4835-8933-102ce735cccc-tls-certs\" (UniqueName: \"kubernetes.io/secret/34e9cf3a-3456-4835-8933-102ce735cccc-tls-certs\") pod \"34e9cf3a-3456-4835-8933-102ce735cccc\" (UID: \"34e9cf3a-3456-4835-8933-102ce735cccc\") " Nov 03 23:07:59.045820 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:59.045702 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34e9cf3a-3456-4835-8933-102ce735cccc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/34e9cf3a-3456-4835-8933-102ce735cccc-dshm\") pod \"34e9cf3a-3456-4835-8933-102ce735cccc\" (UID: \"34e9cf3a-3456-4835-8933-102ce735cccc\") " Nov 03 23:07:59.045820 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:59.045738 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34e9cf3a-3456-4835-8933-102ce735cccc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34e9cf3a-3456-4835-8933-102ce735cccc-tmp-dir\") pod \"34e9cf3a-3456-4835-8933-102ce735cccc\" (UID: \"34e9cf3a-3456-4835-8933-102ce735cccc\") " Nov 03 23:07:59.045820 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:59.045779 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34e9cf3a-3456-4835-8933-102ce735cccc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34e9cf3a-3456-4835-8933-102ce735cccc-model-cache\") pod \"34e9cf3a-3456-4835-8933-102ce735cccc\" (UID: \"34e9cf3a-3456-4835-8933-102ce735cccc\") " Nov 03 23:07:59.046024 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:59.045851 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34e9cf3a-3456-4835-8933-102ce735cccc-kube-api-access-kmwsq\" (UniqueName: \"kubernetes.io/projected/34e9cf3a-3456-4835-8933-102ce735cccc-kube-api-access-kmwsq\") pod \"34e9cf3a-3456-4835-8933-102ce735cccc\" (UID: \"34e9cf3a-3456-4835-8933-102ce735cccc\") " Nov 03 23:07:59.046024 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:59.045900 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34e9cf3a-3456-4835-8933-102ce735cccc-home\" (UniqueName: \"kubernetes.io/empty-dir/34e9cf3a-3456-4835-8933-102ce735cccc-home\") pod \"34e9cf3a-3456-4835-8933-102ce735cccc\" (UID: \"34e9cf3a-3456-4835-8933-102ce735cccc\") " Nov 03 23:07:59.046160 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:59.046136 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e9cf3a-3456-4835-8933-102ce735cccc-model-cache" pod "34e9cf3a-3456-4835-8933-102ce735cccc" (UID: "34e9cf3a-3456-4835-8933-102ce735cccc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:59.046344 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:59.046299 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e9cf3a-3456-4835-8933-102ce735cccc-tmp-dir" pod "34e9cf3a-3456-4835-8933-102ce735cccc" (UID: "34e9cf3a-3456-4835-8933-102ce735cccc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:59.046438 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:59.046366 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e9cf3a-3456-4835-8933-102ce735cccc-home" pod "34e9cf3a-3456-4835-8933-102ce735cccc" (UID: "34e9cf3a-3456-4835-8933-102ce735cccc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:59.046658 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:59.046632 5196 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34e9cf3a-3456-4835-8933-102ce735cccc-tmp-dir\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:59.046658 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:59.046659 5196 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34e9cf3a-3456-4835-8933-102ce735cccc-model-cache\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:59.046851 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:59.046674 5196 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34e9cf3a-3456-4835-8933-102ce735cccc-home\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:59.048596 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:59.048554 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34e9cf3a-3456-4835-8933-102ce735cccc-tls-certs" pod "34e9cf3a-3456-4835-8933-102ce735cccc" (UID: "34e9cf3a-3456-4835-8933-102ce735cccc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:07:59.048740 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:59.048677 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e9cf3a-3456-4835-8933-102ce735cccc-dshm" pod "34e9cf3a-3456-4835-8933-102ce735cccc" (UID: "34e9cf3a-3456-4835-8933-102ce735cccc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:59.048813 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:59.048773 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34e9cf3a-3456-4835-8933-102ce735cccc-kube-api-access-kmwsq" pod "34e9cf3a-3456-4835-8933-102ce735cccc" (UID: "34e9cf3a-3456-4835-8933-102ce735cccc"). InnerVolumeSpecName "kube-api-access-kmwsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:59.148299 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:59.148224 5196 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34e9cf3a-3456-4835-8933-102ce735cccc-tls-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:59.148299 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:59.148251 5196 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34e9cf3a-3456-4835-8933-102ce735cccc-dshm\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:59.148299 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:59.148261 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmwsq\" (UniqueName: \"kubernetes.io/projected/34e9cf3a-3456-4835-8933-102ce735cccc-kube-api-access-kmwsq\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:07:59.576795 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:59.576665 5196 generic.go:356] "Generic (PLEG): container finished" podID="34e9cf3a-3456-4835-8933-102ce735cccc" containerID="e05cefab23c6656ebcf697f93755ee7a8e941637acf10f2594a65d1f99625170" exitCode=0 Nov 03 23:07:59.576795 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:59.576726 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" Nov 03 23:07:59.576795 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:59.576746 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" event={"ID":"34e9cf3a-3456-4835-8933-102ce735cccc","Type":"ContainerDied","Data":"e05cefab23c6656ebcf697f93755ee7a8e941637acf10f2594a65d1f99625170"} Nov 03 23:07:59.576795 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:59.576786 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc" event={"ID":"34e9cf3a-3456-4835-8933-102ce735cccc","Type":"ContainerDied","Data":"c8695b22ecdf3d1f97569320db04bbb22add9146bc97aa6f6000a4d2908e2b02"} Nov 03 23:07:59.577124 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:59.576805 5196 scope.go:122] "RemoveContainer" containerID="e05cefab23c6656ebcf697f93755ee7a8e941637acf10f2594a65d1f99625170" Nov 03 23:07:59.586449 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:59.586430 5196 scope.go:122] "RemoveContainer" containerID="e05cefab23c6656ebcf697f93755ee7a8e941637acf10f2594a65d1f99625170" Nov 03 23:07:59.586756 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:07:59.586737 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e05cefab23c6656ebcf697f93755ee7a8e941637acf10f2594a65d1f99625170\": container with ID starting with e05cefab23c6656ebcf697f93755ee7a8e941637acf10f2594a65d1f99625170 not found: ID does not exist" containerID="e05cefab23c6656ebcf697f93755ee7a8e941637acf10f2594a65d1f99625170" Nov 03 23:07:59.586796 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:59.586766 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e05cefab23c6656ebcf697f93755ee7a8e941637acf10f2594a65d1f99625170"} err="failed to get container status \"e05cefab23c6656ebcf697f93755ee7a8e941637acf10f2594a65d1f99625170\": rpc error: code = NotFound desc = could not find container \"e05cefab23c6656ebcf697f93755ee7a8e941637acf10f2594a65d1f99625170\": container with ID starting with e05cefab23c6656ebcf697f93755ee7a8e941637acf10f2594a65d1f99625170 not found: ID does not exist" Nov 03 23:07:59.742988 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:59.742953 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc"] Nov 03 23:07:59.793700 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:07:59.793668 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-164c36/premium-w2-164c36-kserve-74c7c9fdcc-8hlrc"] Nov 03 23:08:00.440358 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:00.440322 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:00.590707 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:00.590675 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:00.814460 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:00.814384 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34e9cf3a-3456-4835-8933-102ce735cccc" path="/var/lib/kubelet/pods/34e9cf3a-3456-4835-8933-102ce735cccc/volumes" Nov 03 23:08:00.916298 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:00.916255 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:01.254533 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:01.254499 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:03.183920 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:03.183893 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:03.330514 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:03.330485 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:03.898572 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:03.898526 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:04.339439 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:04.339416 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx"] Nov 03 23:08:04.339958 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:04.339617 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" podUID="e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf" containerName="istio-proxy" containerID="cri-o://e9af8ca6fbec64658df6f30faf22ec5b3b7945328e10b5c2cb0d9c1d20c97a66" gracePeriod=30 Nov 03 23:08:06.309174 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:06.309122 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:09.396799 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:09.396767 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:09.558589 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:09.558542 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:09.586735 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:09.586714 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" Nov 03 23:08:09.631394 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:09.631311 5196 generic.go:356] "Generic (PLEG): container finished" podID="e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf" containerID="e9af8ca6fbec64658df6f30faf22ec5b3b7945328e10b5c2cb0d9c1d20c97a66" exitCode=0 Nov 03 23:08:09.631394 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:09.631358 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" Nov 03 23:08:09.631563 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:09.631396 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" event={"ID":"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf","Type":"ContainerDied","Data":"e9af8ca6fbec64658df6f30faf22ec5b3b7945328e10b5c2cb0d9c1d20c97a66"} Nov 03 23:08:09.631563 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:09.631451 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx" event={"ID":"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf","Type":"ContainerDied","Data":"d80e477abc32c79038988bf21470c8334287e28221ed11c7c55770e14dfc55d2"} Nov 03 23:08:09.631563 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:09.631474 5196 scope.go:122] "RemoveContainer" containerID="e9af8ca6fbec64658df6f30faf22ec5b3b7945328e10b5c2cb0d9c1d20c97a66" Nov 03 23:08:09.640817 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:09.640797 5196 scope.go:122] "RemoveContainer" containerID="e9af8ca6fbec64658df6f30faf22ec5b3b7945328e10b5c2cb0d9c1d20c97a66" Nov 03 23:08:09.641132 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:08:09.641109 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9af8ca6fbec64658df6f30faf22ec5b3b7945328e10b5c2cb0d9c1d20c97a66\": container with ID starting with e9af8ca6fbec64658df6f30faf22ec5b3b7945328e10b5c2cb0d9c1d20c97a66 not found: ID does not exist" containerID="e9af8ca6fbec64658df6f30faf22ec5b3b7945328e10b5c2cb0d9c1d20c97a66" Nov 03 23:08:09.641209 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:09.641144 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9af8ca6fbec64658df6f30faf22ec5b3b7945328e10b5c2cb0d9c1d20c97a66"} err="failed to get container status \"e9af8ca6fbec64658df6f30faf22ec5b3b7945328e10b5c2cb0d9c1d20c97a66\": rpc error: code = NotFound desc = could not find container \"e9af8ca6fbec64658df6f30faf22ec5b3b7945328e10b5c2cb0d9c1d20c97a66\": container with ID starting with e9af8ca6fbec64658df6f30faf22ec5b3b7945328e10b5c2cb0d9c1d20c97a66 not found: ID does not exist" Nov 03 23:08:09.644417 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:09.644394 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-credential-socket\") pod \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\" (UID: \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\") " Nov 03 23:08:09.644480 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:09.644450 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-kube-api-access-dpt2g\" (UniqueName: \"kubernetes.io/projected/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-kube-api-access-dpt2g\") pod \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\" (UID: \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\") " Nov 03 23:08:09.644517 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:09.644495 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-istiod-ca-cert\") pod \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\" (UID: \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\") " Nov 03 23:08:09.644556 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:09.644538 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-istio-envoy\") pod \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\" (UID: \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\") " Nov 03 23:08:09.644595 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:09.644572 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-istio-data\") pod \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\" (UID: \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\") " Nov 03 23:08:09.644661 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:09.644639 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-istio-podinfo\") pod \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\" (UID: \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\") " Nov 03 23:08:09.644714 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:09.644667 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-credential-socket" pod "e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf" (UID: "e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:09.644714 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:09.644700 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-istio-token\" (UniqueName: \"kubernetes.io/projected/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-istio-token\") pod \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\" (UID: \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\") " Nov 03 23:08:09.644824 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:09.644762 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-workload-socket\") pod \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\" (UID: \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\") " Nov 03 23:08:09.644824 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:09.644803 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-workload-certs\") pod \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\" (UID: \"e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf\") " Nov 03 23:08:09.644949 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:09.644901 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-istio-data" pod "e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf" (UID: "e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:09.645110 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:09.645003 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-istiod-ca-cert" pod "e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf" (UID: "e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:08:09.645110 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:09.645086 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-workload-socket" pod "e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf" (UID: "e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:09.645267 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:09.645235 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-workload-certs" pod "e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf" (UID: "e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:09.645513 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:09.645495 5196 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-istio-data\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:09.645600 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:09.645517 5196 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-workload-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:09.645600 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:09.645531 5196 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-workload-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:09.645600 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:09.645547 5196 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-credential-socket\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:09.645600 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:09.645561 5196 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-istiod-ca-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:09.647117 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:09.647093 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-istio-envoy" pod "e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf" (UID: "e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:09.647319 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:09.647295 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-istio-token" pod "e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf" (UID: "e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:08:09.647392 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:09.647350 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-istio-podinfo" pod "e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf" (UID: "e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:08:09.647392 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:09.647361 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-kube-api-access-dpt2g" pod "e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf" (UID: "e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf"). InnerVolumeSpecName "kube-api-access-dpt2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:08:09.746703 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:09.746675 5196 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-istio-envoy\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:09.746703 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:09.746701 5196 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-istio-podinfo\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:09.746703 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:09.746710 5196 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-istio-token\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:09.746988 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:09.746721 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpt2g\" (UniqueName: \"kubernetes.io/projected/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf-kube-api-access-dpt2g\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:10.059493 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:10.059461 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx"] Nov 03 23:08:10.063915 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:10.063887 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kpmpx"] Nov 03 23:08:10.813653 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:10.813623 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf" path="/var/lib/kubelet/pods/e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf/volumes" Nov 03 23:08:10.993207 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:10.993164 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" podUID="9d8b4fed-b4b0-49ce-9238-e23175e90b70" containerName="main" containerID="cri-o://a0a53dce8b1236693f2cd3a6ca67797ad4fa6a5b7b6d183257902d4ae4935cd5" gracePeriod=45 Nov 03 23:08:11.043293 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.043255 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" podUID="26fa51c2-0f2e-482f-9c67-0ea7924d7b3c" containerName="main" containerID="cri-o://04a6c30c79f2ec1112aa05316ce756ef91d51fa48ad93415c09b03f8c1f0778a" gracePeriod=45 Nov 03 23:08:11.289425 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.289402 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" Nov 03 23:08:11.313426 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.313402 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" Nov 03 23:08:11.360968 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.360944 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d8b4fed-b4b0-49ce-9238-e23175e90b70-kube-api-access-9rl4d\" (UniqueName: \"kubernetes.io/projected/9d8b4fed-b4b0-49ce-9238-e23175e90b70-kube-api-access-9rl4d\") pod \"9d8b4fed-b4b0-49ce-9238-e23175e90b70\" (UID: \"9d8b4fed-b4b0-49ce-9238-e23175e90b70\") " Nov 03 23:08:11.361120 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.360985 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-dshm\") pod \"26fa51c2-0f2e-482f-9c67-0ea7924d7b3c\" (UID: \"26fa51c2-0f2e-482f-9c67-0ea7924d7b3c\") " Nov 03 23:08:11.361120 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.361018 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d8b4fed-b4b0-49ce-9238-e23175e90b70-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d8b4fed-b4b0-49ce-9238-e23175e90b70-dshm\") pod \"9d8b4fed-b4b0-49ce-9238-e23175e90b70\" (UID: \"9d8b4fed-b4b0-49ce-9238-e23175e90b70\") " Nov 03 23:08:11.361120 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.361069 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d8b4fed-b4b0-49ce-9238-e23175e90b70-tls-certs\" (UniqueName: \"kubernetes.io/secret/9d8b4fed-b4b0-49ce-9238-e23175e90b70-tls-certs\") pod \"9d8b4fed-b4b0-49ce-9238-e23175e90b70\" (UID: \"9d8b4fed-b4b0-49ce-9238-e23175e90b70\") " Nov 03 23:08:11.361120 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.361099 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-home\" (UniqueName: \"kubernetes.io/empty-dir/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-home\") pod \"26fa51c2-0f2e-482f-9c67-0ea7924d7b3c\" (UID: \"26fa51c2-0f2e-482f-9c67-0ea7924d7b3c\") " Nov 03 23:08:11.361359 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.361136 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-tmp-dir\") pod \"26fa51c2-0f2e-482f-9c67-0ea7924d7b3c\" (UID: \"26fa51c2-0f2e-482f-9c67-0ea7924d7b3c\") " Nov 03 23:08:11.361359 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.361164 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d8b4fed-b4b0-49ce-9238-e23175e90b70-home\" (UniqueName: \"kubernetes.io/empty-dir/9d8b4fed-b4b0-49ce-9238-e23175e90b70-home\") pod \"9d8b4fed-b4b0-49ce-9238-e23175e90b70\" (UID: \"9d8b4fed-b4b0-49ce-9238-e23175e90b70\") " Nov 03 23:08:11.361359 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.361193 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-kube-api-access-fbfnl\" (UniqueName: \"kubernetes.io/projected/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-kube-api-access-fbfnl\") pod \"26fa51c2-0f2e-482f-9c67-0ea7924d7b3c\" (UID: \"26fa51c2-0f2e-482f-9c67-0ea7924d7b3c\") " Nov 03 23:08:11.361359 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.361243 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d8b4fed-b4b0-49ce-9238-e23175e90b70-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d8b4fed-b4b0-49ce-9238-e23175e90b70-tmp-dir\") pod \"9d8b4fed-b4b0-49ce-9238-e23175e90b70\" (UID: \"9d8b4fed-b4b0-49ce-9238-e23175e90b70\") " Nov 03 23:08:11.361359 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.361293 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-tls-certs\" (UniqueName: \"kubernetes.io/secret/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-tls-certs\") pod \"26fa51c2-0f2e-482f-9c67-0ea7924d7b3c\" (UID: \"26fa51c2-0f2e-482f-9c67-0ea7924d7b3c\") " Nov 03 23:08:11.361604 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.361364 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-model-cache\") pod \"26fa51c2-0f2e-482f-9c67-0ea7924d7b3c\" (UID: \"26fa51c2-0f2e-482f-9c67-0ea7924d7b3c\") " Nov 03 23:08:11.361604 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.361420 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d8b4fed-b4b0-49ce-9238-e23175e90b70-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d8b4fed-b4b0-49ce-9238-e23175e90b70-model-cache\") pod \"9d8b4fed-b4b0-49ce-9238-e23175e90b70\" (UID: \"9d8b4fed-b4b0-49ce-9238-e23175e90b70\") " Nov 03 23:08:11.362206 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.361921 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d8b4fed-b4b0-49ce-9238-e23175e90b70-model-cache" pod "9d8b4fed-b4b0-49ce-9238-e23175e90b70" (UID: "9d8b4fed-b4b0-49ce-9238-e23175e90b70"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:11.362206 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.362035 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-home" pod "26fa51c2-0f2e-482f-9c67-0ea7924d7b3c" (UID: "26fa51c2-0f2e-482f-9c67-0ea7924d7b3c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:11.362460 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.362397 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d8b4fed-b4b0-49ce-9238-e23175e90b70-home" pod "9d8b4fed-b4b0-49ce-9238-e23175e90b70" (UID: "9d8b4fed-b4b0-49ce-9238-e23175e90b70"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:11.362709 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.362580 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-tmp-dir" pod "26fa51c2-0f2e-482f-9c67-0ea7924d7b3c" (UID: "26fa51c2-0f2e-482f-9c67-0ea7924d7b3c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:11.362820 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.362763 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-model-cache" pod "26fa51c2-0f2e-482f-9c67-0ea7924d7b3c" (UID: "26fa51c2-0f2e-482f-9c67-0ea7924d7b3c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:11.362982 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.362959 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d8b4fed-b4b0-49ce-9238-e23175e90b70-tmp-dir" pod "9d8b4fed-b4b0-49ce-9238-e23175e90b70" (UID: "9d8b4fed-b4b0-49ce-9238-e23175e90b70"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:11.364160 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.364095 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d8b4fed-b4b0-49ce-9238-e23175e90b70-dshm" pod "9d8b4fed-b4b0-49ce-9238-e23175e90b70" (UID: "9d8b4fed-b4b0-49ce-9238-e23175e90b70"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:11.364297 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.364275 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d8b4fed-b4b0-49ce-9238-e23175e90b70-kube-api-access-9rl4d" pod "9d8b4fed-b4b0-49ce-9238-e23175e90b70" (UID: "9d8b4fed-b4b0-49ce-9238-e23175e90b70"). InnerVolumeSpecName "kube-api-access-9rl4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:08:11.364535 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.364504 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-tls-certs" pod "26fa51c2-0f2e-482f-9c67-0ea7924d7b3c" (UID: "26fa51c2-0f2e-482f-9c67-0ea7924d7b3c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:08:11.364815 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.364801 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-kube-api-access-fbfnl" pod "26fa51c2-0f2e-482f-9c67-0ea7924d7b3c" (UID: "26fa51c2-0f2e-482f-9c67-0ea7924d7b3c"). InnerVolumeSpecName "kube-api-access-fbfnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:08:11.365210 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.365186 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-dshm" pod "26fa51c2-0f2e-482f-9c67-0ea7924d7b3c" (UID: "26fa51c2-0f2e-482f-9c67-0ea7924d7b3c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:08:11.365590 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.365573 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d8b4fed-b4b0-49ce-9238-e23175e90b70-tls-certs" pod "9d8b4fed-b4b0-49ce-9238-e23175e90b70" (UID: "9d8b4fed-b4b0-49ce-9238-e23175e90b70"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:08:11.462741 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.462657 5196 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-tls-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:11.462741 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.462686 5196 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-model-cache\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:11.462741 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.462700 5196 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d8b4fed-b4b0-49ce-9238-e23175e90b70-model-cache\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:11.462741 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.462715 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rl4d\" (UniqueName: \"kubernetes.io/projected/9d8b4fed-b4b0-49ce-9238-e23175e90b70-kube-api-access-9rl4d\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:11.462741 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.462729 5196 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-dshm\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:11.462741 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.462740 5196 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d8b4fed-b4b0-49ce-9238-e23175e90b70-dshm\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:11.463097 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.462751 5196 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d8b4fed-b4b0-49ce-9238-e23175e90b70-tls-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:11.463097 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.462762 5196 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-home\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:11.463097 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.462773 5196 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-tmp-dir\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:11.463097 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.462783 5196 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d8b4fed-b4b0-49ce-9238-e23175e90b70-home\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:11.463097 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.462793 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbfnl\" (UniqueName: \"kubernetes.io/projected/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c-kube-api-access-fbfnl\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:11.463097 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.462804 5196 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d8b4fed-b4b0-49ce-9238-e23175e90b70-tmp-dir\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:11.644087 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.644052 5196 generic.go:356] "Generic (PLEG): container finished" podID="9d8b4fed-b4b0-49ce-9238-e23175e90b70" containerID="a0a53dce8b1236693f2cd3a6ca67797ad4fa6a5b7b6d183257902d4ae4935cd5" exitCode=0 Nov 03 23:08:11.644268 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.644107 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" Nov 03 23:08:11.644268 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.644138 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" event={"ID":"9d8b4fed-b4b0-49ce-9238-e23175e90b70","Type":"ContainerDied","Data":"a0a53dce8b1236693f2cd3a6ca67797ad4fa6a5b7b6d183257902d4ae4935cd5"} Nov 03 23:08:11.644268 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.644183 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd" event={"ID":"9d8b4fed-b4b0-49ce-9238-e23175e90b70","Type":"ContainerDied","Data":"c3fef13c29f5fbf3a1cd3448be2d976639383e809144bed468749e30b97b05a7"} Nov 03 23:08:11.644268 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.644208 5196 scope.go:122] "RemoveContainer" containerID="a0a53dce8b1236693f2cd3a6ca67797ad4fa6a5b7b6d183257902d4ae4935cd5" Nov 03 23:08:11.645640 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.645606 5196 generic.go:356] "Generic (PLEG): container finished" podID="26fa51c2-0f2e-482f-9c67-0ea7924d7b3c" containerID="04a6c30c79f2ec1112aa05316ce756ef91d51fa48ad93415c09b03f8c1f0778a" exitCode=0 Nov 03 23:08:11.645640 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.645647 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" event={"ID":"26fa51c2-0f2e-482f-9c67-0ea7924d7b3c","Type":"ContainerDied","Data":"04a6c30c79f2ec1112aa05316ce756ef91d51fa48ad93415c09b03f8c1f0778a"} Nov 03 23:08:11.645640 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.645688 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" event={"ID":"26fa51c2-0f2e-482f-9c67-0ea7924d7b3c","Type":"ContainerDied","Data":"6bf83f66d5585d5ec2677d8911e1b1f08b4a5fe6fdc57ffa0d11661431e5f371"} Nov 03 23:08:11.645640 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.645656 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" Nov 03 23:08:11.653471 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.653452 5196 scope.go:122] "RemoveContainer" containerID="a0a53dce8b1236693f2cd3a6ca67797ad4fa6a5b7b6d183257902d4ae4935cd5" Nov 03 23:08:11.653734 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:08:11.653715 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0a53dce8b1236693f2cd3a6ca67797ad4fa6a5b7b6d183257902d4ae4935cd5\": container with ID starting with a0a53dce8b1236693f2cd3a6ca67797ad4fa6a5b7b6d183257902d4ae4935cd5 not found: ID does not exist" containerID="a0a53dce8b1236693f2cd3a6ca67797ad4fa6a5b7b6d183257902d4ae4935cd5" Nov 03 23:08:11.653821 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.653743 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0a53dce8b1236693f2cd3a6ca67797ad4fa6a5b7b6d183257902d4ae4935cd5"} err="failed to get container status \"a0a53dce8b1236693f2cd3a6ca67797ad4fa6a5b7b6d183257902d4ae4935cd5\": rpc error: code = NotFound desc = could not find container \"a0a53dce8b1236693f2cd3a6ca67797ad4fa6a5b7b6d183257902d4ae4935cd5\": container with ID starting with a0a53dce8b1236693f2cd3a6ca67797ad4fa6a5b7b6d183257902d4ae4935cd5 not found: ID does not exist" Nov 03 23:08:11.653821 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.653766 5196 scope.go:122] "RemoveContainer" containerID="04a6c30c79f2ec1112aa05316ce756ef91d51fa48ad93415c09b03f8c1f0778a" Nov 03 23:08:11.661944 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.661928 5196 scope.go:122] "RemoveContainer" containerID="04a6c30c79f2ec1112aa05316ce756ef91d51fa48ad93415c09b03f8c1f0778a" Nov 03 23:08:11.662222 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:08:11.662201 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04a6c30c79f2ec1112aa05316ce756ef91d51fa48ad93415c09b03f8c1f0778a\": container with ID starting with 04a6c30c79f2ec1112aa05316ce756ef91d51fa48ad93415c09b03f8c1f0778a not found: ID does not exist" containerID="04a6c30c79f2ec1112aa05316ce756ef91d51fa48ad93415c09b03f8c1f0778a" Nov 03 23:08:11.662271 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.662230 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04a6c30c79f2ec1112aa05316ce756ef91d51fa48ad93415c09b03f8c1f0778a"} err="failed to get container status \"04a6c30c79f2ec1112aa05316ce756ef91d51fa48ad93415c09b03f8c1f0778a\": rpc error: code = NotFound desc = could not find container \"04a6c30c79f2ec1112aa05316ce756ef91d51fa48ad93415c09b03f8c1f0778a\": container with ID starting with 04a6c30c79f2ec1112aa05316ce756ef91d51fa48ad93415c09b03f8c1f0778a not found: ID does not exist" Nov 03 23:08:11.810289 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.810212 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd"] Nov 03 23:08:11.814425 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.814399 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-1d31f8/distinct-2-w3-1d31f8-kserve-5c44c87f8-bx5dd"] Nov 03 23:08:11.924706 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.924680 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p"] Nov 03 23:08:11.929293 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:11.929270 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p"] Nov 03 23:08:12.028346 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:12.028317 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:12.282957 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:12.282917 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1d31f8/distinct-w3-1d31f8-kserve-6b64664bd9-7lr2p" podUID="26fa51c2-0f2e-482f-9c67-0ea7924d7b3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 03 23:08:12.814138 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:12.814098 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26fa51c2-0f2e-482f-9c67-0ea7924d7b3c" path="/var/lib/kubelet/pods/26fa51c2-0f2e-482f-9c67-0ea7924d7b3c/volumes" Nov 03 23:08:12.814730 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:12.814708 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d8b4fed-b4b0-49ce-9238-e23175e90b70" path="/var/lib/kubelet/pods/9d8b4fed-b4b0-49ce-9238-e23175e90b70/volumes" Nov 03 23:08:13.726268 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:13.726238 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:13.874715 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:13.874687 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:17.387026 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:17.386997 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:21.276797 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:21.276724 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:21.419687 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:21.419657 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:21.560982 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:21.560901 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:24.084479 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:24.084436 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm"] Nov 03 23:08:24.084998 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:24.084762 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm" podUID="3692067f-4a4d-4bd0-bc74-8957aaf8531c" containerName="maas-api" containerID="cri-o://782dd45c8c888507deaaabb5b572ef215ea0d5c1f7e75ba56ceafe1226840954" gracePeriod=30 Nov 03 23:08:24.171398 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:24.171362 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm" podUID="3692067f-4a4d-4bd0-bc74-8957aaf8531c" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.46:8443/health\": dial tcp 10.133.0.46:8443: connect: connection refused" Nov 03 23:08:24.184326 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:24.184292 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:24.383113 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:24.383088 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm" Nov 03 23:08:24.481507 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:24.481473 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3692067f-4a4d-4bd0-bc74-8957aaf8531c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3692067f-4a4d-4bd0-bc74-8957aaf8531c-metrics-certs\") pod \"3692067f-4a4d-4bd0-bc74-8957aaf8531c\" (UID: \"3692067f-4a4d-4bd0-bc74-8957aaf8531c\") " Nov 03 23:08:24.481710 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:24.481521 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3692067f-4a4d-4bd0-bc74-8957aaf8531c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3692067f-4a4d-4bd0-bc74-8957aaf8531c-maas-api-tls\") pod \"3692067f-4a4d-4bd0-bc74-8957aaf8531c\" (UID: \"3692067f-4a4d-4bd0-bc74-8957aaf8531c\") " Nov 03 23:08:24.481710 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:24.481632 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3692067f-4a4d-4bd0-bc74-8957aaf8531c-kube-api-access-xxzrk\" (UniqueName: \"kubernetes.io/projected/3692067f-4a4d-4bd0-bc74-8957aaf8531c-kube-api-access-xxzrk\") pod \"3692067f-4a4d-4bd0-bc74-8957aaf8531c\" (UID: \"3692067f-4a4d-4bd0-bc74-8957aaf8531c\") " Nov 03 23:08:24.481710 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:24.481681 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3692067f-4a4d-4bd0-bc74-8957aaf8531c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3692067f-4a4d-4bd0-bc74-8957aaf8531c-openshift-service-ca\") pod \"3692067f-4a4d-4bd0-bc74-8957aaf8531c\" (UID: \"3692067f-4a4d-4bd0-bc74-8957aaf8531c\") " Nov 03 23:08:24.482086 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:24.482049 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3692067f-4a4d-4bd0-bc74-8957aaf8531c-openshift-service-ca" pod "3692067f-4a4d-4bd0-bc74-8957aaf8531c" (UID: "3692067f-4a4d-4bd0-bc74-8957aaf8531c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:08:24.483911 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:24.483870 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3692067f-4a4d-4bd0-bc74-8957aaf8531c-kube-api-access-xxzrk" pod "3692067f-4a4d-4bd0-bc74-8957aaf8531c" (UID: "3692067f-4a4d-4bd0-bc74-8957aaf8531c"). InnerVolumeSpecName "kube-api-access-xxzrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:08:24.484026 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:24.483910 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3692067f-4a4d-4bd0-bc74-8957aaf8531c-metrics-certs" pod "3692067f-4a4d-4bd0-bc74-8957aaf8531c" (UID: "3692067f-4a4d-4bd0-bc74-8957aaf8531c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:08:24.484142 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:24.484119 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3692067f-4a4d-4bd0-bc74-8957aaf8531c-maas-api-tls" pod "3692067f-4a4d-4bd0-bc74-8957aaf8531c" (UID: "3692067f-4a4d-4bd0-bc74-8957aaf8531c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:08:24.582682 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:24.582642 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxzrk\" (UniqueName: \"kubernetes.io/projected/3692067f-4a4d-4bd0-bc74-8957aaf8531c-kube-api-access-xxzrk\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:24.582682 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:24.582676 5196 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3692067f-4a4d-4bd0-bc74-8957aaf8531c-openshift-service-ca\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:24.582682 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:24.582687 5196 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3692067f-4a4d-4bd0-bc74-8957aaf8531c-metrics-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:24.582966 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:24.582696 5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3692067f-4a4d-4bd0-bc74-8957aaf8531c-maas-api-tls\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:24.721510 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:24.721417 5196 generic.go:356] "Generic (PLEG): container finished" podID="3692067f-4a4d-4bd0-bc74-8957aaf8531c" containerID="782dd45c8c888507deaaabb5b572ef215ea0d5c1f7e75ba56ceafe1226840954" exitCode=0 Nov 03 23:08:24.721510 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:24.721466 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm" event={"ID":"3692067f-4a4d-4bd0-bc74-8957aaf8531c","Type":"ContainerDied","Data":"782dd45c8c888507deaaabb5b572ef215ea0d5c1f7e75ba56ceafe1226840954"} Nov 03 23:08:24.721510 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:24.721469 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm" Nov 03 23:08:24.721510 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:24.721509 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm" event={"ID":"3692067f-4a4d-4bd0-bc74-8957aaf8531c","Type":"ContainerDied","Data":"a293ed3cc42d1f1b8acb7f872f9c0a1a9058bbe2e7653ab38fc49bd02abc9b59"} Nov 03 23:08:24.721846 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:24.721530 5196 scope.go:122] "RemoveContainer" containerID="782dd45c8c888507deaaabb5b572ef215ea0d5c1f7e75ba56ceafe1226840954" Nov 03 23:08:24.734703 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:24.734685 5196 scope.go:122] "RemoveContainer" containerID="782dd45c8c888507deaaabb5b572ef215ea0d5c1f7e75ba56ceafe1226840954" Nov 03 23:08:24.735062 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:08:24.735038 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"782dd45c8c888507deaaabb5b572ef215ea0d5c1f7e75ba56ceafe1226840954\": container with ID starting with 782dd45c8c888507deaaabb5b572ef215ea0d5c1f7e75ba56ceafe1226840954 not found: ID does not exist" containerID="782dd45c8c888507deaaabb5b572ef215ea0d5c1f7e75ba56ceafe1226840954" Nov 03 23:08:24.735172 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:24.735070 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"782dd45c8c888507deaaabb5b572ef215ea0d5c1f7e75ba56ceafe1226840954"} err="failed to get container status \"782dd45c8c888507deaaabb5b572ef215ea0d5c1f7e75ba56ceafe1226840954\": rpc error: code = NotFound desc = could not find container \"782dd45c8c888507deaaabb5b572ef215ea0d5c1f7e75ba56ceafe1226840954\": container with ID starting with 782dd45c8c888507deaaabb5b572ef215ea0d5c1f7e75ba56ceafe1226840954 not found: ID does not exist" Nov 03 23:08:24.888062 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:24.888031 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm"] Nov 03 23:08:24.891616 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:24.891592 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-164c36-6cfc6bdb96-hchgm"] Nov 03 23:08:25.216155 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:25.215895 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65"] Nov 03 23:08:25.216155 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:25.216121 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65" podUID="de6a6c43-96f4-435e-9552-292d2112ed70" containerName="maas-api" containerID="cri-o://87d840da773fc1763e9e586a9d8de75c2a16f73f7269335e0aeb2bebfa3ab531" gracePeriod=30 Nov 03 23:08:25.282354 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:25.282314 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65" podUID="de6a6c43-96f4-435e-9552-292d2112ed70" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.47:8443/health\": dial tcp 10.133.0.47:8443: connect: connection refused" Nov 03 23:08:25.315401 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:25.315368 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:25.487109 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:25.487071 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65" Nov 03 23:08:25.595693 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:25.595653 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de6a6c43-96f4-435e-9552-292d2112ed70-kube-api-access-qb9pd\" (UniqueName: \"kubernetes.io/projected/de6a6c43-96f4-435e-9552-292d2112ed70-kube-api-access-qb9pd\") pod \"de6a6c43-96f4-435e-9552-292d2112ed70\" (UID: \"de6a6c43-96f4-435e-9552-292d2112ed70\") " Nov 03 23:08:25.595931 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:25.595829 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de6a6c43-96f4-435e-9552-292d2112ed70-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de6a6c43-96f4-435e-9552-292d2112ed70-maas-api-tls\") pod \"de6a6c43-96f4-435e-9552-292d2112ed70\" (UID: \"de6a6c43-96f4-435e-9552-292d2112ed70\") " Nov 03 23:08:25.595931 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:25.595887 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de6a6c43-96f4-435e-9552-292d2112ed70-metrics-certs\" (UniqueName: \"kubernetes.io/secret/de6a6c43-96f4-435e-9552-292d2112ed70-metrics-certs\") pod \"de6a6c43-96f4-435e-9552-292d2112ed70\" (UID: \"de6a6c43-96f4-435e-9552-292d2112ed70\") " Nov 03 23:08:25.595931 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:25.595922 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de6a6c43-96f4-435e-9552-292d2112ed70-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de6a6c43-96f4-435e-9552-292d2112ed70-openshift-service-ca\") pod \"de6a6c43-96f4-435e-9552-292d2112ed70\" (UID: \"de6a6c43-96f4-435e-9552-292d2112ed70\") " Nov 03 23:08:25.596664 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:25.596616 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de6a6c43-96f4-435e-9552-292d2112ed70-openshift-service-ca" pod "de6a6c43-96f4-435e-9552-292d2112ed70" (UID: "de6a6c43-96f4-435e-9552-292d2112ed70"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:08:25.598349 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:25.598319 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de6a6c43-96f4-435e-9552-292d2112ed70-maas-api-tls" pod "de6a6c43-96f4-435e-9552-292d2112ed70" (UID: "de6a6c43-96f4-435e-9552-292d2112ed70"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:08:25.598704 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:25.598673 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de6a6c43-96f4-435e-9552-292d2112ed70-metrics-certs" pod "de6a6c43-96f4-435e-9552-292d2112ed70" (UID: "de6a6c43-96f4-435e-9552-292d2112ed70"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:08:25.598923 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:25.598900 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de6a6c43-96f4-435e-9552-292d2112ed70-kube-api-access-qb9pd" pod "de6a6c43-96f4-435e-9552-292d2112ed70" (UID: "de6a6c43-96f4-435e-9552-292d2112ed70"). InnerVolumeSpecName "kube-api-access-qb9pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:08:25.675997 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:25.675959 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:25.697203 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:25.697177 5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de6a6c43-96f4-435e-9552-292d2112ed70-maas-api-tls\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:25.697203 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:25.697198 5196 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de6a6c43-96f4-435e-9552-292d2112ed70-metrics-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:25.697372 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:25.697210 5196 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de6a6c43-96f4-435e-9552-292d2112ed70-openshift-service-ca\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:25.697372 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:25.697220 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb9pd\" (UniqueName: \"kubernetes.io/projected/de6a6c43-96f4-435e-9552-292d2112ed70-kube-api-access-qb9pd\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:25.728293 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:25.728222 5196 generic.go:356] "Generic (PLEG): container finished" podID="de6a6c43-96f4-435e-9552-292d2112ed70" containerID="87d840da773fc1763e9e586a9d8de75c2a16f73f7269335e0aeb2bebfa3ab531" exitCode=0 Nov 03 23:08:25.728293 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:25.728275 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65" Nov 03 23:08:25.728496 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:25.728309 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65" event={"ID":"de6a6c43-96f4-435e-9552-292d2112ed70","Type":"ContainerDied","Data":"87d840da773fc1763e9e586a9d8de75c2a16f73f7269335e0aeb2bebfa3ab531"} Nov 03 23:08:25.728496 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:25.728373 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65" event={"ID":"de6a6c43-96f4-435e-9552-292d2112ed70","Type":"ContainerDied","Data":"8160dd2856e1aac282c17df520454ad274a7980678d1b9ae1a041ef04f0bb0e0"} Nov 03 23:08:25.728496 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:25.728401 5196 scope.go:122] "RemoveContainer" containerID="87d840da773fc1763e9e586a9d8de75c2a16f73f7269335e0aeb2bebfa3ab531" Nov 03 23:08:25.738774 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:25.738759 5196 scope.go:122] "RemoveContainer" containerID="87d840da773fc1763e9e586a9d8de75c2a16f73f7269335e0aeb2bebfa3ab531" Nov 03 23:08:25.739083 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:08:25.739061 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87d840da773fc1763e9e586a9d8de75c2a16f73f7269335e0aeb2bebfa3ab531\": container with ID starting with 87d840da773fc1763e9e586a9d8de75c2a16f73f7269335e0aeb2bebfa3ab531 not found: ID does not exist" containerID="87d840da773fc1763e9e586a9d8de75c2a16f73f7269335e0aeb2bebfa3ab531" Nov 03 23:08:25.739142 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:25.739092 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87d840da773fc1763e9e586a9d8de75c2a16f73f7269335e0aeb2bebfa3ab531"} err="failed to get container status \"87d840da773fc1763e9e586a9d8de75c2a16f73f7269335e0aeb2bebfa3ab531\": rpc error: code = NotFound desc = could not find container \"87d840da773fc1763e9e586a9d8de75c2a16f73f7269335e0aeb2bebfa3ab531\": container with ID starting with 87d840da773fc1763e9e586a9d8de75c2a16f73f7269335e0aeb2bebfa3ab531 not found: ID does not exist" Nov 03 23:08:25.898013 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:25.897985 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65"] Nov 03 23:08:25.903160 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:25.903130 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-1d31f8-54964665b5-56v65"] Nov 03 23:08:25.919156 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:08:25.919126 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:08:25.956511 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:25.956483 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:26.613023 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:26.612982 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:26.813811 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:26.813778 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3692067f-4a4d-4bd0-bc74-8957aaf8531c" path="/var/lib/kubelet/pods/3692067f-4a4d-4bd0-bc74-8957aaf8531c/volumes" Nov 03 23:08:26.814279 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:26.814258 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de6a6c43-96f4-435e-9552-292d2112ed70" path="/var/lib/kubelet/pods/de6a6c43-96f4-435e-9552-292d2112ed70/volumes" Nov 03 23:08:26.852984 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:26.852955 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:27.291754 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:27.291712 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:27.797846 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:27.797806 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:27.938512 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:27.938478 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:32.508730 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:32.508697 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:32.653073 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:32.653040 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:32.925665 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:32.925630 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m"] Nov 03 23:08:32.925901 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:32.925879 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m" podUID="89748893-72b3-46f6-9419-c8fd5fe4cd73" containerName="maas-api" containerID="cri-o://788c68cedef14248acd1037df4c9e18416c1e99832cf67bf494886c4d8bd8e0a" gracePeriod=30 Nov 03 23:08:33.030012 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:33.029969 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:33.073544 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:33.073496 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m" podUID="89748893-72b3-46f6-9419-c8fd5fe4cd73" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.44:8443/health\": dial tcp 10.133.0.44:8443: connect: connection refused" Nov 03 23:08:33.228699 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:33.228670 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m" Nov 03 23:08:33.372180 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:33.372148 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89748893-72b3-46f6-9419-c8fd5fe4cd73-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89748893-72b3-46f6-9419-c8fd5fe4cd73-openshift-service-ca\") pod \"89748893-72b3-46f6-9419-c8fd5fe4cd73\" (UID: \"89748893-72b3-46f6-9419-c8fd5fe4cd73\") " Nov 03 23:08:33.372361 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:33.372245 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89748893-72b3-46f6-9419-c8fd5fe4cd73-metrics-certs\" (UniqueName: \"kubernetes.io/secret/89748893-72b3-46f6-9419-c8fd5fe4cd73-metrics-certs\") pod \"89748893-72b3-46f6-9419-c8fd5fe4cd73\" (UID: \"89748893-72b3-46f6-9419-c8fd5fe4cd73\") " Nov 03 23:08:33.372361 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:33.372315 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89748893-72b3-46f6-9419-c8fd5fe4cd73-kube-api-access-ltfm6\" (UniqueName: \"kubernetes.io/projected/89748893-72b3-46f6-9419-c8fd5fe4cd73-kube-api-access-ltfm6\") pod \"89748893-72b3-46f6-9419-c8fd5fe4cd73\" (UID: \"89748893-72b3-46f6-9419-c8fd5fe4cd73\") " Nov 03 23:08:33.372361 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:33.372358 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89748893-72b3-46f6-9419-c8fd5fe4cd73-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89748893-72b3-46f6-9419-c8fd5fe4cd73-maas-api-tls\") pod \"89748893-72b3-46f6-9419-c8fd5fe4cd73\" (UID: \"89748893-72b3-46f6-9419-c8fd5fe4cd73\") " Nov 03 23:08:33.372595 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:33.372565 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89748893-72b3-46f6-9419-c8fd5fe4cd73-openshift-service-ca" pod "89748893-72b3-46f6-9419-c8fd5fe4cd73" (UID: "89748893-72b3-46f6-9419-c8fd5fe4cd73"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:08:33.375256 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:33.375215 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89748893-72b3-46f6-9419-c8fd5fe4cd73-kube-api-access-ltfm6" pod "89748893-72b3-46f6-9419-c8fd5fe4cd73" (UID: "89748893-72b3-46f6-9419-c8fd5fe4cd73"). InnerVolumeSpecName "kube-api-access-ltfm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:08:33.375406 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:33.375381 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89748893-72b3-46f6-9419-c8fd5fe4cd73-metrics-certs" pod "89748893-72b3-46f6-9419-c8fd5fe4cd73" (UID: "89748893-72b3-46f6-9419-c8fd5fe4cd73"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:08:33.375484 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:33.375466 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89748893-72b3-46f6-9419-c8fd5fe4cd73-maas-api-tls" pod "89748893-72b3-46f6-9419-c8fd5fe4cd73" (UID: "89748893-72b3-46f6-9419-c8fd5fe4cd73"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:08:33.474092 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:33.474018 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltfm6\" (UniqueName: \"kubernetes.io/projected/89748893-72b3-46f6-9419-c8fd5fe4cd73-kube-api-access-ltfm6\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:33.474092 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:33.474046 5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89748893-72b3-46f6-9419-c8fd5fe4cd73-maas-api-tls\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:33.474092 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:33.474056 5196 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89748893-72b3-46f6-9419-c8fd5fe4cd73-openshift-service-ca\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:33.474092 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:33.474064 5196 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89748893-72b3-46f6-9419-c8fd5fe4cd73-metrics-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:33.495896 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:33.495838 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:33.647273 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:33.647240 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:33.782717 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:33.782630 5196 generic.go:356] "Generic (PLEG): container finished" podID="89748893-72b3-46f6-9419-c8fd5fe4cd73" containerID="788c68cedef14248acd1037df4c9e18416c1e99832cf67bf494886c4d8bd8e0a" exitCode=0 Nov 03 23:08:33.782911 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:33.782724 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m" Nov 03 23:08:33.782911 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:33.782738 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m" event={"ID":"89748893-72b3-46f6-9419-c8fd5fe4cd73","Type":"ContainerDied","Data":"788c68cedef14248acd1037df4c9e18416c1e99832cf67bf494886c4d8bd8e0a"} Nov 03 23:08:33.782911 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:33.782806 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m" event={"ID":"89748893-72b3-46f6-9419-c8fd5fe4cd73","Type":"ContainerDied","Data":"61ab4cf7e669129c8ac6ad5a43b7a7fb27df7bb168026be7d796851ea83ab7f2"} Nov 03 23:08:33.782911 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:33.782846 5196 scope.go:122] "RemoveContainer" containerID="788c68cedef14248acd1037df4c9e18416c1e99832cf67bf494886c4d8bd8e0a" Nov 03 23:08:33.802739 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:33.802720 5196 scope.go:122] "RemoveContainer" containerID="788c68cedef14248acd1037df4c9e18416c1e99832cf67bf494886c4d8bd8e0a" Nov 03 23:08:33.803119 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:08:33.803093 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"788c68cedef14248acd1037df4c9e18416c1e99832cf67bf494886c4d8bd8e0a\": container with ID starting with 788c68cedef14248acd1037df4c9e18416c1e99832cf67bf494886c4d8bd8e0a not found: ID does not exist" containerID="788c68cedef14248acd1037df4c9e18416c1e99832cf67bf494886c4d8bd8e0a" Nov 03 23:08:33.803215 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:33.803133 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"788c68cedef14248acd1037df4c9e18416c1e99832cf67bf494886c4d8bd8e0a"} err="failed to get container status \"788c68cedef14248acd1037df4c9e18416c1e99832cf67bf494886c4d8bd8e0a\": rpc error: code = NotFound desc = could not find container \"788c68cedef14248acd1037df4c9e18416c1e99832cf67bf494886c4d8bd8e0a\": container with ID starting with 788c68cedef14248acd1037df4c9e18416c1e99832cf67bf494886c4d8bd8e0a not found: ID does not exist" Nov 03 23:08:33.823704 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:33.823677 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:33.949947 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:33.949916 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m"] Nov 03 23:08:33.954307 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:33.954280 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-32449d-58f8479b5c-s9q6m"] Nov 03 23:08:34.377886 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:34.374607 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:34.545551 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:34.545516 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:34.814578 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:34.814499 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89748893-72b3-46f6-9419-c8fd5fe4cd73" path="/var/lib/kubelet/pods/89748893-72b3-46f6-9419-c8fd5fe4cd73/volumes" Nov 03 23:08:34.847610 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:34.847575 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:35.483532 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:35.483501 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:35.669787 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:35.669752 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:35.816464 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:35.816390 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:36.054814 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:36.054778 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:41.967112 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:41.967070 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:42.170344 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:42.170307 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:42.322654 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:42.322569 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:42.868983 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:42.868954 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:43.014310 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:43.014273 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:44.053597 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:44.053561 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:47.487576 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:47.487538 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8657649675-k4qdp"] Nov 03 23:08:47.488083 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:47.487728 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-8657649675-k4qdp" podUID="69b6bfce-f5ca-4c52-a9b1-5749383d2370" containerName="manager" containerID="cri-o://17a4819a1111b76dfc2ec3cec7039813a057fa0b5f3df4b3edfd03d67bd6dbde" gracePeriod=30 Nov 03 23:08:47.746754 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:47.746693 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8657649675-k4qdp" Nov 03 23:08:47.854107 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:47.854068 5196 generic.go:356] "Generic (PLEG): container finished" podID="69b6bfce-f5ca-4c52-a9b1-5749383d2370" containerID="17a4819a1111b76dfc2ec3cec7039813a057fa0b5f3df4b3edfd03d67bd6dbde" exitCode=0 Nov 03 23:08:47.854107 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:47.854109 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8657649675-k4qdp" Nov 03 23:08:47.854304 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:47.854141 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8657649675-k4qdp" event={"ID":"69b6bfce-f5ca-4c52-a9b1-5749383d2370","Type":"ContainerDied","Data":"17a4819a1111b76dfc2ec3cec7039813a057fa0b5f3df4b3edfd03d67bd6dbde"} Nov 03 23:08:47.854304 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:47.854192 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8657649675-k4qdp" event={"ID":"69b6bfce-f5ca-4c52-a9b1-5749383d2370","Type":"ContainerDied","Data":"899949a651254138fb34d6137ad6e2fc96c4f8f9555bbe91df48a59169659b54"} Nov 03 23:08:47.854304 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:47.854214 5196 scope.go:122] "RemoveContainer" containerID="17a4819a1111b76dfc2ec3cec7039813a057fa0b5f3df4b3edfd03d67bd6dbde" Nov 03 23:08:47.863050 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:47.863032 5196 scope.go:122] "RemoveContainer" containerID="ea84c679a1684e677cf3fe9c886e526d3439eedbea1ef2da0a75fb7cf18b960c" Nov 03 23:08:47.872336 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:47.872316 5196 scope.go:122] "RemoveContainer" containerID="17a4819a1111b76dfc2ec3cec7039813a057fa0b5f3df4b3edfd03d67bd6dbde" Nov 03 23:08:47.872723 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:08:47.872693 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17a4819a1111b76dfc2ec3cec7039813a057fa0b5f3df4b3edfd03d67bd6dbde\": container with ID starting with 17a4819a1111b76dfc2ec3cec7039813a057fa0b5f3df4b3edfd03d67bd6dbde not found: ID does not exist" containerID="17a4819a1111b76dfc2ec3cec7039813a057fa0b5f3df4b3edfd03d67bd6dbde" Nov 03 23:08:47.872882 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:47.872734 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17a4819a1111b76dfc2ec3cec7039813a057fa0b5f3df4b3edfd03d67bd6dbde"} err="failed to get container status \"17a4819a1111b76dfc2ec3cec7039813a057fa0b5f3df4b3edfd03d67bd6dbde\": rpc error: code = NotFound desc = could not find container \"17a4819a1111b76dfc2ec3cec7039813a057fa0b5f3df4b3edfd03d67bd6dbde\": container with ID starting with 17a4819a1111b76dfc2ec3cec7039813a057fa0b5f3df4b3edfd03d67bd6dbde not found: ID does not exist" Nov 03 23:08:47.872882 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:47.872759 5196 scope.go:122] "RemoveContainer" containerID="ea84c679a1684e677cf3fe9c886e526d3439eedbea1ef2da0a75fb7cf18b960c" Nov 03 23:08:47.873147 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:08:47.873123 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea84c679a1684e677cf3fe9c886e526d3439eedbea1ef2da0a75fb7cf18b960c\": container with ID starting with ea84c679a1684e677cf3fe9c886e526d3439eedbea1ef2da0a75fb7cf18b960c not found: ID does not exist" containerID="ea84c679a1684e677cf3fe9c886e526d3439eedbea1ef2da0a75fb7cf18b960c" Nov 03 23:08:47.873245 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:47.873157 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea84c679a1684e677cf3fe9c886e526d3439eedbea1ef2da0a75fb7cf18b960c"} err="failed to get container status \"ea84c679a1684e677cf3fe9c886e526d3439eedbea1ef2da0a75fb7cf18b960c\": rpc error: code = NotFound desc = could not find container \"ea84c679a1684e677cf3fe9c886e526d3439eedbea1ef2da0a75fb7cf18b960c\": container with ID starting with ea84c679a1684e677cf3fe9c886e526d3439eedbea1ef2da0a75fb7cf18b960c not found: ID does not exist" Nov 03 23:08:47.921113 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:47.921089 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69b6bfce-f5ca-4c52-a9b1-5749383d2370-webhook-cert\" (UniqueName: \"kubernetes.io/secret/69b6bfce-f5ca-4c52-a9b1-5749383d2370-webhook-cert\") pod \"69b6bfce-f5ca-4c52-a9b1-5749383d2370\" (UID: \"69b6bfce-f5ca-4c52-a9b1-5749383d2370\") " Nov 03 23:08:47.921274 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:47.921124 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69b6bfce-f5ca-4c52-a9b1-5749383d2370-kube-api-access-8zrn4\" (UniqueName: \"kubernetes.io/projected/69b6bfce-f5ca-4c52-a9b1-5749383d2370-kube-api-access-8zrn4\") pod \"69b6bfce-f5ca-4c52-a9b1-5749383d2370\" (UID: \"69b6bfce-f5ca-4c52-a9b1-5749383d2370\") " Nov 03 23:08:47.921274 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:47.921165 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69b6bfce-f5ca-4c52-a9b1-5749383d2370-metrics-certs\" (UniqueName: \"kubernetes.io/secret/69b6bfce-f5ca-4c52-a9b1-5749383d2370-metrics-certs\") pod \"69b6bfce-f5ca-4c52-a9b1-5749383d2370\" (UID: \"69b6bfce-f5ca-4c52-a9b1-5749383d2370\") " Nov 03 23:08:47.923337 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:47.923309 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69b6bfce-f5ca-4c52-a9b1-5749383d2370-webhook-cert" pod "69b6bfce-f5ca-4c52-a9b1-5749383d2370" (UID: "69b6bfce-f5ca-4c52-a9b1-5749383d2370"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:08:47.923759 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:47.923740 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69b6bfce-f5ca-4c52-a9b1-5749383d2370-metrics-certs" pod "69b6bfce-f5ca-4c52-a9b1-5749383d2370" (UID: "69b6bfce-f5ca-4c52-a9b1-5749383d2370"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:08:47.923827 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:47.923801 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69b6bfce-f5ca-4c52-a9b1-5749383d2370-kube-api-access-8zrn4" pod "69b6bfce-f5ca-4c52-a9b1-5749383d2370" (UID: "69b6bfce-f5ca-4c52-a9b1-5749383d2370"). InnerVolumeSpecName "kube-api-access-8zrn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:08:48.022150 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:48.022092 5196 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/69b6bfce-f5ca-4c52-a9b1-5749383d2370-webhook-cert\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:48.022150 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:48.022116 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zrn4\" (UniqueName: \"kubernetes.io/projected/69b6bfce-f5ca-4c52-a9b1-5749383d2370-kube-api-access-8zrn4\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:48.022150 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:48.022126 5196 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69b6bfce-f5ca-4c52-a9b1-5749383d2370-metrics-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:08:48.275052 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:48.274967 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8657649675-k4qdp"] Nov 03 23:08:48.325460 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:48.325432 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-8657649675-k4qdp"] Nov 03 23:08:48.813614 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:48.813572 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69b6bfce-f5ca-4c52-a9b1-5749383d2370" path="/var/lib/kubelet/pods/69b6bfce-f5ca-4c52-a9b1-5749383d2370/volumes" Nov 03 23:08:49.939243 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:08:49.939194 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:08:55.917973 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:08:55.917933 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:09:04.850198 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850169 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr"] Nov 03 23:09:04.850607 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850477 5196 cpu_manager.go:414] "containerMap: removing container" podUID="34e9cf3a-3456-4835-8933-102ce735cccc" containerName="main" Nov 03 23:09:04.850607 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850487 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34e9cf3a-3456-4835-8933-102ce735cccc" containerName="main" Nov 03 23:09:04.850607 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850499 5196 cpu_manager.go:414] "containerMap: removing container" podUID="3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b" containerName="maas-api" Nov 03 23:09:04.850607 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850504 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b" containerName="maas-api" Nov 03 23:09:04.850607 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850515 5196 cpu_manager.go:414] "containerMap: removing container" podUID="89748893-72b3-46f6-9419-c8fd5fe4cd73" containerName="maas-api" Nov 03 23:09:04.850607 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850520 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89748893-72b3-46f6-9419-c8fd5fe4cd73" containerName="maas-api" Nov 03 23:09:04.850607 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850527 5196 cpu_manager.go:414] "containerMap: removing container" podUID="de6a6c43-96f4-435e-9552-292d2112ed70" containerName="maas-api" Nov 03 23:09:04.850607 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850531 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de6a6c43-96f4-435e-9552-292d2112ed70" containerName="maas-api" Nov 03 23:09:04.850607 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850537 5196 cpu_manager.go:414] "containerMap: removing container" podUID="ff5d39d8-8b0c-4dc2-ab16-7626474d04de" containerName="maas-api" Nov 03 23:09:04.850607 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850543 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff5d39d8-8b0c-4dc2-ab16-7626474d04de" containerName="maas-api" Nov 03 23:09:04.850607 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850549 5196 cpu_manager.go:414] "containerMap: removing container" podUID="69b6bfce-f5ca-4c52-a9b1-5749383d2370" containerName="manager" Nov 03 23:09:04.850607 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850554 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69b6bfce-f5ca-4c52-a9b1-5749383d2370" containerName="manager" Nov 03 23:09:04.850607 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850561 5196 cpu_manager.go:414] "containerMap: removing container" podUID="cef4b051-45b0-4db8-8ab5-afcfe3b9149d" containerName="istio-proxy" Nov 03 23:09:04.850607 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850565 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cef4b051-45b0-4db8-8ab5-afcfe3b9149d" containerName="istio-proxy" Nov 03 23:09:04.850607 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850572 5196 cpu_manager.go:414] "containerMap: removing container" podUID="3692067f-4a4d-4bd0-bc74-8957aaf8531c" containerName="maas-api" Nov 03 23:09:04.850607 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850576 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3692067f-4a4d-4bd0-bc74-8957aaf8531c" containerName="maas-api" Nov 03 23:09:04.850607 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850582 5196 cpu_manager.go:414] "containerMap: removing container" podUID="7914cbce-00fd-4d6b-84c9-4519c50af650" containerName="maas-api" Nov 03 23:09:04.850607 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850586 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7914cbce-00fd-4d6b-84c9-4519c50af650" containerName="maas-api" Nov 03 23:09:04.850607 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850591 5196 cpu_manager.go:414] "containerMap: removing container" podUID="59200876-e179-4170-832d-d85cba111281" containerName="istio-proxy" Nov 03 23:09:04.850607 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850595 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59200876-e179-4170-832d-d85cba111281" containerName="istio-proxy" Nov 03 23:09:04.850607 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850601 5196 cpu_manager.go:414] "containerMap: removing container" podUID="b8d72750-08ae-4995-ac23-6304ada9b53f" containerName="istio-proxy" Nov 03 23:09:04.850607 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850605 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8d72750-08ae-4995-ac23-6304ada9b53f" containerName="istio-proxy" Nov 03 23:09:04.850607 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850611 5196 cpu_manager.go:414] "containerMap: removing container" podUID="5f4dad96-aaa9-467a-b599-6b45196c5eaa" containerName="maas-api" Nov 03 23:09:04.850607 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850615 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f4dad96-aaa9-467a-b599-6b45196c5eaa" containerName="maas-api" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850622 5196 cpu_manager.go:414] "containerMap: removing container" podUID="d4042310-83e8-44e6-b8e4-73ecb70403b3" containerName="maas-api" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850626 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4042310-83e8-44e6-b8e4-73ecb70403b3" containerName="maas-api" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850632 5196 cpu_manager.go:414] "containerMap: removing container" podUID="26fa51c2-0f2e-482f-9c67-0ea7924d7b3c" containerName="main" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850636 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26fa51c2-0f2e-482f-9c67-0ea7924d7b3c" containerName="main" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850647 5196 cpu_manager.go:414] "containerMap: removing container" podUID="8f7ce2e3-e5ee-45b8-99e6-985011216f8b" containerName="maas-api" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850651 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f7ce2e3-e5ee-45b8-99e6-985011216f8b" containerName="maas-api" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850659 5196 cpu_manager.go:414] "containerMap: removing container" podUID="cef4b051-45b0-4db8-8ab5-afcfe3b9149d" containerName="istio-proxy" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850663 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cef4b051-45b0-4db8-8ab5-afcfe3b9149d" containerName="istio-proxy" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850669 5196 cpu_manager.go:414] "containerMap: removing container" podUID="c0801535-901a-46f0-a882-e9634537f0fc" containerName="maas-api" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850673 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0801535-901a-46f0-a882-e9634537f0fc" containerName="maas-api" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850680 5196 cpu_manager.go:414] "containerMap: removing container" podUID="e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf" containerName="istio-proxy" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850684 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf" containerName="istio-proxy" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850689 5196 cpu_manager.go:414] "containerMap: removing container" podUID="9eec3a9d-0eea-44e8-92e4-2d19e4508148" containerName="istio-proxy" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850694 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eec3a9d-0eea-44e8-92e4-2d19e4508148" containerName="istio-proxy" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850699 5196 cpu_manager.go:414] "containerMap: removing container" podUID="69b6bfce-f5ca-4c52-a9b1-5749383d2370" containerName="manager" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850703 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69b6bfce-f5ca-4c52-a9b1-5749383d2370" containerName="manager" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850710 5196 cpu_manager.go:414] "containerMap: removing container" podUID="59200876-e179-4170-832d-d85cba111281" containerName="istio-proxy" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850714 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59200876-e179-4170-832d-d85cba111281" containerName="istio-proxy" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850729 5196 cpu_manager.go:414] "containerMap: removing container" podUID="d6540706-d21a-4432-8f80-a6d427a13153" containerName="maas-api" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850733 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6540706-d21a-4432-8f80-a6d427a13153" containerName="maas-api" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850739 5196 cpu_manager.go:414] "containerMap: removing container" podUID="0d375a3f-2f84-4495-a683-42a98477ff52" containerName="istio-proxy" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850743 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d375a3f-2f84-4495-a683-42a98477ff52" containerName="istio-proxy" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850749 5196 cpu_manager.go:414] "containerMap: removing container" podUID="4b847dde-7257-44e0-b25c-9ebe4cdad697" containerName="istio-proxy" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850753 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b847dde-7257-44e0-b25c-9ebe4cdad697" containerName="istio-proxy" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850759 5196 cpu_manager.go:414] "containerMap: removing container" podUID="41e5ef76-ed91-4c59-a3e5-36a288fbc075" containerName="maas-api" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850763 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41e5ef76-ed91-4c59-a3e5-36a288fbc075" containerName="maas-api" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850770 5196 cpu_manager.go:414] "containerMap: removing container" podUID="0d375a3f-2f84-4495-a683-42a98477ff52" containerName="istio-proxy" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850774 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d375a3f-2f84-4495-a683-42a98477ff52" containerName="istio-proxy" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850779 5196 cpu_manager.go:414] "containerMap: removing container" podUID="4b847dde-7257-44e0-b25c-9ebe4cdad697" containerName="istio-proxy" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850783 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b847dde-7257-44e0-b25c-9ebe4cdad697" containerName="istio-proxy" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850791 5196 cpu_manager.go:414] "containerMap: removing container" podUID="9d8b4fed-b4b0-49ce-9238-e23175e90b70" containerName="main" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850795 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d8b4fed-b4b0-49ce-9238-e23175e90b70" containerName="main" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850801 5196 cpu_manager.go:414] "containerMap: removing container" podUID="47d17627-fece-4154-a00b-bfa9838574ca" containerName="maas-api" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850806 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47d17627-fece-4154-a00b-bfa9838574ca" containerName="maas-api" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850881 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="34e9cf3a-3456-4835-8933-102ce735cccc" containerName="main" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850890 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f495de2-3c7c-4d3b-9da6-bdb97ac16e8b" containerName="maas-api" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850896 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="cef4b051-45b0-4db8-8ab5-afcfe3b9149d" containerName="istio-proxy" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850901 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0801535-901a-46f0-a882-e9634537f0fc" containerName="maas-api" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850907 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f2ded9-7d5f-4ea5-a2c2-fb9c6c1f99cf" containerName="istio-proxy" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850912 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="47d17627-fece-4154-a00b-bfa9838574ca" containerName="maas-api" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850919 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eec3a9d-0eea-44e8-92e4-2d19e4508148" containerName="istio-proxy" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850925 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="69b6bfce-f5ca-4c52-a9b1-5749383d2370" containerName="manager" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850929 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="89748893-72b3-46f6-9419-c8fd5fe4cd73" containerName="maas-api" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850934 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="de6a6c43-96f4-435e-9552-292d2112ed70" containerName="maas-api" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850939 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff5d39d8-8b0c-4dc2-ab16-7626474d04de" containerName="maas-api" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850945 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="59200876-e179-4170-832d-d85cba111281" containerName="istio-proxy" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850952 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="69b6bfce-f5ca-4c52-a9b1-5749383d2370" containerName="manager" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850957 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6540706-d21a-4432-8f80-a6d427a13153" containerName="maas-api" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850962 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="cef4b051-45b0-4db8-8ab5-afcfe3b9149d" containerName="istio-proxy" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850966 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d375a3f-2f84-4495-a683-42a98477ff52" containerName="istio-proxy" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850970 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b847dde-7257-44e0-b25c-9ebe4cdad697" containerName="istio-proxy" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850975 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="3692067f-4a4d-4bd0-bc74-8957aaf8531c" containerName="maas-api" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850980 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="41e5ef76-ed91-4c59-a3e5-36a288fbc075" containerName="maas-api" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850984 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d375a3f-2f84-4495-a683-42a98477ff52" containerName="istio-proxy" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850988 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="7914cbce-00fd-4d6b-84c9-4519c50af650" containerName="maas-api" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850992 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="59200876-e179-4170-832d-d85cba111281" containerName="istio-proxy" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.850998 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8d72750-08ae-4995-ac23-6304ada9b53f" containerName="istio-proxy" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.851003 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b847dde-7257-44e0-b25c-9ebe4cdad697" containerName="istio-proxy" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.851009 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f4dad96-aaa9-467a-b599-6b45196c5eaa" containerName="maas-api" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.851013 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4042310-83e8-44e6-b8e4-73ecb70403b3" containerName="maas-api" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.851017 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="26fa51c2-0f2e-482f-9c67-0ea7924d7b3c" containerName="main" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.851023 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d8b4fed-b4b0-49ce-9238-e23175e90b70" containerName="main" Nov 03 23:09:04.851373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.851028 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f7ce2e3-e5ee-45b8-99e6-985011216f8b" containerName="maas-api" Nov 03 23:09:04.855901 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.855884 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" Nov 03 23:09:04.965005 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.964975 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07738db5-20e8-47f1-8339-82866bfa91fc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cdxtr\" (UID: \"07738db5-20e8-47f1-8339-82866bfa91fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" Nov 03 23:09:04.965005 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.965010 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07738db5-20e8-47f1-8339-82866bfa91fc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cdxtr\" (UID: \"07738db5-20e8-47f1-8339-82866bfa91fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" Nov 03 23:09:04.965216 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.965091 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07738db5-20e8-47f1-8339-82866bfa91fc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cdxtr\" (UID: \"07738db5-20e8-47f1-8339-82866bfa91fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" Nov 03 23:09:04.965216 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.965121 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07738db5-20e8-47f1-8339-82866bfa91fc-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cdxtr\" (UID: \"07738db5-20e8-47f1-8339-82866bfa91fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" Nov 03 23:09:04.965216 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.965195 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07738db5-20e8-47f1-8339-82866bfa91fc-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cdxtr\" (UID: \"07738db5-20e8-47f1-8339-82866bfa91fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" Nov 03 23:09:04.965365 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.965233 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07738db5-20e8-47f1-8339-82866bfa91fc-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cdxtr\" (UID: \"07738db5-20e8-47f1-8339-82866bfa91fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" Nov 03 23:09:04.965365 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.965316 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtpnq\" (UniqueName: \"kubernetes.io/projected/07738db5-20e8-47f1-8339-82866bfa91fc-kube-api-access-dtpnq\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cdxtr\" (UID: \"07738db5-20e8-47f1-8339-82866bfa91fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" Nov 03 23:09:04.965484 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.965374 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07738db5-20e8-47f1-8339-82866bfa91fc-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cdxtr\" (UID: \"07738db5-20e8-47f1-8339-82866bfa91fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" Nov 03 23:09:04.965484 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:04.965401 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07738db5-20e8-47f1-8339-82866bfa91fc-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cdxtr\" (UID: \"07738db5-20e8-47f1-8339-82866bfa91fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" Nov 03 23:09:05.062667 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:05.062638 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr"] Nov 03 23:09:05.066812 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:05.066786 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtpnq\" (UniqueName: \"kubernetes.io/projected/07738db5-20e8-47f1-8339-82866bfa91fc-kube-api-access-dtpnq\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cdxtr\" (UID: \"07738db5-20e8-47f1-8339-82866bfa91fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" Nov 03 23:09:05.066982 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:05.066823 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07738db5-20e8-47f1-8339-82866bfa91fc-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cdxtr\" (UID: \"07738db5-20e8-47f1-8339-82866bfa91fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" Nov 03 23:09:05.066982 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:05.066850 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07738db5-20e8-47f1-8339-82866bfa91fc-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cdxtr\" (UID: \"07738db5-20e8-47f1-8339-82866bfa91fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" Nov 03 23:09:05.066982 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:05.066934 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07738db5-20e8-47f1-8339-82866bfa91fc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cdxtr\" (UID: \"07738db5-20e8-47f1-8339-82866bfa91fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" Nov 03 23:09:05.066982 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:05.066962 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07738db5-20e8-47f1-8339-82866bfa91fc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cdxtr\" (UID: \"07738db5-20e8-47f1-8339-82866bfa91fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" Nov 03 23:09:05.067223 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:05.067034 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07738db5-20e8-47f1-8339-82866bfa91fc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cdxtr\" (UID: \"07738db5-20e8-47f1-8339-82866bfa91fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" Nov 03 23:09:05.067223 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:05.067077 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07738db5-20e8-47f1-8339-82866bfa91fc-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cdxtr\" (UID: \"07738db5-20e8-47f1-8339-82866bfa91fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" Nov 03 23:09:05.067223 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:05.067101 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07738db5-20e8-47f1-8339-82866bfa91fc-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cdxtr\" (UID: \"07738db5-20e8-47f1-8339-82866bfa91fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" Nov 03 23:09:05.067223 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:05.067127 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07738db5-20e8-47f1-8339-82866bfa91fc-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cdxtr\" (UID: \"07738db5-20e8-47f1-8339-82866bfa91fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" Nov 03 23:09:05.067454 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:05.067244 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07738db5-20e8-47f1-8339-82866bfa91fc-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cdxtr\" (UID: \"07738db5-20e8-47f1-8339-82866bfa91fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" Nov 03 23:09:05.067454 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:05.067396 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07738db5-20e8-47f1-8339-82866bfa91fc-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cdxtr\" (UID: \"07738db5-20e8-47f1-8339-82866bfa91fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" Nov 03 23:09:05.067573 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:05.067515 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07738db5-20e8-47f1-8339-82866bfa91fc-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cdxtr\" (UID: \"07738db5-20e8-47f1-8339-82866bfa91fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" Nov 03 23:09:05.067640 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:05.067595 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07738db5-20e8-47f1-8339-82866bfa91fc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cdxtr\" (UID: \"07738db5-20e8-47f1-8339-82866bfa91fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" Nov 03 23:09:05.068059 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:05.068033 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07738db5-20e8-47f1-8339-82866bfa91fc-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cdxtr\" (UID: \"07738db5-20e8-47f1-8339-82866bfa91fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" Nov 03 23:09:05.069543 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:05.069495 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07738db5-20e8-47f1-8339-82866bfa91fc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cdxtr\" (UID: \"07738db5-20e8-47f1-8339-82866bfa91fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" Nov 03 23:09:05.069941 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:05.069922 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07738db5-20e8-47f1-8339-82866bfa91fc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cdxtr\" (UID: \"07738db5-20e8-47f1-8339-82866bfa91fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" Nov 03 23:09:05.128280 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:05.128207 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07738db5-20e8-47f1-8339-82866bfa91fc-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cdxtr\" (UID: \"07738db5-20e8-47f1-8339-82866bfa91fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" Nov 03 23:09:05.128514 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:05.128493 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtpnq\" (UniqueName: \"kubernetes.io/projected/07738db5-20e8-47f1-8339-82866bfa91fc-kube-api-access-dtpnq\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cdxtr\" (UID: \"07738db5-20e8-47f1-8339-82866bfa91fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" Nov 03 23:09:05.168401 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:05.168375 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" Nov 03 23:09:05.595520 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:05.595489 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr"] Nov 03 23:09:05.598065 ip-10-0-143-132 kubenswrapper[5196]: W1103 23:09:05.598040 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07738db5_20e8_47f1_8339_82866bfa91fc.slice/crio-ce148bd58420f76d5ff445f34bd1c0dfe7c0e2916de91010bc5613698a77d697 WatchSource:0}: Error finding container ce148bd58420f76d5ff445f34bd1c0dfe7c0e2916de91010bc5613698a77d697: Status 404 returned error can't find the container with id ce148bd58420f76d5ff445f34bd1c0dfe7c0e2916de91010bc5613698a77d697 Nov 03 23:09:05.600129 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:05.600090 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:09:05.600239 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:05.600171 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:09:05.600239 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:05.600215 5196 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 23:09:05.942520 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:05.942427 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" event={"ID":"07738db5-20e8-47f1-8339-82866bfa91fc","Type":"ContainerStarted","Data":"e0c7789874a30273ff30af7f041a73cdc9f40300d9bc3520cd2042d5e168c5f4"} Nov 03 23:09:05.942520 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:05.942484 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" event={"ID":"07738db5-20e8-47f1-8339-82866bfa91fc","Type":"ContainerStarted","Data":"ce148bd58420f76d5ff445f34bd1c0dfe7c0e2916de91010bc5613698a77d697"} Nov 03 23:09:06.066935 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:06.066878 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" podStartSLOduration=2.06684887 podStartE2EDuration="2.06684887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:09:06.065524928 +0000 UTC m=+2775.375084221" watchObservedRunningTime="2026-11-03 23:09:06.06684887 +0000 UTC m=+2775.376408156" Nov 03 23:09:06.168942 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:06.168908 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" Nov 03 23:09:06.170210 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:06.170189 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cdxtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 23:09:06.170307 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:06.170236 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" podUID="07738db5-20e8-47f1-8339-82866bfa91fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 23:09:07.169749 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:07.169712 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cdxtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 23:09:07.170161 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:07.169768 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" podUID="07738db5-20e8-47f1-8339-82866bfa91fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 23:09:08.168833 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:08.168799 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cdxtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 23:09:08.169054 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:08.168888 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" podUID="07738db5-20e8-47f1-8339-82866bfa91fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 23:09:08.352933 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:08.352898 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:09:09.169087 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:09.169051 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cdxtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 23:09:09.169268 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:09.169114 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" podUID="07738db5-20e8-47f1-8339-82866bfa91fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 23:09:10.169373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:10.169340 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cdxtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 23:09:10.169796 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:10.169423 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" podUID="07738db5-20e8-47f1-8339-82866bfa91fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 23:09:11.169246 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:11.169209 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cdxtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 23:09:11.169431 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:11.169285 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" podUID="07738db5-20e8-47f1-8339-82866bfa91fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 23:09:12.169284 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:12.169244 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cdxtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 23:09:12.169432 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:12.169311 5196 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" podUID="07738db5-20e8-47f1-8339-82866bfa91fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 23:09:12.376177 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:12.376140 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:09:13.961580 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:13.961500 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" Nov 03 23:09:13.962095 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:13.961843 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" Nov 03 23:09:13.962887 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:13.962851 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" Nov 03 23:09:17.729378 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:17.729339 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:09:23.759396 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:23.759358 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:09:25.838411 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:25.838375 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:09:25.917684 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:09:25.917645 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:09:32.173408 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:32.173333 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:09:32.321146 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:32.321105 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:09:43.417874 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:09:43.417818 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:09:55.918150 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:09:55.918107 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:10:12.831882 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:10:12.825535 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:10:13.522894 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:10:13.522005 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-njmdj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:10:13.522894 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:10:13.522092 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" podUID="9f644685-a2d7-48ee-833d-d49c9959f598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:10:13.901562 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:10:13.901511 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8q2rz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:10:13.902280 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:10:13.902244 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" podUID="6ef06915-cf1c-45cb-b61b-8c80c45f1dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:10:18.076384 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:10:18.076333 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:10:25.917105 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:10:25.917068 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:10:55.916929 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:10:55.916891 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:11:11.838018 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:11:11.837939 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:11:11.985034 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:11:11.985003 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:11:13.118051 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:11:13.118002 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:11:18.421933 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:11:18.421898 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:11:25.917960 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:11:25.917921 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:11:55.917510 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:11:55.917476 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:11:59.295592 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:11:59.295555 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:11:59.485672 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:11:59.485643 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:12:13.532696 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:12:13.532654 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:12:18.748591 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:12:18.748558 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:12:21.140665 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:12:21.140639 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:12:25.917302 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:12:25.917263 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:12:45.004994 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:12:45.004954 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cdxtr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:12:45.005750 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:12:45.005717 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" podUID="07738db5-20e8-47f1-8339-82866bfa91fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:12:55.917311 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:12:55.917275 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:13:13.922882 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:13:13.922303 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:13:19.113549 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:13:19.113517 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:13:25.918178 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:13:25.918138 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:13:55.917259 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:13:55.917213 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:14:14.240882 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:14:14.237949 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:14:19.464563 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:14:19.464521 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:14:25.917761 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:14:25.917723 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:14:55.916954 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:14:55.916917 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:15:06.804740 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:15:06.804705 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:15:14.566964 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:15:14.566928 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:15:21.353828 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:15:21.353791 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:15:25.917500 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:15:25.917461 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:15:55.917935 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:15:55.917894 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:16:01.488366 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:01.488327 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:16:14.920671 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:14.920638 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:16:18.566119 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:18.566078 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:16:19.208100 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:19.208069 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:16:19.367151 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:19.367112 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:16:20.145349 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:20.145292 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:16:23.422222 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:23.422193 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:16:23.605874 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:23.605835 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:16:23.753830 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:23.753734 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:16:25.918214 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:16:25.918176 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:16:26.622279 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:26.622249 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:16:26.795326 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:26.795289 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:16:29.563544 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:29.563506 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:16:29.721665 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:29.721625 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:16:29.863481 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:29.863443 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:16:31.984153 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:31.984117 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:16:34.595192 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:34.595155 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:16:39.480599 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:39.480565 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:16:40.341868 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:40.341804 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:16:44.284483 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:44.284445 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:16:48.100569 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:48.100504 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:16:48.676401 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:48.676366 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:16:48.821662 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:48.821630 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:16:48.974373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:48.974284 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:16:49.387120 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:49.387086 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:16:51.573832 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:51.573800 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-8a134f/simulator-single-main-8a134f-kserve-7cbccd79-n9p7s"] Nov 03 23:16:51.577262 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:51.577245 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-8a134f/simulator-single-main-8a134f-kserve-7cbccd79-n9p7s" Nov 03 23:16:51.681124 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:51.681088 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-8a134f\"/\"kube-root-ca.crt\"" Nov 03 23:16:51.681286 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:51.681126 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-8a134f\"/\"openshift-service-ca.crt\"" Nov 03 23:16:51.681286 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:51.681262 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-8a134f\"/\"simulator-single-main-8a134f-kserve-self-signed-certs\"" Nov 03 23:16:51.681368 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:51.681319 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-8a134f\"/\"default-dockercfg-cmklt\"" Nov 03 23:16:51.683126 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:51.683101 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:16:51.683650 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:51.683630 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-8a134f/simulator-single-main-8a134f-kserve-7cbccd79-n9p7s"] Nov 03 23:16:51.683969 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:51.683953 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-782xx\" (UniqueName: \"kubernetes.io/projected/8c4cd87b-b413-47b6-bf22-c1cff144de50-kube-api-access-782xx\") pod \"simulator-single-main-8a134f-kserve-7cbccd79-n9p7s\" (UID: \"8c4cd87b-b413-47b6-bf22-c1cff144de50\") " pod="e2e-models-e2e-worker-single-main-8a134f/simulator-single-main-8a134f-kserve-7cbccd79-n9p7s" Nov 03 23:16:51.684035 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:51.683982 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c4cd87b-b413-47b6-bf22-c1cff144de50-tmp-dir\") pod \"simulator-single-main-8a134f-kserve-7cbccd79-n9p7s\" (UID: \"8c4cd87b-b413-47b6-bf22-c1cff144de50\") " pod="e2e-models-e2e-worker-single-main-8a134f/simulator-single-main-8a134f-kserve-7cbccd79-n9p7s" Nov 03 23:16:51.684035 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:51.684015 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c4cd87b-b413-47b6-bf22-c1cff144de50-model-cache\") pod \"simulator-single-main-8a134f-kserve-7cbccd79-n9p7s\" (UID: \"8c4cd87b-b413-47b6-bf22-c1cff144de50\") " pod="e2e-models-e2e-worker-single-main-8a134f/simulator-single-main-8a134f-kserve-7cbccd79-n9p7s" Nov 03 23:16:51.684144 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:51.684041 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c4cd87b-b413-47b6-bf22-c1cff144de50-tls-certs\") pod \"simulator-single-main-8a134f-kserve-7cbccd79-n9p7s\" (UID: \"8c4cd87b-b413-47b6-bf22-c1cff144de50\") " pod="e2e-models-e2e-worker-single-main-8a134f/simulator-single-main-8a134f-kserve-7cbccd79-n9p7s" Nov 03 23:16:51.684196 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:51.684160 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c4cd87b-b413-47b6-bf22-c1cff144de50-home\") pod \"simulator-single-main-8a134f-kserve-7cbccd79-n9p7s\" (UID: \"8c4cd87b-b413-47b6-bf22-c1cff144de50\") " pod="e2e-models-e2e-worker-single-main-8a134f/simulator-single-main-8a134f-kserve-7cbccd79-n9p7s" Nov 03 23:16:51.684284 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:51.684269 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c4cd87b-b413-47b6-bf22-c1cff144de50-dshm\") pod \"simulator-single-main-8a134f-kserve-7cbccd79-n9p7s\" (UID: \"8c4cd87b-b413-47b6-bf22-c1cff144de50\") " pod="e2e-models-e2e-worker-single-main-8a134f/simulator-single-main-8a134f-kserve-7cbccd79-n9p7s" Nov 03 23:16:51.785113 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:51.785076 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c4cd87b-b413-47b6-bf22-c1cff144de50-dshm\") pod \"simulator-single-main-8a134f-kserve-7cbccd79-n9p7s\" (UID: \"8c4cd87b-b413-47b6-bf22-c1cff144de50\") " pod="e2e-models-e2e-worker-single-main-8a134f/simulator-single-main-8a134f-kserve-7cbccd79-n9p7s" Nov 03 23:16:51.785302 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:51.785198 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-782xx\" (UniqueName: \"kubernetes.io/projected/8c4cd87b-b413-47b6-bf22-c1cff144de50-kube-api-access-782xx\") pod \"simulator-single-main-8a134f-kserve-7cbccd79-n9p7s\" (UID: \"8c4cd87b-b413-47b6-bf22-c1cff144de50\") " pod="e2e-models-e2e-worker-single-main-8a134f/simulator-single-main-8a134f-kserve-7cbccd79-n9p7s" Nov 03 23:16:51.785302 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:51.785237 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c4cd87b-b413-47b6-bf22-c1cff144de50-tmp-dir\") pod \"simulator-single-main-8a134f-kserve-7cbccd79-n9p7s\" (UID: \"8c4cd87b-b413-47b6-bf22-c1cff144de50\") " pod="e2e-models-e2e-worker-single-main-8a134f/simulator-single-main-8a134f-kserve-7cbccd79-n9p7s" Nov 03 23:16:51.785302 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:51.785296 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c4cd87b-b413-47b6-bf22-c1cff144de50-model-cache\") pod \"simulator-single-main-8a134f-kserve-7cbccd79-n9p7s\" (UID: \"8c4cd87b-b413-47b6-bf22-c1cff144de50\") " pod="e2e-models-e2e-worker-single-main-8a134f/simulator-single-main-8a134f-kserve-7cbccd79-n9p7s" Nov 03 23:16:51.785489 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:51.785328 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c4cd87b-b413-47b6-bf22-c1cff144de50-tls-certs\") pod \"simulator-single-main-8a134f-kserve-7cbccd79-n9p7s\" (UID: \"8c4cd87b-b413-47b6-bf22-c1cff144de50\") " pod="e2e-models-e2e-worker-single-main-8a134f/simulator-single-main-8a134f-kserve-7cbccd79-n9p7s" Nov 03 23:16:51.785489 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:51.785370 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c4cd87b-b413-47b6-bf22-c1cff144de50-home\") pod \"simulator-single-main-8a134f-kserve-7cbccd79-n9p7s\" (UID: \"8c4cd87b-b413-47b6-bf22-c1cff144de50\") " pod="e2e-models-e2e-worker-single-main-8a134f/simulator-single-main-8a134f-kserve-7cbccd79-n9p7s" Nov 03 23:16:51.785698 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:51.785673 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c4cd87b-b413-47b6-bf22-c1cff144de50-tmp-dir\") pod \"simulator-single-main-8a134f-kserve-7cbccd79-n9p7s\" (UID: \"8c4cd87b-b413-47b6-bf22-c1cff144de50\") " pod="e2e-models-e2e-worker-single-main-8a134f/simulator-single-main-8a134f-kserve-7cbccd79-n9p7s" Nov 03 23:16:51.785958 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:51.785702 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c4cd87b-b413-47b6-bf22-c1cff144de50-model-cache\") pod \"simulator-single-main-8a134f-kserve-7cbccd79-n9p7s\" (UID: \"8c4cd87b-b413-47b6-bf22-c1cff144de50\") " pod="e2e-models-e2e-worker-single-main-8a134f/simulator-single-main-8a134f-kserve-7cbccd79-n9p7s" Nov 03 23:16:51.785958 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:51.785728 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c4cd87b-b413-47b6-bf22-c1cff144de50-home\") pod \"simulator-single-main-8a134f-kserve-7cbccd79-n9p7s\" (UID: \"8c4cd87b-b413-47b6-bf22-c1cff144de50\") " pod="e2e-models-e2e-worker-single-main-8a134f/simulator-single-main-8a134f-kserve-7cbccd79-n9p7s" Nov 03 23:16:51.787467 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:51.787445 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c4cd87b-b413-47b6-bf22-c1cff144de50-dshm\") pod \"simulator-single-main-8a134f-kserve-7cbccd79-n9p7s\" (UID: \"8c4cd87b-b413-47b6-bf22-c1cff144de50\") " pod="e2e-models-e2e-worker-single-main-8a134f/simulator-single-main-8a134f-kserve-7cbccd79-n9p7s" Nov 03 23:16:51.787905 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:51.787890 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c4cd87b-b413-47b6-bf22-c1cff144de50-tls-certs\") pod \"simulator-single-main-8a134f-kserve-7cbccd79-n9p7s\" (UID: \"8c4cd87b-b413-47b6-bf22-c1cff144de50\") " pod="e2e-models-e2e-worker-single-main-8a134f/simulator-single-main-8a134f-kserve-7cbccd79-n9p7s" Nov 03 23:16:51.847049 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:51.847024 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-782xx\" (UniqueName: \"kubernetes.io/projected/8c4cd87b-b413-47b6-bf22-c1cff144de50-kube-api-access-782xx\") pod \"simulator-single-main-8a134f-kserve-7cbccd79-n9p7s\" (UID: \"8c4cd87b-b413-47b6-bf22-c1cff144de50\") " pod="e2e-models-e2e-worker-single-main-8a134f/simulator-single-main-8a134f-kserve-7cbccd79-n9p7s" Nov 03 23:16:51.888805 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:51.888775 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-8a134f/simulator-single-main-8a134f-kserve-7cbccd79-n9p7s" Nov 03 23:16:52.270838 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:52.270817 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-8a134f/simulator-single-main-8a134f-kserve-7cbccd79-n9p7s"] Nov 03 23:16:52.272899 ip-10-0-143-132 kubenswrapper[5196]: W1103 23:16:52.272871 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c4cd87b_b413_47b6_bf22_c1cff144de50.slice/crio-51d1036cfd14d92e4069bf4c8aaed4d08bfd69c03a210c98e870f37b7f2f18bc WatchSource:0}: Error finding container 51d1036cfd14d92e4069bf4c8aaed4d08bfd69c03a210c98e870f37b7f2f18bc: Status 404 returned error can't find the container with id 51d1036cfd14d92e4069bf4c8aaed4d08bfd69c03a210c98e870f37b7f2f18bc Nov 03 23:16:52.465811 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:52.465771 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-8a134f/simulator-single-main-8a134f-kserve-7cbccd79-n9p7s" event={"ID":"8c4cd87b-b413-47b6-bf22-c1cff144de50","Type":"ContainerStarted","Data":"3504c0ac1ff4997d54fb543c32283f8fecf5cdabda91674af871a4670b52c644"} Nov 03 23:16:52.465811 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:52.465809 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-8a134f/simulator-single-main-8a134f-kserve-7cbccd79-n9p7s" event={"ID":"8c4cd87b-b413-47b6-bf22-c1cff144de50","Type":"ContainerStarted","Data":"51d1036cfd14d92e4069bf4c8aaed4d08bfd69c03a210c98e870f37b7f2f18bc"} Nov 03 23:16:52.615440 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:52.615395 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-8a134f/simulator-single-main-8a134f-kserve-7cbccd79-n9p7s" podStartSLOduration=1.615380995 podStartE2EDuration="1.615380995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:16:52.614698626 +0000 UTC m=+3241.924257919" watchObservedRunningTime="2026-11-03 23:16:52.615380995 +0000 UTC m=+3241.924940289" Nov 03 23:16:52.889354 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:52.889282 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-8a134f/simulator-single-main-8a134f-kserve-7cbccd79-n9p7s" Nov 03 23:16:54.176466 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:54.176433 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:16:55.917840 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:16:55.917802 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:16:58.985790 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:16:58.985755 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:17:01.889689 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:17:01.889655 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-8a134f/simulator-single-main-8a134f-kserve-7cbccd79-n9p7s" Nov 03 23:17:01.902410 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:17:01.902387 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-8a134f/simulator-single-main-8a134f-kserve-7cbccd79-n9p7s" Nov 03 23:17:02.524766 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:17:02.524735 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-8a134f/simulator-single-main-8a134f-kserve-7cbccd79-n9p7s" Nov 03 23:17:03.782779 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:17:03.782733 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:17:08.599533 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:17:08.599493 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:17:13.390133 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:17:13.390094 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:17:15.259438 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:17:15.259401 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:17:18.286729 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:17:18.286697 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:17:20.479456 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:17:20.479424 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:17:23.074572 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:17:23.074528 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:17:23.238335 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:17:23.238288 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:17:25.917411 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:17:25.917379 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:17:27.888968 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:17:27.888939 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:17:32.612161 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:17:32.612126 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:17:37.413640 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:17:37.413602 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:17:42.195406 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:17:42.195378 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:17:46.994280 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:17:46.994247 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:17:50.146504 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:17:50.146473 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:17:51.863790 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:17:51.863761 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:17:55.106965 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:17:55.101918 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-8a134f/simulator-single-main-8a134f-kserve-7cbccd79-n9p7s"] Nov 03 23:17:55.196949 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:17:55.196919 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:17:55.917188 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:17:55.917155 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:17:57.013599 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:17:57.013554 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:18:01.791281 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:01.791248 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:18:02.509159 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:02.509117 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:18:02.656025 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:02.655993 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:18:05.987972 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:05.987925 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:18:07.503264 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:07.503235 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:18:07.785664 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:07.785572 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:18:08.528753 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:08.528705 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:18:10.126794 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:10.126745 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-8a134f/simulator-single-main-8a134f-kserve-7cbccd79-n9p7s" podUID="8c4cd87b-b413-47b6-bf22-c1cff144de50" containerName="main" containerID="cri-o://3504c0ac1ff4997d54fb543c32283f8fecf5cdabda91674af871a4670b52c644" gracePeriod=45 Nov 03 23:18:10.375002 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:10.374976 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-8a134f/simulator-single-main-8a134f-kserve-7cbccd79-n9p7s" Nov 03 23:18:10.496424 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:10.496340 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c4cd87b-b413-47b6-bf22-c1cff144de50-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c4cd87b-b413-47b6-bf22-c1cff144de50-dshm\") pod \"8c4cd87b-b413-47b6-bf22-c1cff144de50\" (UID: \"8c4cd87b-b413-47b6-bf22-c1cff144de50\") " Nov 03 23:18:10.496424 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:10.496387 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c4cd87b-b413-47b6-bf22-c1cff144de50-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c4cd87b-b413-47b6-bf22-c1cff144de50-model-cache\") pod \"8c4cd87b-b413-47b6-bf22-c1cff144de50\" (UID: \"8c4cd87b-b413-47b6-bf22-c1cff144de50\") " Nov 03 23:18:10.496424 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:10.496425 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c4cd87b-b413-47b6-bf22-c1cff144de50-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c4cd87b-b413-47b6-bf22-c1cff144de50-tmp-dir\") pod \"8c4cd87b-b413-47b6-bf22-c1cff144de50\" (UID: \"8c4cd87b-b413-47b6-bf22-c1cff144de50\") " Nov 03 23:18:10.497391 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:10.496445 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c4cd87b-b413-47b6-bf22-c1cff144de50-tls-certs\" (UniqueName: \"kubernetes.io/secret/8c4cd87b-b413-47b6-bf22-c1cff144de50-tls-certs\") pod \"8c4cd87b-b413-47b6-bf22-c1cff144de50\" (UID: \"8c4cd87b-b413-47b6-bf22-c1cff144de50\") " Nov 03 23:18:10.497391 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:10.496481 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c4cd87b-b413-47b6-bf22-c1cff144de50-home\" (UniqueName: \"kubernetes.io/empty-dir/8c4cd87b-b413-47b6-bf22-c1cff144de50-home\") pod \"8c4cd87b-b413-47b6-bf22-c1cff144de50\" (UID: \"8c4cd87b-b413-47b6-bf22-c1cff144de50\") " Nov 03 23:18:10.497391 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:10.496549 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c4cd87b-b413-47b6-bf22-c1cff144de50-kube-api-access-782xx\" (UniqueName: \"kubernetes.io/projected/8c4cd87b-b413-47b6-bf22-c1cff144de50-kube-api-access-782xx\") pod \"8c4cd87b-b413-47b6-bf22-c1cff144de50\" (UID: \"8c4cd87b-b413-47b6-bf22-c1cff144de50\") " Nov 03 23:18:10.497391 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:10.496775 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4cd87b-b413-47b6-bf22-c1cff144de50-model-cache" pod "8c4cd87b-b413-47b6-bf22-c1cff144de50" (UID: "8c4cd87b-b413-47b6-bf22-c1cff144de50"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:18:10.497391 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:10.496789 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4cd87b-b413-47b6-bf22-c1cff144de50-tmp-dir" pod "8c4cd87b-b413-47b6-bf22-c1cff144de50" (UID: "8c4cd87b-b413-47b6-bf22-c1cff144de50"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:18:10.497391 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:10.497178 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4cd87b-b413-47b6-bf22-c1cff144de50-home" pod "8c4cd87b-b413-47b6-bf22-c1cff144de50" (UID: "8c4cd87b-b413-47b6-bf22-c1cff144de50"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:18:10.499242 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:10.499217 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c4cd87b-b413-47b6-bf22-c1cff144de50-tls-certs" pod "8c4cd87b-b413-47b6-bf22-c1cff144de50" (UID: "8c4cd87b-b413-47b6-bf22-c1cff144de50"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:18:10.499371 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:10.499315 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c4cd87b-b413-47b6-bf22-c1cff144de50-kube-api-access-782xx" pod "8c4cd87b-b413-47b6-bf22-c1cff144de50" (UID: "8c4cd87b-b413-47b6-bf22-c1cff144de50"). InnerVolumeSpecName "kube-api-access-782xx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:18:10.499498 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:10.499481 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4cd87b-b413-47b6-bf22-c1cff144de50-dshm" pod "8c4cd87b-b413-47b6-bf22-c1cff144de50" (UID: "8c4cd87b-b413-47b6-bf22-c1cff144de50"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:18:10.597715 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:10.597685 5196 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c4cd87b-b413-47b6-bf22-c1cff144de50-home\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:18:10.597715 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:10.597715 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-782xx\" (UniqueName: \"kubernetes.io/projected/8c4cd87b-b413-47b6-bf22-c1cff144de50-kube-api-access-782xx\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:18:10.597922 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:10.597725 5196 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c4cd87b-b413-47b6-bf22-c1cff144de50-dshm\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:18:10.597922 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:10.597734 5196 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c4cd87b-b413-47b6-bf22-c1cff144de50-model-cache\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:18:10.597922 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:10.597763 5196 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c4cd87b-b413-47b6-bf22-c1cff144de50-tmp-dir\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:18:10.597922 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:10.597772 5196 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c4cd87b-b413-47b6-bf22-c1cff144de50-tls-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:18:10.847580 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:10.847544 5196 generic.go:356] "Generic (PLEG): container finished" podID="8c4cd87b-b413-47b6-bf22-c1cff144de50" containerID="3504c0ac1ff4997d54fb543c32283f8fecf5cdabda91674af871a4670b52c644" exitCode=0 Nov 03 23:18:10.847721 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:10.847590 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-8a134f/simulator-single-main-8a134f-kserve-7cbccd79-n9p7s" Nov 03 23:18:10.847721 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:10.847600 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-8a134f/simulator-single-main-8a134f-kserve-7cbccd79-n9p7s" event={"ID":"8c4cd87b-b413-47b6-bf22-c1cff144de50","Type":"ContainerDied","Data":"3504c0ac1ff4997d54fb543c32283f8fecf5cdabda91674af871a4670b52c644"} Nov 03 23:18:10.847721 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:10.847639 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-8a134f/simulator-single-main-8a134f-kserve-7cbccd79-n9p7s" event={"ID":"8c4cd87b-b413-47b6-bf22-c1cff144de50","Type":"ContainerDied","Data":"51d1036cfd14d92e4069bf4c8aaed4d08bfd69c03a210c98e870f37b7f2f18bc"} Nov 03 23:18:10.847721 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:10.847657 5196 scope.go:122] "RemoveContainer" containerID="3504c0ac1ff4997d54fb543c32283f8fecf5cdabda91674af871a4670b52c644" Nov 03 23:18:10.856077 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:10.856047 5196 scope.go:122] "RemoveContainer" containerID="3504c0ac1ff4997d54fb543c32283f8fecf5cdabda91674af871a4670b52c644" Nov 03 23:18:10.856352 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:18:10.856333 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3504c0ac1ff4997d54fb543c32283f8fecf5cdabda91674af871a4670b52c644\": container with ID starting with 3504c0ac1ff4997d54fb543c32283f8fecf5cdabda91674af871a4670b52c644 not found: ID does not exist" containerID="3504c0ac1ff4997d54fb543c32283f8fecf5cdabda91674af871a4670b52c644" Nov 03 23:18:10.856405 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:10.856360 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3504c0ac1ff4997d54fb543c32283f8fecf5cdabda91674af871a4670b52c644"} err="failed to get container status \"3504c0ac1ff4997d54fb543c32283f8fecf5cdabda91674af871a4670b52c644\": rpc error: code = NotFound desc = could not find container \"3504c0ac1ff4997d54fb543c32283f8fecf5cdabda91674af871a4670b52c644\": container with ID starting with 3504c0ac1ff4997d54fb543c32283f8fecf5cdabda91674af871a4670b52c644 not found: ID does not exist" Nov 03 23:18:11.015202 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:11.015168 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-8a134f/simulator-single-main-8a134f-kserve-7cbccd79-n9p7s"] Nov 03 23:18:11.018703 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:11.018680 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-8a134f/simulator-single-main-8a134f-kserve-7cbccd79-n9p7s"] Nov 03 23:18:12.813583 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:12.813550 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c4cd87b-b413-47b6-bf22-c1cff144de50" path="/var/lib/kubelet/pods/8c4cd87b-b413-47b6-bf22-c1cff144de50/volumes" Nov 03 23:18:14.563399 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:14.563362 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:18:14.712364 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:14.712332 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:18:14.860240 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:14.860206 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:18:15.615172 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:15.615126 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:18:20.850326 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:20.850293 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:18:25.918000 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:18:25.917963 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:18:31.242611 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:31.242536 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:18:32.468335 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:32.468287 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:18:38.238948 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:18:38.238894 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:18:55.917764 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:18:55.917724 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:19:15.994185 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:19:15.994147 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:19:21.194991 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:19:21.194955 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:19:25.917506 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:19:25.917463 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:19:55.918432 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:19:55.918395 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:20:16.301954 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:20:16.301852 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:20:21.543047 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:20:21.543013 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:20:25.917203 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:20:25.917155 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:20:55.918051 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:20:55.917977 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:21:03.852304 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:03.852254 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:21:04.460143 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:04.460111 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:21:05.045110 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:05.045048 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:21:09.183366 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:09.183334 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:21:09.358112 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:09.358077 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:21:09.502110 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:09.502030 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:21:12.409731 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:12.409696 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:21:12.578403 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:12.578372 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:21:14.845553 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:14.845519 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:21:15.433805 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:15.433770 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:21:15.597280 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:15.597240 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:21:15.741037 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:15.740949 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:21:16.641541 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:16.641513 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:21:20.389625 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:20.389587 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:21:21.895708 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:21.895664 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:21:25.294118 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:25.294079 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:21:25.917383 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:21:25.917345 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:21:30.074269 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:30.074158 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:21:34.056731 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:34.056694 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:21:34.611350 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:34.611318 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:21:34.766353 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:34.766315 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:21:35.199566 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:35.199537 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:21:37.584989 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:37.584954 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74"] Nov 03 23:21:37.585474 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:37.585416 5196 cpu_manager.go:414] "containerMap: removing container" podUID="8c4cd87b-b413-47b6-bf22-c1cff144de50" containerName="main" Nov 03 23:21:37.585474 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:37.585436 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c4cd87b-b413-47b6-bf22-c1cff144de50" containerName="main" Nov 03 23:21:37.585576 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:37.585567 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c4cd87b-b413-47b6-bf22-c1cff144de50" containerName="main" Nov 03 23:21:37.588969 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:37.588952 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74" Nov 03 23:21:37.623683 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:37.623654 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cabdb32-82f9-473b-b0e0-d57093fd7265-dshm\") pod \"simulator-single-main-60ef36-kserve-56f5885d66-9xw74\" (UID: \"6cabdb32-82f9-473b-b0e0-d57093fd7265\") " pod="e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74" Nov 03 23:21:37.623845 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:37.623701 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cabdb32-82f9-473b-b0e0-d57093fd7265-model-cache\") pod \"simulator-single-main-60ef36-kserve-56f5885d66-9xw74\" (UID: \"6cabdb32-82f9-473b-b0e0-d57093fd7265\") " pod="e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74" Nov 03 23:21:37.623845 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:37.623728 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bwg9\" (UniqueName: \"kubernetes.io/projected/6cabdb32-82f9-473b-b0e0-d57093fd7265-kube-api-access-5bwg9\") pod \"simulator-single-main-60ef36-kserve-56f5885d66-9xw74\" (UID: \"6cabdb32-82f9-473b-b0e0-d57093fd7265\") " pod="e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74" Nov 03 23:21:37.623976 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:37.623894 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cabdb32-82f9-473b-b0e0-d57093fd7265-tmp-dir\") pod \"simulator-single-main-60ef36-kserve-56f5885d66-9xw74\" (UID: \"6cabdb32-82f9-473b-b0e0-d57093fd7265\") " pod="e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74" Nov 03 23:21:37.623976 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:37.623926 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cabdb32-82f9-473b-b0e0-d57093fd7265-tls-certs\") pod \"simulator-single-main-60ef36-kserve-56f5885d66-9xw74\" (UID: \"6cabdb32-82f9-473b-b0e0-d57093fd7265\") " pod="e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74" Nov 03 23:21:37.623976 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:37.623950 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cabdb32-82f9-473b-b0e0-d57093fd7265-home\") pod \"simulator-single-main-60ef36-kserve-56f5885d66-9xw74\" (UID: \"6cabdb32-82f9-473b-b0e0-d57093fd7265\") " pod="e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74" Nov 03 23:21:37.674541 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:37.674509 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:21:37.692827 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:37.692804 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-60ef36\"/\"kube-root-ca.crt\"" Nov 03 23:21:37.692916 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:37.692815 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-60ef36\"/\"openshift-service-ca.crt\"" Nov 03 23:21:37.693060 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:37.693044 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-60ef36\"/\"default-dockercfg-7mwt4\"" Nov 03 23:21:37.693180 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:37.693163 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-60ef36\"/\"simulator-single-main-60ef36-kserve-self-signed-certs\"" Nov 03 23:21:37.694907 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:37.694884 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74"] Nov 03 23:21:37.724708 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:37.724681 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cabdb32-82f9-473b-b0e0-d57093fd7265-model-cache\") pod \"simulator-single-main-60ef36-kserve-56f5885d66-9xw74\" (UID: \"6cabdb32-82f9-473b-b0e0-d57093fd7265\") " pod="e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74" Nov 03 23:21:37.724840 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:37.724722 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bwg9\" (UniqueName: \"kubernetes.io/projected/6cabdb32-82f9-473b-b0e0-d57093fd7265-kube-api-access-5bwg9\") pod \"simulator-single-main-60ef36-kserve-56f5885d66-9xw74\" (UID: \"6cabdb32-82f9-473b-b0e0-d57093fd7265\") " pod="e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74" Nov 03 23:21:37.724840 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:37.724748 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cabdb32-82f9-473b-b0e0-d57093fd7265-tmp-dir\") pod \"simulator-single-main-60ef36-kserve-56f5885d66-9xw74\" (UID: \"6cabdb32-82f9-473b-b0e0-d57093fd7265\") " pod="e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74" Nov 03 23:21:37.724840 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:37.724786 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cabdb32-82f9-473b-b0e0-d57093fd7265-tls-certs\") pod \"simulator-single-main-60ef36-kserve-56f5885d66-9xw74\" (UID: \"6cabdb32-82f9-473b-b0e0-d57093fd7265\") " pod="e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74" Nov 03 23:21:37.724840 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:37.724805 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cabdb32-82f9-473b-b0e0-d57093fd7265-home\") pod \"simulator-single-main-60ef36-kserve-56f5885d66-9xw74\" (UID: \"6cabdb32-82f9-473b-b0e0-d57093fd7265\") " pod="e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74" Nov 03 23:21:37.725253 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:37.724875 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cabdb32-82f9-473b-b0e0-d57093fd7265-dshm\") pod \"simulator-single-main-60ef36-kserve-56f5885d66-9xw74\" (UID: \"6cabdb32-82f9-473b-b0e0-d57093fd7265\") " pod="e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74" Nov 03 23:21:37.725253 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:37.725194 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cabdb32-82f9-473b-b0e0-d57093fd7265-model-cache\") pod \"simulator-single-main-60ef36-kserve-56f5885d66-9xw74\" (UID: \"6cabdb32-82f9-473b-b0e0-d57093fd7265\") " pod="e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74" Nov 03 23:21:37.725253 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:37.725220 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cabdb32-82f9-473b-b0e0-d57093fd7265-tmp-dir\") pod \"simulator-single-main-60ef36-kserve-56f5885d66-9xw74\" (UID: \"6cabdb32-82f9-473b-b0e0-d57093fd7265\") " pod="e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74" Nov 03 23:21:37.725611 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:37.725519 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cabdb32-82f9-473b-b0e0-d57093fd7265-home\") pod \"simulator-single-main-60ef36-kserve-56f5885d66-9xw74\" (UID: \"6cabdb32-82f9-473b-b0e0-d57093fd7265\") " pod="e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74" Nov 03 23:21:37.727814 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:37.727788 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cabdb32-82f9-473b-b0e0-d57093fd7265-dshm\") pod \"simulator-single-main-60ef36-kserve-56f5885d66-9xw74\" (UID: \"6cabdb32-82f9-473b-b0e0-d57093fd7265\") " pod="e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74" Nov 03 23:21:37.728101 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:37.728083 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cabdb32-82f9-473b-b0e0-d57093fd7265-tls-certs\") pod \"simulator-single-main-60ef36-kserve-56f5885d66-9xw74\" (UID: \"6cabdb32-82f9-473b-b0e0-d57093fd7265\") " pod="e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74" Nov 03 23:21:37.787361 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:37.787323 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bwg9\" (UniqueName: \"kubernetes.io/projected/6cabdb32-82f9-473b-b0e0-d57093fd7265-kube-api-access-5bwg9\") pod \"simulator-single-main-60ef36-kserve-56f5885d66-9xw74\" (UID: \"6cabdb32-82f9-473b-b0e0-d57093fd7265\") " pod="e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74" Nov 03 23:21:37.900989 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:37.900950 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74" Nov 03 23:21:38.287144 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:38.287067 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74"] Nov 03 23:21:38.290368 ip-10-0-143-132 kubenswrapper[5196]: W1103 23:21:38.290335 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6cabdb32_82f9_473b_b0e0_d57093fd7265.slice/crio-46b79a10f1e49e2d1dadc372e39271144844063509c5436d1edaa15b1f837807 WatchSource:0}: Error finding container 46b79a10f1e49e2d1dadc372e39271144844063509c5436d1edaa15b1f837807: Status 404 returned error can't find the container with id 46b79a10f1e49e2d1dadc372e39271144844063509c5436d1edaa15b1f837807 Nov 03 23:21:38.928463 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:38.928428 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74" event={"ID":"6cabdb32-82f9-473b-b0e0-d57093fd7265","Type":"ContainerStarted","Data":"2f145396730bb62ddc1e48da3b86773455656591e3b45cdb574f8a775fbc6b92"} Nov 03 23:21:38.928463 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:38.928469 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74" event={"ID":"6cabdb32-82f9-473b-b0e0-d57093fd7265","Type":"ContainerStarted","Data":"46b79a10f1e49e2d1dadc372e39271144844063509c5436d1edaa15b1f837807"} Nov 03 23:21:39.047233 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:39.047180 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74" podStartSLOduration=2.047167188 podStartE2EDuration="2.047167188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:21:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:21:39.046926271 +0000 UTC m=+3528.356485563" watchObservedRunningTime="2026-11-03 23:21:39.047167188 +0000 UTC m=+3528.356726512" Nov 03 23:21:39.901374 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:39.901344 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74" Nov 03 23:21:39.996191 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:39.996159 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:21:44.786971 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:44.786935 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:21:47.901482 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:47.901451 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74" Nov 03 23:21:47.913924 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:47.913903 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74" Nov 03 23:21:47.981930 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:47.981908 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74" Nov 03 23:21:49.586454 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:49.586423 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:21:54.396886 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:54.396836 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:21:55.918237 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:21:55.918202 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:21:59.196072 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:21:59.196044 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:22:00.942665 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:00.942632 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:22:04.001227 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:04.001192 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:22:08.818694 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:08.818664 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:22:13.625548 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:13.625486 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:22:16.991244 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:16.991211 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:22:18.394148 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:18.394116 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:22:22.232502 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:22.232456 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:22:23.441632 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:23.441601 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:22:25.917826 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:22:25.917784 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:22:27.994198 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:27.994159 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:22:32.798052 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:32.798011 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:22:33.763740 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:33.763702 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:22:37.888338 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:37.888305 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:22:38.726888 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:38.726829 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74"] Nov 03 23:22:38.822319 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:38.822284 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:22:42.876638 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:42.876608 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:22:46.098681 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:46.098646 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:22:46.245663 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:46.245604 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:22:47.679681 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:47.679648 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:22:51.893200 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:51.893157 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:22:53.460631 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:53.460596 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:22:53.746902 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:53.746801 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74" podUID="6cabdb32-82f9-473b-b0e0-d57093fd7265" containerName="main" containerID="cri-o://2f145396730bb62ddc1e48da3b86773455656591e3b45cdb574f8a775fbc6b92" gracePeriod=45 Nov 03 23:22:53.812491 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:53.812467 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:22:53.997137 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:53.997080 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74" Nov 03 23:22:54.126061 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:54.126031 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cabdb32-82f9-473b-b0e0-d57093fd7265-tls-certs\" (UniqueName: \"kubernetes.io/secret/6cabdb32-82f9-473b-b0e0-d57093fd7265-tls-certs\") pod \"6cabdb32-82f9-473b-b0e0-d57093fd7265\" (UID: \"6cabdb32-82f9-473b-b0e0-d57093fd7265\") " Nov 03 23:22:54.126237 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:54.126079 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cabdb32-82f9-473b-b0e0-d57093fd7265-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cabdb32-82f9-473b-b0e0-d57093fd7265-tmp-dir\") pod \"6cabdb32-82f9-473b-b0e0-d57093fd7265\" (UID: \"6cabdb32-82f9-473b-b0e0-d57093fd7265\") " Nov 03 23:22:54.126237 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:54.126114 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cabdb32-82f9-473b-b0e0-d57093fd7265-home\" (UniqueName: \"kubernetes.io/empty-dir/6cabdb32-82f9-473b-b0e0-d57093fd7265-home\") pod \"6cabdb32-82f9-473b-b0e0-d57093fd7265\" (UID: \"6cabdb32-82f9-473b-b0e0-d57093fd7265\") " Nov 03 23:22:54.126237 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:54.126155 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cabdb32-82f9-473b-b0e0-d57093fd7265-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cabdb32-82f9-473b-b0e0-d57093fd7265-dshm\") pod \"6cabdb32-82f9-473b-b0e0-d57093fd7265\" (UID: \"6cabdb32-82f9-473b-b0e0-d57093fd7265\") " Nov 03 23:22:54.126237 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:54.126193 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cabdb32-82f9-473b-b0e0-d57093fd7265-kube-api-access-5bwg9\" (UniqueName: \"kubernetes.io/projected/6cabdb32-82f9-473b-b0e0-d57093fd7265-kube-api-access-5bwg9\") pod \"6cabdb32-82f9-473b-b0e0-d57093fd7265\" (UID: \"6cabdb32-82f9-473b-b0e0-d57093fd7265\") " Nov 03 23:22:54.126237 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:54.126209 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cabdb32-82f9-473b-b0e0-d57093fd7265-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cabdb32-82f9-473b-b0e0-d57093fd7265-model-cache\") pod \"6cabdb32-82f9-473b-b0e0-d57093fd7265\" (UID: \"6cabdb32-82f9-473b-b0e0-d57093fd7265\") " Nov 03 23:22:54.126522 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:54.126401 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cabdb32-82f9-473b-b0e0-d57093fd7265-tmp-dir" pod "6cabdb32-82f9-473b-b0e0-d57093fd7265" (UID: "6cabdb32-82f9-473b-b0e0-d57093fd7265"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:22:54.126522 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:54.126407 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cabdb32-82f9-473b-b0e0-d57093fd7265-home" pod "6cabdb32-82f9-473b-b0e0-d57093fd7265" (UID: "6cabdb32-82f9-473b-b0e0-d57093fd7265"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:22:54.126620 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:54.126569 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cabdb32-82f9-473b-b0e0-d57093fd7265-model-cache" pod "6cabdb32-82f9-473b-b0e0-d57093fd7265" (UID: "6cabdb32-82f9-473b-b0e0-d57093fd7265"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:22:54.128535 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:54.128504 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cabdb32-82f9-473b-b0e0-d57093fd7265-dshm" pod "6cabdb32-82f9-473b-b0e0-d57093fd7265" (UID: "6cabdb32-82f9-473b-b0e0-d57093fd7265"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:22:54.128664 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:54.128565 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cabdb32-82f9-473b-b0e0-d57093fd7265-tls-certs" pod "6cabdb32-82f9-473b-b0e0-d57093fd7265" (UID: "6cabdb32-82f9-473b-b0e0-d57093fd7265"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:22:54.128664 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:54.128577 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cabdb32-82f9-473b-b0e0-d57093fd7265-kube-api-access-5bwg9" pod "6cabdb32-82f9-473b-b0e0-d57093fd7265" (UID: "6cabdb32-82f9-473b-b0e0-d57093fd7265"). InnerVolumeSpecName "kube-api-access-5bwg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:22:54.227194 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:54.227163 5196 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cabdb32-82f9-473b-b0e0-d57093fd7265-home\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:22:54.227194 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:54.227191 5196 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cabdb32-82f9-473b-b0e0-d57093fd7265-dshm\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:22:54.227400 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:54.227210 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bwg9\" (UniqueName: \"kubernetes.io/projected/6cabdb32-82f9-473b-b0e0-d57093fd7265-kube-api-access-5bwg9\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:22:54.227400 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:54.227221 5196 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cabdb32-82f9-473b-b0e0-d57093fd7265-model-cache\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:22:54.227400 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:54.227230 5196 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cabdb32-82f9-473b-b0e0-d57093fd7265-tls-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:22:54.227400 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:54.227238 5196 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cabdb32-82f9-473b-b0e0-d57093fd7265-tmp-dir\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:22:54.300133 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:54.300046 5196 generic.go:356] "Generic (PLEG): container finished" podID="6cabdb32-82f9-473b-b0e0-d57093fd7265" containerID="2f145396730bb62ddc1e48da3b86773455656591e3b45cdb574f8a775fbc6b92" exitCode=0 Nov 03 23:22:54.300133 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:54.300095 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74" event={"ID":"6cabdb32-82f9-473b-b0e0-d57093fd7265","Type":"ContainerDied","Data":"2f145396730bb62ddc1e48da3b86773455656591e3b45cdb574f8a775fbc6b92"} Nov 03 23:22:54.300133 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:54.300121 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74" Nov 03 23:22:54.300482 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:54.300148 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74" event={"ID":"6cabdb32-82f9-473b-b0e0-d57093fd7265","Type":"ContainerDied","Data":"46b79a10f1e49e2d1dadc372e39271144844063509c5436d1edaa15b1f837807"} Nov 03 23:22:54.300482 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:54.300168 5196 scope.go:122] "RemoveContainer" containerID="2f145396730bb62ddc1e48da3b86773455656591e3b45cdb574f8a775fbc6b92" Nov 03 23:22:54.309797 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:54.309781 5196 scope.go:122] "RemoveContainer" containerID="2f145396730bb62ddc1e48da3b86773455656591e3b45cdb574f8a775fbc6b92" Nov 03 23:22:54.310079 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:22:54.310059 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f145396730bb62ddc1e48da3b86773455656591e3b45cdb574f8a775fbc6b92\": container with ID starting with 2f145396730bb62ddc1e48da3b86773455656591e3b45cdb574f8a775fbc6b92 not found: ID does not exist" containerID="2f145396730bb62ddc1e48da3b86773455656591e3b45cdb574f8a775fbc6b92" Nov 03 23:22:54.310150 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:54.310090 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f145396730bb62ddc1e48da3b86773455656591e3b45cdb574f8a775fbc6b92"} err="failed to get container status \"2f145396730bb62ddc1e48da3b86773455656591e3b45cdb574f8a775fbc6b92\": rpc error: code = NotFound desc = could not find container \"2f145396730bb62ddc1e48da3b86773455656591e3b45cdb574f8a775fbc6b92\": container with ID starting with 2f145396730bb62ddc1e48da3b86773455656591e3b45cdb574f8a775fbc6b92 not found: ID does not exist" Nov 03 23:22:54.465505 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:54.465474 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74"] Nov 03 23:22:54.470123 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:54.470098 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74"] Nov 03 23:22:54.525677 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:54.525650 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:22:54.814043 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:54.814010 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cabdb32-82f9-473b-b0e0-d57093fd7265" path="/var/lib/kubelet/pods/6cabdb32-82f9-473b-b0e0-d57093fd7265/volumes" Nov 03 23:22:54.972077 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:22:54.972039 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-60ef36/simulator-single-main-60ef36-kserve-56f5885d66-9xw74" podUID="6cabdb32-82f9-473b-b0e0-d57093fd7265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.131:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:22:55.918033 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:22:55.917999 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:23:00.500951 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:00.500806 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:23:00.641126 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:00.641093 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:23:00.834705 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:00.834615 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:23:07.053192 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:07.053161 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:23:15.014074 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:15.014029 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cdxtr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:23:15.014852 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:15.014800 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cdxtr" podUID="07738db5-20e8-47f1-8339-82866bfa91fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:23:15.371129 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:15.371091 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:23:15.868061 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:15.868029 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:23:17.339811 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:17.339781 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:23:22.582005 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:22.581972 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:23:25.918602 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:23:25.918561 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:23:30.099217 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:30.099182 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:23:48.800199 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:48.800169 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp"] Nov 03 23:23:48.800708 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:48.800497 5196 cpu_manager.go:414] "containerMap: removing container" podUID="6cabdb32-82f9-473b-b0e0-d57093fd7265" containerName="main" Nov 03 23:23:48.800708 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:48.800508 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cabdb32-82f9-473b-b0e0-d57093fd7265" containerName="main" Nov 03 23:23:48.800708 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:48.800578 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cabdb32-82f9-473b-b0e0-d57093fd7265" containerName="main" Nov 03 23:23:48.803655 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:48.803639 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp" Nov 03 23:23:48.852568 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:48.852534 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-model-cache\") pod \"e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp\" (UID: \"2cbf1ba2-93d9-4a31-927f-e60ed20baebb\") " pod="llm/e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp" Nov 03 23:23:48.852731 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:48.852664 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrf9x\" (UniqueName: \"kubernetes.io/projected/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-kube-api-access-nrf9x\") pod \"e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp\" (UID: \"2cbf1ba2-93d9-4a31-927f-e60ed20baebb\") " pod="llm/e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp" Nov 03 23:23:48.852781 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:48.852726 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-dshm\") pod \"e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp\" (UID: \"2cbf1ba2-93d9-4a31-927f-e60ed20baebb\") " pod="llm/e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp" Nov 03 23:23:48.852781 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:48.852766 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-home\") pod \"e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp\" (UID: \"2cbf1ba2-93d9-4a31-927f-e60ed20baebb\") " pod="llm/e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp" Nov 03 23:23:48.852916 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:48.852895 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-tmp-dir\") pod \"e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp\" (UID: \"2cbf1ba2-93d9-4a31-927f-e60ed20baebb\") " pod="llm/e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp" Nov 03 23:23:48.852966 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:48.852924 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-tls-certs\") pod \"e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp\" (UID: \"2cbf1ba2-93d9-4a31-927f-e60ed20baebb\") " pod="llm/e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp" Nov 03 23:23:48.862135 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:48.862114 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 23:23:48.912981 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:48.912949 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-374e2aad-kserve-self-signed-certs\"" Nov 03 23:23:48.912981 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:48.912954 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-h7bmr\"" Nov 03 23:23:48.913182 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:48.913132 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 23:23:48.914709 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:48.914690 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp"] Nov 03 23:23:48.953653 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:48.953622 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrf9x\" (UniqueName: \"kubernetes.io/projected/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-kube-api-access-nrf9x\") pod \"e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp\" (UID: \"2cbf1ba2-93d9-4a31-927f-e60ed20baebb\") " pod="llm/e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp" Nov 03 23:23:48.953870 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:48.953666 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-dshm\") pod \"e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp\" (UID: \"2cbf1ba2-93d9-4a31-927f-e60ed20baebb\") " pod="llm/e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp" Nov 03 23:23:48.953870 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:48.953684 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-home\") pod \"e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp\" (UID: \"2cbf1ba2-93d9-4a31-927f-e60ed20baebb\") " pod="llm/e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp" Nov 03 23:23:48.953870 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:48.953724 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-tmp-dir\") pod \"e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp\" (UID: \"2cbf1ba2-93d9-4a31-927f-e60ed20baebb\") " pod="llm/e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp" Nov 03 23:23:48.953870 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:48.953747 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-tls-certs\") pod \"e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp\" (UID: \"2cbf1ba2-93d9-4a31-927f-e60ed20baebb\") " pod="llm/e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp" Nov 03 23:23:48.953870 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:48.953812 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-model-cache\") pod \"e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp\" (UID: \"2cbf1ba2-93d9-4a31-927f-e60ed20baebb\") " pod="llm/e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp" Nov 03 23:23:48.954255 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:48.954226 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-home\") pod \"e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp\" (UID: \"2cbf1ba2-93d9-4a31-927f-e60ed20baebb\") " pod="llm/e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp" Nov 03 23:23:48.954377 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:48.954235 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-model-cache\") pod \"e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp\" (UID: \"2cbf1ba2-93d9-4a31-927f-e60ed20baebb\") " pod="llm/e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp" Nov 03 23:23:48.954377 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:48.954283 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-tmp-dir\") pod \"e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp\" (UID: \"2cbf1ba2-93d9-4a31-927f-e60ed20baebb\") " pod="llm/e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp" Nov 03 23:23:48.956225 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:48.956205 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-dshm\") pod \"e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp\" (UID: \"2cbf1ba2-93d9-4a31-927f-e60ed20baebb\") " pod="llm/e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp" Nov 03 23:23:48.956659 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:48.956642 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-tls-certs\") pod \"e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp\" (UID: \"2cbf1ba2-93d9-4a31-927f-e60ed20baebb\") " pod="llm/e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp" Nov 03 23:23:48.991540 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:48.991517 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:23:49.018389 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:49.018362 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrf9x\" (UniqueName: \"kubernetes.io/projected/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-kube-api-access-nrf9x\") pod \"e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp\" (UID: \"2cbf1ba2-93d9-4a31-927f-e60ed20baebb\") " pod="llm/e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp" Nov 03 23:23:49.116206 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:49.116177 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp" Nov 03 23:23:49.493700 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:49.493641 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp"] Nov 03 23:23:49.495842 ip-10-0-143-132 kubenswrapper[5196]: W1103 23:23:49.495811 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cbf1ba2_93d9_4a31_927f_e60ed20baebb.slice/crio-150e7f0e46c178e93ecdb8d489a4f115c3517b0d37bd379fc51a6e3e7449e5bd WatchSource:0}: Error finding container 150e7f0e46c178e93ecdb8d489a4f115c3517b0d37bd379fc51a6e3e7449e5bd: Status 404 returned error can't find the container with id 150e7f0e46c178e93ecdb8d489a4f115c3517b0d37bd379fc51a6e3e7449e5bd Nov 03 23:23:49.546233 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:49.546204 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:23:49.586035 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:49.585998 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp" event={"ID":"2cbf1ba2-93d9-4a31-927f-e60ed20baebb","Type":"ContainerStarted","Data":"9946227879fdc43059109c1f37fbcd12cd68d5d5bf2ac72511b91113d4030abc"} Nov 03 23:23:49.586193 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:49.586048 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp" event={"ID":"2cbf1ba2-93d9-4a31-927f-e60ed20baebb","Type":"ContainerStarted","Data":"150e7f0e46c178e93ecdb8d489a4f115c3517b0d37bd379fc51a6e3e7449e5bd"} Nov 03 23:23:49.695069 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:49.695038 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:23:49.705669 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:49.705619 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp" podStartSLOduration=1.705606255 podStartE2EDuration="1.705606255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:23:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:23:49.703960312 +0000 UTC m=+3659.013519604" watchObservedRunningTime="2026-11-03 23:23:49.705606255 +0000 UTC m=+3659.015165584" Nov 03 23:23:50.116777 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:50.116745 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp" Nov 03 23:23:55.989370 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:23:55.989053 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:23:59.116751 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:59.116719 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp" Nov 03 23:23:59.129679 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:59.129652 5196 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp" Nov 03 23:23:59.662058 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:23:59.662024 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp" Nov 03 23:24:17.704589 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:24:17.704525 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:24:22.923977 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:24:22.923937 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:24:25.918237 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:24:25.918197 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:24:28.521449 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:24:28.520963 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-njmdj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:24:28.521449 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:24:28.521047 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" podUID="9f644685-a2d7-48ee-833d-d49c9959f598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:24:28.876048 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:24:28.876008 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8q2rz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:24:28.876294 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:24:28.876079 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" podUID="6ef06915-cf1c-45cb-b61b-8c80c45f1dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:24:37.224316 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:24:37.224278 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:24:43.526022 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:24:43.525978 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-njmdj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:24:43.526695 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:24:43.526644 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-njmdj" podUID="9f644685-a2d7-48ee-833d-d49c9959f598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:24:43.875058 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:24:43.875017 5196 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8q2rz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:24:43.875240 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:24:43.875083 5196 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8q2rz" podUID="6ef06915-cf1c-45cb-b61b-8c80c45f1dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:24:49.751246 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:24:49.751212 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp"] Nov 03 23:24:49.767304 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:24:49.767271 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:24:55.918345 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:24:55.918304 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:25:04.772015 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:04.771960 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp" podUID="2cbf1ba2-93d9-4a31-927f-e60ed20baebb" containerName="main" containerID="cri-o://9946227879fdc43059109c1f37fbcd12cd68d5d5bf2ac72511b91113d4030abc" gracePeriod=45 Nov 03 23:25:05.023800 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:05.023745 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp" Nov 03 23:25:05.053157 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:05.053127 5196 generic.go:356] "Generic (PLEG): container finished" podID="2cbf1ba2-93d9-4a31-927f-e60ed20baebb" containerID="9946227879fdc43059109c1f37fbcd12cd68d5d5bf2ac72511b91113d4030abc" exitCode=0 Nov 03 23:25:05.053296 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:05.053162 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp" Nov 03 23:25:05.053296 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:05.053215 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp" event={"ID":"2cbf1ba2-93d9-4a31-927f-e60ed20baebb","Type":"ContainerDied","Data":"9946227879fdc43059109c1f37fbcd12cd68d5d5bf2ac72511b91113d4030abc"} Nov 03 23:25:05.053296 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:05.053263 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp" event={"ID":"2cbf1ba2-93d9-4a31-927f-e60ed20baebb","Type":"ContainerDied","Data":"150e7f0e46c178e93ecdb8d489a4f115c3517b0d37bd379fc51a6e3e7449e5bd"} Nov 03 23:25:05.053296 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:05.053285 5196 scope.go:122] "RemoveContainer" containerID="9946227879fdc43059109c1f37fbcd12cd68d5d5bf2ac72511b91113d4030abc" Nov 03 23:25:05.061680 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:05.061665 5196 scope.go:122] "RemoveContainer" containerID="9946227879fdc43059109c1f37fbcd12cd68d5d5bf2ac72511b91113d4030abc" Nov 03 23:25:05.061972 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:25:05.061954 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9946227879fdc43059109c1f37fbcd12cd68d5d5bf2ac72511b91113d4030abc\": container with ID starting with 9946227879fdc43059109c1f37fbcd12cd68d5d5bf2ac72511b91113d4030abc not found: ID does not exist" containerID="9946227879fdc43059109c1f37fbcd12cd68d5d5bf2ac72511b91113d4030abc" Nov 03 23:25:05.062034 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:05.061982 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9946227879fdc43059109c1f37fbcd12cd68d5d5bf2ac72511b91113d4030abc"} err="failed to get container status \"9946227879fdc43059109c1f37fbcd12cd68d5d5bf2ac72511b91113d4030abc\": rpc error: code = NotFound desc = could not find container \"9946227879fdc43059109c1f37fbcd12cd68d5d5bf2ac72511b91113d4030abc\": container with ID starting with 9946227879fdc43059109c1f37fbcd12cd68d5d5bf2ac72511b91113d4030abc not found: ID does not exist" Nov 03 23:25:05.103110 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:05.103091 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-kube-api-access-nrf9x\" (UniqueName: \"kubernetes.io/projected/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-kube-api-access-nrf9x\") pod \"2cbf1ba2-93d9-4a31-927f-e60ed20baebb\" (UID: \"2cbf1ba2-93d9-4a31-927f-e60ed20baebb\") " Nov 03 23:25:05.103214 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:05.103121 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-home\" (UniqueName: \"kubernetes.io/empty-dir/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-home\") pod \"2cbf1ba2-93d9-4a31-927f-e60ed20baebb\" (UID: \"2cbf1ba2-93d9-4a31-927f-e60ed20baebb\") " Nov 03 23:25:05.103214 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:05.103155 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-tls-certs\" (UniqueName: \"kubernetes.io/secret/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-tls-certs\") pod \"2cbf1ba2-93d9-4a31-927f-e60ed20baebb\" (UID: \"2cbf1ba2-93d9-4a31-927f-e60ed20baebb\") " Nov 03 23:25:05.103214 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:05.103191 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-dshm\") pod \"2cbf1ba2-93d9-4a31-927f-e60ed20baebb\" (UID: \"2cbf1ba2-93d9-4a31-927f-e60ed20baebb\") " Nov 03 23:25:05.103384 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:05.103236 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-tmp-dir\") pod \"2cbf1ba2-93d9-4a31-927f-e60ed20baebb\" (UID: \"2cbf1ba2-93d9-4a31-927f-e60ed20baebb\") " Nov 03 23:25:05.103384 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:05.103265 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-model-cache\") pod \"2cbf1ba2-93d9-4a31-927f-e60ed20baebb\" (UID: \"2cbf1ba2-93d9-4a31-927f-e60ed20baebb\") " Nov 03 23:25:05.103554 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:05.103399 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-home" pod "2cbf1ba2-93d9-4a31-927f-e60ed20baebb" (UID: "2cbf1ba2-93d9-4a31-927f-e60ed20baebb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:25:05.103554 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:05.103543 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-tmp-dir" pod "2cbf1ba2-93d9-4a31-927f-e60ed20baebb" (UID: "2cbf1ba2-93d9-4a31-927f-e60ed20baebb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:25:05.103670 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:05.103560 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-model-cache" pod "2cbf1ba2-93d9-4a31-927f-e60ed20baebb" (UID: "2cbf1ba2-93d9-4a31-927f-e60ed20baebb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:25:05.103670 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:05.103605 5196 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-home\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:25:05.105500 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:05.105476 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-kube-api-access-nrf9x" pod "2cbf1ba2-93d9-4a31-927f-e60ed20baebb" (UID: "2cbf1ba2-93d9-4a31-927f-e60ed20baebb"). InnerVolumeSpecName "kube-api-access-nrf9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:25:05.105837 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:05.105816 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-dshm" pod "2cbf1ba2-93d9-4a31-927f-e60ed20baebb" (UID: "2cbf1ba2-93d9-4a31-927f-e60ed20baebb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:25:05.105950 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:05.105906 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-tls-certs" pod "2cbf1ba2-93d9-4a31-927f-e60ed20baebb" (UID: "2cbf1ba2-93d9-4a31-927f-e60ed20baebb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:25:05.204224 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:05.204200 5196 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-model-cache\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:25:05.204224 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:05.204224 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrf9x\" (UniqueName: \"kubernetes.io/projected/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-kube-api-access-nrf9x\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:25:05.204407 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:05.204236 5196 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-tls-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:25:05.204407 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:05.204244 5196 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-dshm\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:25:05.204407 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:05.204253 5196 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cbf1ba2-93d9-4a31-927f-e60ed20baebb-tmp-dir\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:25:05.474764 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:05.474731 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp"] Nov 03 23:25:05.478490 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:05.478463 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-374e2aad-kserve-5b6b6cd9fc-84dkp"] Nov 03 23:25:06.813371 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:06.813339 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cbf1ba2-93d9-4a31-927f-e60ed20baebb" path="/var/lib/kubelet/pods/2cbf1ba2-93d9-4a31-927f-e60ed20baebb/volumes" Nov 03 23:25:18.030243 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:18.030207 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:25:23.278150 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:23.278114 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:25:25.917338 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:25:25.917301 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:25:30.107255 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:30.107217 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:25:36.848910 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:36.848876 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6ddf589cff-wl4x4"] Nov 03 23:25:36.849297 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:36.849201 5196 cpu_manager.go:414] "containerMap: removing container" podUID="2cbf1ba2-93d9-4a31-927f-e60ed20baebb" containerName="main" Nov 03 23:25:36.849297 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:36.849211 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cbf1ba2-93d9-4a31-927f-e60ed20baebb" containerName="main" Nov 03 23:25:36.849297 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:36.849280 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cbf1ba2-93d9-4a31-927f-e60ed20baebb" containerName="main" Nov 03 23:25:36.852373 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:36.852353 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6ddf589cff-wl4x4" Nov 03 23:25:36.907429 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:36.907399 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:25:36.936818 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:36.936793 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb1b2710-6672-4551-8a8d-088509c0317c-metrics-certs\") pod \"maas-api-6ddf589cff-wl4x4\" (UID: \"eb1b2710-6672-4551-8a8d-088509c0317c\") " pod="odh-ai-gateway-infra/maas-api-6ddf589cff-wl4x4" Nov 03 23:25:36.936969 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:36.936825 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1b2710-6672-4551-8a8d-088509c0317c-openshift-service-ca\") pod \"maas-api-6ddf589cff-wl4x4\" (UID: \"eb1b2710-6672-4551-8a8d-088509c0317c\") " pod="odh-ai-gateway-infra/maas-api-6ddf589cff-wl4x4" Nov 03 23:25:36.937044 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:36.937015 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb1b2710-6672-4551-8a8d-088509c0317c-maas-api-tls\") pod \"maas-api-6ddf589cff-wl4x4\" (UID: \"eb1b2710-6672-4551-8a8d-088509c0317c\") " pod="odh-ai-gateway-infra/maas-api-6ddf589cff-wl4x4" Nov 03 23:25:36.937116 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:36.937098 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgrc2\" (UniqueName: \"kubernetes.io/projected/eb1b2710-6672-4551-8a8d-088509c0317c-kube-api-access-jgrc2\") pod \"maas-api-6ddf589cff-wl4x4\" (UID: \"eb1b2710-6672-4551-8a8d-088509c0317c\") " pod="odh-ai-gateway-infra/maas-api-6ddf589cff-wl4x4" Nov 03 23:25:36.958302 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:36.958278 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:25:36.958434 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:36.958335 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 23:25:36.958434 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:36.958400 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 23:25:36.958602 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:36.958585 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sw68g\"" Nov 03 23:25:36.960110 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:36.960092 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6ddf589cff-wl4x4"] Nov 03 23:25:37.037940 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:37.037914 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgrc2\" (UniqueName: \"kubernetes.io/projected/eb1b2710-6672-4551-8a8d-088509c0317c-kube-api-access-jgrc2\") pod \"maas-api-6ddf589cff-wl4x4\" (UID: \"eb1b2710-6672-4551-8a8d-088509c0317c\") " pod="odh-ai-gateway-infra/maas-api-6ddf589cff-wl4x4" Nov 03 23:25:37.038062 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:37.037951 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb1b2710-6672-4551-8a8d-088509c0317c-metrics-certs\") pod \"maas-api-6ddf589cff-wl4x4\" (UID: \"eb1b2710-6672-4551-8a8d-088509c0317c\") " pod="odh-ai-gateway-infra/maas-api-6ddf589cff-wl4x4" Nov 03 23:25:37.038062 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:37.037975 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1b2710-6672-4551-8a8d-088509c0317c-openshift-service-ca\") pod \"maas-api-6ddf589cff-wl4x4\" (UID: \"eb1b2710-6672-4551-8a8d-088509c0317c\") " pod="odh-ai-gateway-infra/maas-api-6ddf589cff-wl4x4" Nov 03 23:25:37.038149 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:37.038092 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb1b2710-6672-4551-8a8d-088509c0317c-maas-api-tls\") pod \"maas-api-6ddf589cff-wl4x4\" (UID: \"eb1b2710-6672-4551-8a8d-088509c0317c\") " pod="odh-ai-gateway-infra/maas-api-6ddf589cff-wl4x4" Nov 03 23:25:37.038642 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:37.038618 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1b2710-6672-4551-8a8d-088509c0317c-openshift-service-ca\") pod \"maas-api-6ddf589cff-wl4x4\" (UID: \"eb1b2710-6672-4551-8a8d-088509c0317c\") " pod="odh-ai-gateway-infra/maas-api-6ddf589cff-wl4x4" Nov 03 23:25:37.040438 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:37.040420 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb1b2710-6672-4551-8a8d-088509c0317c-metrics-certs\") pod \"maas-api-6ddf589cff-wl4x4\" (UID: \"eb1b2710-6672-4551-8a8d-088509c0317c\") " pod="odh-ai-gateway-infra/maas-api-6ddf589cff-wl4x4" Nov 03 23:25:37.040499 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:37.040458 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb1b2710-6672-4551-8a8d-088509c0317c-maas-api-tls\") pod \"maas-api-6ddf589cff-wl4x4\" (UID: \"eb1b2710-6672-4551-8a8d-088509c0317c\") " pod="odh-ai-gateway-infra/maas-api-6ddf589cff-wl4x4" Nov 03 23:25:37.099825 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:37.099759 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgrc2\" (UniqueName: \"kubernetes.io/projected/eb1b2710-6672-4551-8a8d-088509c0317c-kube-api-access-jgrc2\") pod \"maas-api-6ddf589cff-wl4x4\" (UID: \"eb1b2710-6672-4551-8a8d-088509c0317c\") " pod="odh-ai-gateway-infra/maas-api-6ddf589cff-wl4x4" Nov 03 23:25:37.164683 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:37.164655 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6ddf589cff-wl4x4" Nov 03 23:25:37.539203 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:37.539175 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6ddf589cff-wl4x4"] Nov 03 23:25:37.541539 ip-10-0-143-132 kubenswrapper[5196]: W1103 23:25:37.541515 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb1b2710_6672_4551_8a8d_088509c0317c.slice/crio-1cb0838fe942dbe9ac5512034c06730c4c24beca04bdd5ebe78ba94e586e2ed2 WatchSource:0}: Error finding container 1cb0838fe942dbe9ac5512034c06730c4c24beca04bdd5ebe78ba94e586e2ed2: Status 404 returned error can't find the container with id 1cb0838fe942dbe9ac5512034c06730c4c24beca04bdd5ebe78ba94e586e2ed2 Nov 03 23:25:37.542808 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:37.542790 5196 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 23:25:38.231602 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:38.231567 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6ddf589cff-wl4x4" event={"ID":"eb1b2710-6672-4551-8a8d-088509c0317c","Type":"ContainerStarted","Data":"1cb0838fe942dbe9ac5512034c06730c4c24beca04bdd5ebe78ba94e586e2ed2"} Nov 03 23:25:39.241177 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:39.241141 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6ddf589cff-wl4x4" event={"ID":"eb1b2710-6672-4551-8a8d-088509c0317c","Type":"ContainerStarted","Data":"433b3bd0b9aafda16629cfecd421a0059163b7ba6d033ae405d827098ff71f29"} Nov 03 23:25:39.241615 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:39.241214 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6ddf589cff-wl4x4" Nov 03 23:25:39.360255 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:39.360202 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6ddf589cff-wl4x4" podStartSLOduration=2.466327511 podStartE2EDuration="3.360185859s" totalImagesPullingTime="893.858348ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:25:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:25:39.359352822 +0000 UTC m=+3768.668912114" watchObservedRunningTime="2026-11-03 23:25:39.360185859 +0000 UTC m=+3768.669745152" Nov 03 23:25:43.218534 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:43.218498 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:25:43.826217 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:43.826183 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:25:44.173227 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:44.173185 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:25:45.190206 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:45.190178 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6ddf589cff-wl4x4"] Nov 03 23:25:45.190666 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:45.190394 5196 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6ddf589cff-wl4x4" podUID="eb1b2710-6672-4551-8a8d-088509c0317c" containerName="maas-api" containerID="cri-o://433b3bd0b9aafda16629cfecd421a0059163b7ba6d033ae405d827098ff71f29" gracePeriod=30 Nov 03 23:25:45.197976 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:45.197953 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6ddf589cff-wl4x4" Nov 03 23:25:45.240680 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:45.240655 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6c76b5d8d7-dk44v"] Nov 03 23:25:45.244088 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:45.244069 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c76b5d8d7-dk44v" Nov 03 23:25:45.289589 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:45.289567 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c76b5d8d7-dk44v"] Nov 03 23:25:45.313446 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:45.313417 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c8eb84e-c803-4538-ac9e-4f2d146e3f85-maas-api-tls\") pod \"maas-api-6c76b5d8d7-dk44v\" (UID: \"7c8eb84e-c803-4538-ac9e-4f2d146e3f85\") " pod="odh-ai-gateway-infra/maas-api-6c76b5d8d7-dk44v" Nov 03 23:25:45.313585 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:45.313479 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c8eb84e-c803-4538-ac9e-4f2d146e3f85-metrics-certs\") pod \"maas-api-6c76b5d8d7-dk44v\" (UID: \"7c8eb84e-c803-4538-ac9e-4f2d146e3f85\") " pod="odh-ai-gateway-infra/maas-api-6c76b5d8d7-dk44v" Nov 03 23:25:45.313585 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:45.313518 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjpsl\" (UniqueName: \"kubernetes.io/projected/7c8eb84e-c803-4538-ac9e-4f2d146e3f85-kube-api-access-rjpsl\") pod \"maas-api-6c76b5d8d7-dk44v\" (UID: \"7c8eb84e-c803-4538-ac9e-4f2d146e3f85\") " pod="odh-ai-gateway-infra/maas-api-6c76b5d8d7-dk44v" Nov 03 23:25:45.313923 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:45.313582 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c8eb84e-c803-4538-ac9e-4f2d146e3f85-openshift-service-ca\") pod \"maas-api-6c76b5d8d7-dk44v\" (UID: \"7c8eb84e-c803-4538-ac9e-4f2d146e3f85\") " pod="odh-ai-gateway-infra/maas-api-6c76b5d8d7-dk44v" Nov 03 23:25:45.414480 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:45.414441 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c8eb84e-c803-4538-ac9e-4f2d146e3f85-metrics-certs\") pod \"maas-api-6c76b5d8d7-dk44v\" (UID: \"7c8eb84e-c803-4538-ac9e-4f2d146e3f85\") " pod="odh-ai-gateway-infra/maas-api-6c76b5d8d7-dk44v" Nov 03 23:25:45.414480 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:45.414477 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjpsl\" (UniqueName: \"kubernetes.io/projected/7c8eb84e-c803-4538-ac9e-4f2d146e3f85-kube-api-access-rjpsl\") pod \"maas-api-6c76b5d8d7-dk44v\" (UID: \"7c8eb84e-c803-4538-ac9e-4f2d146e3f85\") " pod="odh-ai-gateway-infra/maas-api-6c76b5d8d7-dk44v" Nov 03 23:25:45.414724 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:45.414514 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c8eb84e-c803-4538-ac9e-4f2d146e3f85-openshift-service-ca\") pod \"maas-api-6c76b5d8d7-dk44v\" (UID: \"7c8eb84e-c803-4538-ac9e-4f2d146e3f85\") " pod="odh-ai-gateway-infra/maas-api-6c76b5d8d7-dk44v" Nov 03 23:25:45.414724 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:45.414606 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c8eb84e-c803-4538-ac9e-4f2d146e3f85-maas-api-tls\") pod \"maas-api-6c76b5d8d7-dk44v\" (UID: \"7c8eb84e-c803-4538-ac9e-4f2d146e3f85\") " pod="odh-ai-gateway-infra/maas-api-6c76b5d8d7-dk44v" Nov 03 23:25:45.415441 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:45.415407 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c8eb84e-c803-4538-ac9e-4f2d146e3f85-openshift-service-ca\") pod \"maas-api-6c76b5d8d7-dk44v\" (UID: \"7c8eb84e-c803-4538-ac9e-4f2d146e3f85\") " pod="odh-ai-gateway-infra/maas-api-6c76b5d8d7-dk44v" Nov 03 23:25:45.417097 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:45.417069 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c8eb84e-c803-4538-ac9e-4f2d146e3f85-maas-api-tls\") pod \"maas-api-6c76b5d8d7-dk44v\" (UID: \"7c8eb84e-c803-4538-ac9e-4f2d146e3f85\") " pod="odh-ai-gateway-infra/maas-api-6c76b5d8d7-dk44v" Nov 03 23:25:45.417215 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:45.417168 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c8eb84e-c803-4538-ac9e-4f2d146e3f85-metrics-certs\") pod \"maas-api-6c76b5d8d7-dk44v\" (UID: \"7c8eb84e-c803-4538-ac9e-4f2d146e3f85\") " pod="odh-ai-gateway-infra/maas-api-6c76b5d8d7-dk44v" Nov 03 23:25:45.456445 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:45.456422 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6ddf589cff-wl4x4" Nov 03 23:25:45.476032 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:45.476012 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjpsl\" (UniqueName: \"kubernetes.io/projected/7c8eb84e-c803-4538-ac9e-4f2d146e3f85-kube-api-access-rjpsl\") pod \"maas-api-6c76b5d8d7-dk44v\" (UID: \"7c8eb84e-c803-4538-ac9e-4f2d146e3f85\") " pod="odh-ai-gateway-infra/maas-api-6c76b5d8d7-dk44v" Nov 03 23:25:45.515739 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:45.515714 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb1b2710-6672-4551-8a8d-088509c0317c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb1b2710-6672-4551-8a8d-088509c0317c-maas-api-tls\") pod \"eb1b2710-6672-4551-8a8d-088509c0317c\" (UID: \"eb1b2710-6672-4551-8a8d-088509c0317c\") " Nov 03 23:25:45.515889 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:45.515776 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb1b2710-6672-4551-8a8d-088509c0317c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1b2710-6672-4551-8a8d-088509c0317c-openshift-service-ca\") pod \"eb1b2710-6672-4551-8a8d-088509c0317c\" (UID: \"eb1b2710-6672-4551-8a8d-088509c0317c\") " Nov 03 23:25:45.515889 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:45.515797 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb1b2710-6672-4551-8a8d-088509c0317c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb1b2710-6672-4551-8a8d-088509c0317c-metrics-certs\") pod \"eb1b2710-6672-4551-8a8d-088509c0317c\" (UID: \"eb1b2710-6672-4551-8a8d-088509c0317c\") " Nov 03 23:25:45.515889 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:45.515840 5196 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb1b2710-6672-4551-8a8d-088509c0317c-kube-api-access-jgrc2\" (UniqueName: \"kubernetes.io/projected/eb1b2710-6672-4551-8a8d-088509c0317c-kube-api-access-jgrc2\") pod \"eb1b2710-6672-4551-8a8d-088509c0317c\" (UID: \"eb1b2710-6672-4551-8a8d-088509c0317c\") " Nov 03 23:25:45.516187 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:45.516154 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb1b2710-6672-4551-8a8d-088509c0317c-openshift-service-ca" pod "eb1b2710-6672-4551-8a8d-088509c0317c" (UID: "eb1b2710-6672-4551-8a8d-088509c0317c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:25:45.517829 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:45.517809 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb1b2710-6672-4551-8a8d-088509c0317c-maas-api-tls" pod "eb1b2710-6672-4551-8a8d-088509c0317c" (UID: "eb1b2710-6672-4551-8a8d-088509c0317c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:25:45.517993 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:45.517975 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb1b2710-6672-4551-8a8d-088509c0317c-kube-api-access-jgrc2" pod "eb1b2710-6672-4551-8a8d-088509c0317c" (UID: "eb1b2710-6672-4551-8a8d-088509c0317c"). InnerVolumeSpecName "kube-api-access-jgrc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:25:45.518143 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:45.518114 5196 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb1b2710-6672-4551-8a8d-088509c0317c-metrics-certs" pod "eb1b2710-6672-4551-8a8d-088509c0317c" (UID: "eb1b2710-6672-4551-8a8d-088509c0317c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:25:45.587139 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:45.587108 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c76b5d8d7-dk44v" Nov 03 23:25:45.617412 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:45.617384 5196 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb1b2710-6672-4551-8a8d-088509c0317c-metrics-certs\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:25:45.617412 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:45.617414 5196 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgrc2\" (UniqueName: \"kubernetes.io/projected/eb1b2710-6672-4551-8a8d-088509c0317c-kube-api-access-jgrc2\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:25:45.617566 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:45.617425 5196 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb1b2710-6672-4551-8a8d-088509c0317c-maas-api-tls\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:25:45.617566 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:45.617434 5196 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1b2710-6672-4551-8a8d-088509c0317c-openshift-service-ca\") on node \"ip-10-0-143-132.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 23:25:45.961679 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:45.961653 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c76b5d8d7-dk44v"] Nov 03 23:25:45.963800 ip-10-0-143-132 kubenswrapper[5196]: W1103 23:25:45.963773 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c8eb84e_c803_4538_ac9e_4f2d146e3f85.slice/crio-921b0e5a544b5af872184fd4444b6b6e1af3b9814589ab0bb66b669142fc7809 WatchSource:0}: Error finding container 921b0e5a544b5af872184fd4444b6b6e1af3b9814589ab0bb66b669142fc7809: Status 404 returned error can't find the container with id 921b0e5a544b5af872184fd4444b6b6e1af3b9814589ab0bb66b669142fc7809 Nov 03 23:25:46.275992 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:46.275957 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c76b5d8d7-dk44v" event={"ID":"7c8eb84e-c803-4538-ac9e-4f2d146e3f85","Type":"ContainerStarted","Data":"921b0e5a544b5af872184fd4444b6b6e1af3b9814589ab0bb66b669142fc7809"} Nov 03 23:25:46.277298 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:46.277272 5196 generic.go:356] "Generic (PLEG): container finished" podID="eb1b2710-6672-4551-8a8d-088509c0317c" containerID="433b3bd0b9aafda16629cfecd421a0059163b7ba6d033ae405d827098ff71f29" exitCode=0 Nov 03 23:25:46.277424 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:46.277313 5196 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6ddf589cff-wl4x4" Nov 03 23:25:46.277424 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:46.277337 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6ddf589cff-wl4x4" event={"ID":"eb1b2710-6672-4551-8a8d-088509c0317c","Type":"ContainerDied","Data":"433b3bd0b9aafda16629cfecd421a0059163b7ba6d033ae405d827098ff71f29"} Nov 03 23:25:46.277424 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:46.277372 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6ddf589cff-wl4x4" event={"ID":"eb1b2710-6672-4551-8a8d-088509c0317c","Type":"ContainerDied","Data":"1cb0838fe942dbe9ac5512034c06730c4c24beca04bdd5ebe78ba94e586e2ed2"} Nov 03 23:25:46.277424 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:46.277392 5196 scope.go:122] "RemoveContainer" containerID="433b3bd0b9aafda16629cfecd421a0059163b7ba6d033ae405d827098ff71f29" Nov 03 23:25:46.286513 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:46.286498 5196 scope.go:122] "RemoveContainer" containerID="433b3bd0b9aafda16629cfecd421a0059163b7ba6d033ae405d827098ff71f29" Nov 03 23:25:46.286790 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:25:46.286774 5196 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"433b3bd0b9aafda16629cfecd421a0059163b7ba6d033ae405d827098ff71f29\": container with ID starting with 433b3bd0b9aafda16629cfecd421a0059163b7ba6d033ae405d827098ff71f29 not found: ID does not exist" containerID="433b3bd0b9aafda16629cfecd421a0059163b7ba6d033ae405d827098ff71f29" Nov 03 23:25:46.286844 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:46.286798 5196 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"433b3bd0b9aafda16629cfecd421a0059163b7ba6d033ae405d827098ff71f29"} err="failed to get container status \"433b3bd0b9aafda16629cfecd421a0059163b7ba6d033ae405d827098ff71f29\": rpc error: code = NotFound desc = could not find container \"433b3bd0b9aafda16629cfecd421a0059163b7ba6d033ae405d827098ff71f29\": container with ID starting with 433b3bd0b9aafda16629cfecd421a0059163b7ba6d033ae405d827098ff71f29 not found: ID does not exist" Nov 03 23:25:46.443158 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:46.443117 5196 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6ddf589cff-wl4x4"] Nov 03 23:25:46.493836 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:46.493766 5196 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6ddf589cff-wl4x4"] Nov 03 23:25:46.813721 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:46.813696 5196 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb1b2710-6672-4551-8a8d-088509c0317c" path="/var/lib/kubelet/pods/eb1b2710-6672-4551-8a8d-088509c0317c/volumes" Nov 03 23:25:47.284064 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:47.284025 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c76b5d8d7-dk44v" event={"ID":"7c8eb84e-c803-4538-ac9e-4f2d146e3f85","Type":"ContainerStarted","Data":"d381123651b08f7cd8559d843bc99e045ca749624859c912bdcb6ea9bea4e8de"} Nov 03 23:25:47.284555 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:47.284165 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6c76b5d8d7-dk44v" Nov 03 23:25:47.323519 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:47.323483 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:25:47.405441 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:47.405384 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6c76b5d8d7-dk44v" podStartSLOduration=1.625977998 podStartE2EDuration="2.405371645s" totalImagesPullingTime="779.393647ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:25:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:25:47.402989569 +0000 UTC m=+3776.712548862" watchObservedRunningTime="2026-11-03 23:25:47.405371645 +0000 UTC m=+3776.714930958" Nov 03 23:25:47.505133 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:47.505097 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:25:47.655007 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:47.654974 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:25:50.522426 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:50.522390 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:25:50.701927 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:50.701896 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:25:52.534913 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:52.534807 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:25:52.698619 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:52.698578 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:25:52.857307 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:52.857273 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:25:53.002767 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:53.002737 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:25:53.295476 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:53.295390 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6c76b5d8d7-dk44v" Nov 03 23:25:55.917702 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:25:55.917664 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:25:57.794601 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:25:57.794567 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:26:00.206881 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:26:00.206782 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:26:02.602237 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:26:02.602202 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:26:07.401604 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:26:07.401571 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:26:12.592734 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:26:12.592675 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:26:17.778969 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:26:17.778935 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:26:18.367706 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:26:18.367672 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:26:22.710028 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:26:22.709994 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:26:23.618969 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:26:23.618940 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:26:25.452515 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:26:25.452467 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:26:25.595948 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:26:25.595900 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:26:25.918042 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:26:25.918006 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:26:27.410388 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:26:27.410358 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:26:31.604473 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:26:31.604436 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:26:32.451155 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:26:32.451126 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:26:32.603838 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:26:32.603797 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:26:33.154511 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:26:33.154480 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:26:33.431570 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:26:33.431491 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:26:34.458574 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:26:34.458525 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:26:35.679598 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:26:35.679562 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:26:36.283674 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:26:36.283645 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:26:36.453353 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:26:36.453312 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:26:40.275166 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:26:40.275133 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:26:40.441428 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:26:40.441384 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:26:40.590314 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:26:40.590279 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:26:43.470357 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:26:43.470327 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:26:43.663403 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:26:43.663367 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:26:43.826344 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:26:43.826267 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:26:44.009844 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:26:44.009803 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:26:48.599286 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:26:48.599247 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:26:53.489507 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:26:53.489471 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:26:55.919505 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:26:55.919471 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:26:58.307252 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:26:58.307211 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:26:59.901002 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:26:59.900968 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:27:03.493687 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:27:03.493648 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:27:08.288673 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:27:08.288635 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:27:13.080220 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:27:13.080193 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:27:17.888729 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:27:17.888695 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:27:18.719663 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:27:18.719628 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:27:22.683865 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:27:22.683804 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:27:23.956615 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:27:23.956581 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:27:25.918312 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:27:25.918275 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:27:27.467138 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:27:27.467103 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:27:32.305934 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:27:32.305836 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:27:37.097159 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:27:37.097125 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:27:41.889714 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:27:41.889678 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:27:46.694046 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:27:46.694002 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:27:51.496174 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:27:51.496142 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:27:55.918074 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:27:55.918040 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:27:56.296241 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:27:56.296155 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:01.089596 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:28:01.089558 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:05.892922 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:28:05.892879 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:10.794237 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:28:10.794205 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:15.688692 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:28:15.688655 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:19.066490 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:28:19.066447 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:20.383191 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:28:20.383154 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:22.003747 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:28:22.003714 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:24.301592 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:28:24.301561 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:25.184992 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:28:25.184963 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:25.918293 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:28:25.918252 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:28:29.987951 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:28:29.987919 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:34.717738 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:28:34.717704 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:39.733587 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:28:39.733551 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:44.701087 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:28:44.701056 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:47.711079 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:28:47.711022 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:47.859220 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:28:47.859182 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:54.050044 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:28:54.049999 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:55.553639 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:28:55.553610 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:55.839448 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:28:55.839406 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:28:55.919262 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:28:55.919224 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:28:56.597493 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:28:56.597456 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:02.738351 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:29:02.738268 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:02.881244 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:29:02.881213 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:03.084023 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:29:03.083942 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:19.434221 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:29:19.434184 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:24.688120 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:29:24.688071 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:29:25.918353 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:29:25.918318 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:29:55.917243 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:29:55.917206 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:30:13.325425 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:30:13.325390 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b98df9fc6-mz7zf_6f268a42-a565-44bc-a2f9-318cfc884c5a/authorino/0.log" Nov 03 23:30:19.756182 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:30:19.756138 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:30:24.447735 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:30:24.447702 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-s2rsn_89fdaa67-48e4-45c9-aca2-5bf2d3125d38/manager/0.log" Nov 03 23:30:25.046283 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:30:25.046251 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:30:25.277782 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:30:25.277752 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-hhp7p_7307cd39-a4a5-45d6-aeee-434e9c640246/manager/0.log" Nov 03 23:30:25.918205 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:30:25.918171 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:30:26.764656 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:30:26.764627 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-b4xp2_f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0/proxy/0.log" Nov 03 23:30:28.080335 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:30:28.080296 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6c76b5d8d7-dk44v_7c8eb84e-c803-4538-ac9e-4f2d146e3f85/maas-api/0.log" Nov 03 23:30:30.609717 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:30:30.609619 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b98df9fc6-mz7zf_6f268a42-a565-44bc-a2f9-318cfc884c5a/authorino/0.log" Nov 03 23:30:31.381068 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:30:31.381039 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-dr2qp_c7b3698d-7883-4a59-aaf6-a022a950bdb0/manager/0.log" Nov 03 23:30:31.600979 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:30:31.600953 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-lhh88_e6a6a86a-e69b-49ec-bc45-a8875edd45e3/kuadrant-console-plugin/0.log" Nov 03 23:30:32.515980 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:30:32.515937 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-xf2bk_ba199d3c-aeb0-44d0-96f8-48286a2c3b40/manager/0.log" Nov 03 23:30:34.146381 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:30:34.146355 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-586b8ddf6d-sgzck_157b1eec-64e7-4c2f-94ad-dfa8a419173b/kube-auth-proxy/0.log" Nov 03 23:30:34.347357 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:30:34.347322 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:30:34.867219 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:30:34.867195 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-8q2rz_6ef06915-cf1c-45cb-b61b-8c80c45f1dcd/istio-proxy/0.log" Nov 03 23:30:35.133455 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:30:35.133370 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-cdxtr_07738db5-20e8-47f1-8339-82866bfa91fc/istio-proxy/0.log" Nov 03 23:30:35.835398 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:30:35.835360 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-njmdj_9f644685-a2d7-48ee-833d-d49c9959f598/istio-proxy/0.log" Nov 03 23:30:41.012220 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:30:41.012190 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b98df9fc6-mz7zf_6f268a42-a565-44bc-a2f9-318cfc884c5a/authorino/0.log" Nov 03 23:30:50.526948 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:30:50.526903 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-s2rsn_89fdaa67-48e4-45c9-aca2-5bf2d3125d38/manager/0.log" Nov 03 23:30:51.188479 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:30:51.188445 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-hhp7p_7307cd39-a4a5-45d6-aeee-434e9c640246/manager/0.log" Nov 03 23:30:52.494621 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:30:52.494584 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-b4xp2_f9d8baf9-a8dc-4101-82b6-a6ab8c7adbf0/proxy/0.log" Nov 03 23:30:53.374359 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:30:53.374329 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6c76b5d8d7-dk44v_7c8eb84e-c803-4538-ac9e-4f2d146e3f85/maas-api/0.log" Nov 03 23:30:55.483075 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:30:55.483045 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b98df9fc6-mz7zf_6f268a42-a565-44bc-a2f9-318cfc884c5a/authorino/0.log" Nov 03 23:30:55.917501 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:30:55.917449 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:30:56.274072 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:30:56.273981 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-dr2qp_c7b3698d-7883-4a59-aaf6-a022a950bdb0/manager/0.log" Nov 03 23:30:56.491523 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:30:56.491498 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-lhh88_e6a6a86a-e69b-49ec-bc45-a8875edd45e3/kuadrant-console-plugin/0.log" Nov 03 23:30:57.389822 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:30:57.389790 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-xf2bk_ba199d3c-aeb0-44d0-96f8-48286a2c3b40/manager/0.log" Nov 03 23:30:58.780851 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:30:58.780822 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-586b8ddf6d-sgzck_157b1eec-64e7-4c2f-94ad-dfa8a419173b/kube-auth-proxy/0.log" Nov 03 23:30:59.510835 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:30:59.510796 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-8q2rz_6ef06915-cf1c-45cb-b61b-8c80c45f1dcd/istio-proxy/0.log" Nov 03 23:30:59.828084 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:30:59.828011 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-cdxtr_07738db5-20e8-47f1-8339-82866bfa91fc/istio-proxy/0.log" Nov 03 23:31:00.546446 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:31:00.546417 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-njmdj_9f644685-a2d7-48ee-833d-d49c9959f598/istio-proxy/0.log" Nov 03 23:31:20.108246 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:31:20.108179 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:31:25.387126 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:31:25.387090 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:31:25.917985 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:31:25.917942 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:31:55.917969 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:31:55.917928 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:32:13.407345 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:13.407259 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:32:14.364828 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:14.364787 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:32:14.521678 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:14.521639 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:32:20.451261 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:20.451218 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:32:23.781622 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:23.781593 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-h8jxb_b6154c5a-4925-4c74-8db4-0acdb525b743/global-pull-secret-syncer/0.log" Nov 03 23:32:25.790504 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:25.790469 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:32:25.917566 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:32:25.917531 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:32:26.311387 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:26.311354 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" Nov 03 23:32:28.765058 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:28.765018 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-132.us-west-1.compute.internal_bd09a9d6123011e89fe0e21d9dfdc2d8/haproxy/0.log" Nov 03 23:32:33.680321 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:33.680289 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b98df9fc6-mz7zf_6f268a42-a565-44bc-a2f9-318cfc884c5a/authorino/0.log" Nov 03 23:32:33.821719 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:33.821691 5196 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-x9d6v/perf-node-gather-daemonset-fvft6"] Nov 03 23:32:33.822036 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:33.822023 5196 cpu_manager.go:414] "containerMap: removing container" podUID="eb1b2710-6672-4551-8a8d-088509c0317c" containerName="maas-api" Nov 03 23:32:33.822085 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:33.822037 5196 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb1b2710-6672-4551-8a8d-088509c0317c" containerName="maas-api" Nov 03 23:32:33.822121 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:33.822112 5196 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb1b2710-6672-4551-8a8d-088509c0317c" containerName="maas-api" Nov 03 23:32:33.825154 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:33.825134 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-fvft6" Nov 03 23:32:33.879681 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:33.879659 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x9d6v\"/\"openshift-service-ca.crt\"" Nov 03 23:32:33.883540 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:33.883516 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x9d6v/perf-node-gather-daemonset-fvft6"] Nov 03 23:32:33.930674 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:33.930622 5196 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-x9d6v\"/\"default-dockercfg-z6rsm\"" Nov 03 23:32:33.938618 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:33.938600 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6030b617-0c2b-4de5-84ee-02d380a91c4c-podres\") pod \"perf-node-gather-daemonset-fvft6\" (UID: \"6030b617-0c2b-4de5-84ee-02d380a91c4c\") " pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-fvft6" Nov 03 23:32:33.938701 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:33.938646 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6030b617-0c2b-4de5-84ee-02d380a91c4c-sys\") pod \"perf-node-gather-daemonset-fvft6\" (UID: \"6030b617-0c2b-4de5-84ee-02d380a91c4c\") " pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-fvft6" Nov 03 23:32:33.938944 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:33.938926 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlqjl\" (UniqueName: \"kubernetes.io/projected/6030b617-0c2b-4de5-84ee-02d380a91c4c-kube-api-access-xlqjl\") pod \"perf-node-gather-daemonset-fvft6\" (UID: \"6030b617-0c2b-4de5-84ee-02d380a91c4c\") " pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-fvft6" Nov 03 23:32:33.939119 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:33.939101 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6030b617-0c2b-4de5-84ee-02d380a91c4c-lib-modules\") pod \"perf-node-gather-daemonset-fvft6\" (UID: \"6030b617-0c2b-4de5-84ee-02d380a91c4c\") " pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-fvft6" Nov 03 23:32:33.939190 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:33.939142 5196 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6030b617-0c2b-4de5-84ee-02d380a91c4c-proc\") pod \"perf-node-gather-daemonset-fvft6\" (UID: \"6030b617-0c2b-4de5-84ee-02d380a91c4c\") " pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-fvft6" Nov 03 23:32:33.982327 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:33.982310 5196 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x9d6v\"/\"kube-root-ca.crt\"" Nov 03 23:32:34.039763 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:34.039738 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6030b617-0c2b-4de5-84ee-02d380a91c4c-lib-modules\") pod \"perf-node-gather-daemonset-fvft6\" (UID: \"6030b617-0c2b-4de5-84ee-02d380a91c4c\") " pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-fvft6" Nov 03 23:32:34.039852 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:34.039770 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6030b617-0c2b-4de5-84ee-02d380a91c4c-proc\") pod \"perf-node-gather-daemonset-fvft6\" (UID: \"6030b617-0c2b-4de5-84ee-02d380a91c4c\") " pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-fvft6" Nov 03 23:32:34.039852 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:34.039802 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6030b617-0c2b-4de5-84ee-02d380a91c4c-podres\") pod \"perf-node-gather-daemonset-fvft6\" (UID: \"6030b617-0c2b-4de5-84ee-02d380a91c4c\") " pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-fvft6" Nov 03 23:32:34.039852 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:34.039832 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6030b617-0c2b-4de5-84ee-02d380a91c4c-sys\") pod \"perf-node-gather-daemonset-fvft6\" (UID: \"6030b617-0c2b-4de5-84ee-02d380a91c4c\") " pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-fvft6" Nov 03 23:32:34.040031 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:34.039888 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6030b617-0c2b-4de5-84ee-02d380a91c4c-lib-modules\") pod \"perf-node-gather-daemonset-fvft6\" (UID: \"6030b617-0c2b-4de5-84ee-02d380a91c4c\") " pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-fvft6" Nov 03 23:32:34.040031 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:34.039888 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6030b617-0c2b-4de5-84ee-02d380a91c4c-proc\") pod \"perf-node-gather-daemonset-fvft6\" (UID: \"6030b617-0c2b-4de5-84ee-02d380a91c4c\") " pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-fvft6" Nov 03 23:32:34.040031 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:34.039891 5196 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlqjl\" (UniqueName: \"kubernetes.io/projected/6030b617-0c2b-4de5-84ee-02d380a91c4c-kube-api-access-xlqjl\") pod \"perf-node-gather-daemonset-fvft6\" (UID: \"6030b617-0c2b-4de5-84ee-02d380a91c4c\") " pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-fvft6" Nov 03 23:32:34.040031 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:34.039940 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6030b617-0c2b-4de5-84ee-02d380a91c4c-podres\") pod \"perf-node-gather-daemonset-fvft6\" (UID: \"6030b617-0c2b-4de5-84ee-02d380a91c4c\") " pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-fvft6" Nov 03 23:32:34.040031 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:34.039942 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6030b617-0c2b-4de5-84ee-02d380a91c4c-sys\") pod \"perf-node-gather-daemonset-fvft6\" (UID: \"6030b617-0c2b-4de5-84ee-02d380a91c4c\") " pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-fvft6" Nov 03 23:32:34.101556 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:34.101529 5196 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlqjl\" (UniqueName: \"kubernetes.io/projected/6030b617-0c2b-4de5-84ee-02d380a91c4c-kube-api-access-xlqjl\") pod \"perf-node-gather-daemonset-fvft6\" (UID: \"6030b617-0c2b-4de5-84ee-02d380a91c4c\") " pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-fvft6" Nov 03 23:32:34.137030 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:34.137013 5196 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-fvft6" Nov 03 23:32:34.472536 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:34.472454 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-dr2qp_c7b3698d-7883-4a59-aaf6-a022a950bdb0/manager/0.log" Nov 03 23:32:34.512516 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:34.512492 5196 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x9d6v/perf-node-gather-daemonset-fvft6"] Nov 03 23:32:34.514944 ip-10-0-143-132 kubenswrapper[5196]: W1103 23:32:34.514919 5196 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6030b617_0c2b_4de5_84ee_02d380a91c4c.slice/crio-b70a5e19abcee3886e041780abc2a49e1628251024b6424f26054cf458b90cf5 WatchSource:0}: Error finding container b70a5e19abcee3886e041780abc2a49e1628251024b6424f26054cf458b90cf5: Status 404 returned error can't find the container with id b70a5e19abcee3886e041780abc2a49e1628251024b6424f26054cf458b90cf5 Nov 03 23:32:34.887121 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:34.887091 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-lhh88_e6a6a86a-e69b-49ec-bc45-a8875edd45e3/kuadrant-console-plugin/0.log" Nov 03 23:32:35.287338 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:35.287249 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-fvft6" event={"ID":"6030b617-0c2b-4de5-84ee-02d380a91c4c","Type":"ContainerStarted","Data":"5b716b8ab3a7a044e7c1e50e3e944fc3626c1dc985f8e62e672ca6d54c7e1b7f"} Nov 03 23:32:35.287338 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:35.287312 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-fvft6" Nov 03 23:32:35.287338 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:35.287335 5196 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-fvft6" event={"ID":"6030b617-0c2b-4de5-84ee-02d380a91c4c","Type":"ContainerStarted","Data":"b70a5e19abcee3886e041780abc2a49e1628251024b6424f26054cf458b90cf5"} Nov 03 23:32:35.405504 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:35.405448 5196 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-fvft6" podStartSLOduration=2.4054329230000002 podStartE2EDuration="2.405432923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:32:35.403542259 +0000 UTC m=+4184.713101552" watchObservedRunningTime="2026-11-03 23:32:35.405432923 +0000 UTC m=+4184.714992216" Nov 03 23:32:36.231432 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:36.231394 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-xf2bk_ba199d3c-aeb0-44d0-96f8-48286a2c3b40/manager/0.log" Nov 03 23:32:37.319350 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:37.319318 5196 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g6qjz_b0ba09d3-25f2-422b-b5e2-357aabcaf81c/konnectivity-agent/0.log" limit=1024 Nov 03 23:32:41.303305 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:41.303277 5196 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-x9d6v/perf-node-gather-daemonset-fvft6" Nov 03 23:32:43.830487 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:43.830463 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-hmr98_5b40d10d-8dd1-46f6-b897-82780c392c84/monitoring-plugin/0.log" Nov 03 23:32:46.752932 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:46.752899 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zt4dd_0e68894d-ed4f-4bc5-b36e-691e557883bd/node-exporter/0.log" Nov 03 23:32:47.166026 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:47.165995 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zt4dd_0e68894d-ed4f-4bc5-b36e-691e557883bd/kube-rbac-proxy/0.log" Nov 03 23:32:47.579247 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:47.579162 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zt4dd_0e68894d-ed4f-4bc5-b36e-691e557883bd/init-textfile/0.log" Nov 03 23:32:49.256244 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:49.256211 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a254ea4-03c8-4dbb-904c-a1def38b0d83/prometheus/0.log" Nov 03 23:32:49.668955 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:49.668923 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a254ea4-03c8-4dbb-904c-a1def38b0d83/config-reloader/0.log" Nov 03 23:32:50.083969 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:50.083884 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a254ea4-03c8-4dbb-904c-a1def38b0d83/thanos-sidecar/0.log" Nov 03 23:32:50.498378 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:50.498348 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a254ea4-03c8-4dbb-904c-a1def38b0d83/kube-rbac-proxy-web/0.log" Nov 03 23:32:50.922757 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:50.922731 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a254ea4-03c8-4dbb-904c-a1def38b0d83/kube-rbac-proxy/0.log" Nov 03 23:32:51.336963 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:51.336882 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a254ea4-03c8-4dbb-904c-a1def38b0d83/kube-rbac-proxy-thanos/0.log" Nov 03 23:32:51.754733 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:51.754682 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a254ea4-03c8-4dbb-904c-a1def38b0d83/init-config-reloader/0.log" Nov 03 23:32:52.176282 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:52.176253 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-qb4qm_e6690a9b-44fa-4b1f-9523-173d8481dbea/prometheus-operator/0.log" Nov 03 23:32:52.585920 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:52.585830 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-qb4qm_e6690a9b-44fa-4b1f-9523-173d8481dbea/kube-rbac-proxy/0.log" Nov 03 23:32:55.918171 ip-10-0-143-132 kubenswrapper[5196]: E1103 23:32:55.918129 5196 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jwm6k" containerName="extractor" Nov 03 23:32:57.530755 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:57.530730 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-586b8ddf6d-sgzck_157b1eec-64e7-4c2f-94ad-dfa8a419173b/kube-auth-proxy/0.log" Nov 03 23:32:58.170472 ip-10-0-143-132 kubenswrapper[5196]: I1103 23:32:58.170442 5196 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-8q2rz_6ef06915-cf1c-45cb-b61b-8c80c45f1dcd/istio-proxy/0.log"