Oct 30 21:47:08.507005 ip-10-0-137-81 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 21:47:08.507016 ip-10-0-137-81 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 21:47:08.507023 ip-10-0-137-81 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 21:47:08.507244 ip-10-0-137-81 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 21:47:18.756400 ip-10-0-137-81 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 21:47:18.756423 ip-10-0-137-81 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 43e37fc777f84761ae62e22182a891f6 -- Oct 30 21:49:26.395093 ip-10-0-137-81 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:49:26.781967 ip-10-0-137-81 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:49:26.781967 ip-10-0-137-81 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:49:26.781967 ip-10-0-137-81 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:49:26.781967 ip-10-0-137-81 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:49:26.785905 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.785778 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:49:26.785905 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.785902 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:49:26.785905 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.785906 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:49:26.785905 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.785908 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:49:26.785905 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.785911 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:49:26.786095 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.785914 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:49:26.786095 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.785917 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:49:26.786095 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.785920 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:49:26.786095 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.785923 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:49:26.786095 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.785926 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:49:26.786095 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.785930 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:49:26.786095 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.785932 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:49:26.786095 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.785935 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:49:26.786095 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.785938 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:49:26.786095 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.785940 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:49:26.786095 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.785943 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:49:26.786095 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.785946 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:49:26.786095 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.785948 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:49:26.786095 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.785951 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:49:26.786095 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.785954 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:49:26.786095 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.785956 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:49:26.786095 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.785959 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:49:26.786095 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.785962 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:49:26.786095 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.785964 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:49:26.786095 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.785966 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:49:26.786607 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.785969 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:49:26.786607 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.785971 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:49:26.786607 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.785974 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:49:26.786607 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.785976 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:49:26.786607 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.785978 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:49:26.786607 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.785981 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:49:26.786607 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.785984 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:49:26.786607 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.785990 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:49:26.786607 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.785992 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:49:26.786607 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.785995 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:49:26.786607 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.785997 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:49:26.786607 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786000 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:49:26.786607 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786003 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:49:26.786607 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786005 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:49:26.786607 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786008 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:49:26.786607 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786011 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:49:26.786607 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786014 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:49:26.786607 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786016 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:49:26.786607 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786019 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:49:26.787072 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786022 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:49:26.787072 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786024 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:49:26.787072 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786027 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:49:26.787072 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786029 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:49:26.787072 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786032 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:49:26.787072 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786034 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:49:26.787072 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786037 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:49:26.787072 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786040 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:49:26.787072 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786042 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:49:26.787072 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786045 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:49:26.787072 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786048 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:49:26.787072 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786050 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:49:26.787072 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786053 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:49:26.787072 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786056 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:49:26.787072 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786058 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:49:26.787072 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786061 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:49:26.787072 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786063 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:49:26.787072 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786066 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:49:26.787072 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786069 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:49:26.787072 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786072 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:49:26.787616 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786075 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:49:26.787616 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786077 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:49:26.787616 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786081 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:49:26.787616 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786086 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:49:26.787616 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786088 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:49:26.787616 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786091 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:49:26.787616 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786094 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:49:26.787616 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786097 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:49:26.787616 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786100 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:49:26.787616 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786105 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:49:26.787616 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786110 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:49:26.787616 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786113 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:49:26.787616 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786115 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:49:26.787616 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786118 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:49:26.787616 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786121 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:49:26.787616 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786123 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:49:26.787616 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786126 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:49:26.787616 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786128 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:49:26.787616 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786131 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:49:26.788080 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786135 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:49:26.788080 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786138 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:49:26.788080 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786140 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:49:26.788080 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786143 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:49:26.788080 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786146 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:49:26.788080 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786148 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:49:26.788080 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786151 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:49:26.788080 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786153 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:49:26.788080 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786156 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:49:26.788080 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786158 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:49:26.788080 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786162 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:49:26.788080 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786164 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:49:26.788080 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786167 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:49:26.788080 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786170 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:49:26.788080 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786172 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:49:26.788080 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786175 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:49:26.788080 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786177 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:49:26.788080 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786180 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:49:26.788080 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786182 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:49:26.788080 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786185 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:49:26.788551 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786188 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:49:26.788551 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786191 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:49:26.788551 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.786193 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:49:26.788551 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787000 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:49:26.788551 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787007 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:49:26.788551 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787011 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:49:26.788551 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787014 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:49:26.788551 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787016 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:49:26.788551 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787019 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:49:26.788551 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787023 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:49:26.788551 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787025 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:49:26.788551 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787028 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:49:26.788551 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787031 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:49:26.788551 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787034 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:49:26.788551 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787037 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:49:26.788551 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787039 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:49:26.788551 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787042 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:49:26.788551 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787045 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:49:26.788551 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787047 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:49:26.788551 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787050 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:49:26.789050 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787052 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:49:26.789050 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787055 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:49:26.789050 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787058 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:49:26.789050 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787061 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:49:26.789050 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787063 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:49:26.789050 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787066 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:49:26.789050 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787068 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:49:26.789050 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787070 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:49:26.789050 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787073 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:49:26.789050 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787076 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:49:26.789050 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787078 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:49:26.789050 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787081 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:49:26.789050 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787084 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:49:26.789050 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787087 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:49:26.789050 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787090 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:49:26.789050 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787092 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:49:26.789050 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787095 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:49:26.789050 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787097 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:49:26.789050 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787100 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:49:26.789498 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787102 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:49:26.789498 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787105 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:49:26.789498 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787107 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:49:26.789498 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787110 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:49:26.789498 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787112 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:49:26.789498 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787115 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:49:26.789498 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787117 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:49:26.789498 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787120 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:49:26.789498 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787123 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:49:26.789498 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787125 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:49:26.789498 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787128 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:49:26.789498 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787130 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:49:26.789498 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787132 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:49:26.789498 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787135 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:49:26.789498 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787137 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:49:26.789498 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787140 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:49:26.789498 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787143 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:49:26.789498 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787145 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:49:26.789498 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787148 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:49:26.789498 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787150 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:49:26.789987 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787153 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:49:26.789987 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787155 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:49:26.789987 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787157 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:49:26.789987 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787160 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:49:26.789987 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787163 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:49:26.789987 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787166 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:49:26.789987 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787169 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:49:26.789987 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787172 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:49:26.789987 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787174 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:49:26.789987 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787176 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:49:26.789987 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787179 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:49:26.789987 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787181 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:49:26.789987 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787184 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:49:26.789987 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787186 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:49:26.789987 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787189 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:49:26.789987 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787191 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:49:26.789987 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787193 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:49:26.789987 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787197 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:49:26.789987 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787199 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:49:26.789987 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787202 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:49:26.790554 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787206 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:49:26.790554 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787210 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:49:26.790554 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787213 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:49:26.790554 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787215 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:49:26.790554 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787218 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:49:26.790554 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787220 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:49:26.790554 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787223 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:49:26.790554 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787226 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:49:26.790554 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787229 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:49:26.790554 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787231 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:49:26.790554 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787235 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:49:26.790554 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787239 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:49:26.790554 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787244 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:49:26.790554 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787246 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:49:26.790554 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787249 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:49:26.790554 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787251 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:49:26.790554 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787254 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:49:26.790554 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787257 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:49:26.790554 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787259 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:49:26.791018 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787261 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:49:26.791018 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787264 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:49:26.791018 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787266 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:49:26.791018 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787269 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:49:26.791018 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787271 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:49:26.791018 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787274 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:49:26.791018 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787276 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:49:26.791018 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787278 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:49:26.791018 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787281 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:49:26.791018 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787283 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:49:26.791018 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787286 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:49:26.791018 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787796 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:49:26.791018 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787801 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:49:26.791018 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787821 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:49:26.791018 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787824 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:49:26.791018 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787827 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:49:26.791018 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787829 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:49:26.791018 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787832 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:49:26.791018 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787835 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:49:26.791018 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787838 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:49:26.791478 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787840 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:49:26.791478 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787843 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:49:26.791478 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787845 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:49:26.791478 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787848 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:49:26.791478 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787851 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:49:26.791478 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787853 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:49:26.791478 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787856 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:49:26.791478 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787858 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:49:26.791478 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787861 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:49:26.791478 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787864 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:49:26.791478 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787867 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:49:26.791478 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787869 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:49:26.791478 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787872 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:49:26.791478 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787875 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:49:26.791478 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787878 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:49:26.791478 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787881 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:49:26.791478 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787883 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:49:26.791478 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787886 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:49:26.791478 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787888 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:49:26.791478 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787891 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:49:26.791969 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787893 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:49:26.791969 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787895 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:49:26.791969 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787898 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:49:26.791969 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787900 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:49:26.791969 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787903 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:49:26.791969 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787905 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:49:26.791969 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787908 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:49:26.791969 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787911 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:49:26.791969 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787914 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:49:26.791969 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787918 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:49:26.791969 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787921 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:49:26.791969 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787924 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:49:26.791969 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787926 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:49:26.791969 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787929 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:49:26.791969 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787931 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:49:26.791969 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787934 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:49:26.791969 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787936 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:49:26.791969 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787939 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:49:26.791969 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787941 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:49:26.792408 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787945 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:49:26.792408 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787947 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:49:26.792408 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787950 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:49:26.792408 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787953 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:49:26.792408 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787956 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:49:26.792408 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787958 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:49:26.792408 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787963 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:49:26.792408 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787965 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:49:26.792408 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787968 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:49:26.792408 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787970 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:49:26.792408 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787973 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:49:26.792408 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787975 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:49:26.792408 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787978 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:49:26.792408 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787980 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:49:26.792408 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787982 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:49:26.792408 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787985 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:49:26.792408 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787987 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:49:26.792408 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787990 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:49:26.792408 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787992 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:49:26.792877 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787994 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:49:26.792877 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.787997 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:49:26.792877 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.788000 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:49:26.792877 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.788002 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:49:26.792877 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.788005 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:49:26.792877 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.788007 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:49:26.792877 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.788010 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:49:26.792877 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.788012 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:49:26.792877 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.788014 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:49:26.792877 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.788017 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:49:26.792877 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.788019 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:49:26.792877 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.788022 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:49:26.792877 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.788024 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:49:26.792877 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.788026 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:49:26.792877 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.788029 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:49:26.792877 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.788033 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:49:26.792877 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.788036 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:49:26.792877 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.788038 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:49:26.792877 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.788041 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:49:26.792877 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.788045 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:49:26.793350 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.788048 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:49:26.793350 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.788050 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:49:26.793350 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.788053 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:49:26.793350 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.788055 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:49:26.793350 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.788057 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:49:26.793350 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.788062 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:49:26.793350 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.788066 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:49:26.793350 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.788069 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:49:26.793350 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.788072 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:49:26.793350 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.788075 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:49:26.793350 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.788077 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:49:26.793350 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.788080 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:49:26.793350 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.788082 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:49:26.793350 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.788085 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:49:26.793350 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.788088 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:49:26.793350 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.788091 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:49:26.793350 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.788094 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:49:26.793350 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.788096 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:49:26.793350 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.788099 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:49:26.793779 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788715 2576 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:49:26.793779 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788724 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:49:26.793779 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788731 2576 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:49:26.793779 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788736 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:49:26.793779 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788741 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:49:26.793779 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788745 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:49:26.793779 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788749 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:49:26.793779 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788753 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:49:26.793779 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788757 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:49:26.793779 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788760 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:49:26.793779 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788764 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:49:26.793779 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788767 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:49:26.793779 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788771 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:49:26.793779 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788774 2576 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:49:26.793779 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788777 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:49:26.793779 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788780 2576 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:49:26.793779 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788783 2576 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:49:26.793779 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788786 2576 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:49:26.793779 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788789 2576 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:49:26.793779 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788792 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:49:26.793779 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788795 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:49:26.793779 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788798 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:49:26.793779 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788801 2576 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:49:26.793779 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788818 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:49:26.794367 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788821 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:49:26.794367 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788824 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:49:26.794367 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788827 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:49:26.794367 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788830 2576 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:49:26.794367 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788834 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:49:26.794367 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788837 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:49:26.794367 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788840 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:49:26.794367 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788843 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:49:26.794367 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788847 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:49:26.794367 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788850 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:49:26.794367 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788853 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:49:26.794367 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788855 2576 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:49:26.794367 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788858 2576 flags.go:64] FLAG: --enable-server="true" Oct 30 21:49:26.794367 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788861 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:49:26.794367 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788865 2576 flags.go:64] FLAG: --event-burst="100" Oct 30 21:49:26.794367 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788868 2576 flags.go:64] FLAG: --event-qps="50" Oct 30 21:49:26.794367 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788871 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:49:26.794367 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788874 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:49:26.794367 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788877 2576 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:49:26.794367 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788882 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:49:26.794367 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788886 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:49:26.794367 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788890 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:49:26.794367 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788892 2576 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:49:26.794367 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788895 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:49:26.794367 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788898 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:49:26.795116 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788901 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:49:26.795116 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788904 2576 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:49:26.795116 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788907 2576 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:49:26.795116 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788909 2576 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:49:26.795116 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788912 2576 flags.go:64] FLAG: --feature-gates="" Oct 30 21:49:26.795116 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788916 2576 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:49:26.795116 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788919 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:49:26.795116 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788922 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:49:26.795116 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788925 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:49:26.795116 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788928 2576 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:49:26.795116 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788930 2576 flags.go:64] FLAG: --help="false" Oct 30 21:49:26.795116 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788933 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:49:26.795116 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788937 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:49:26.795116 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788940 2576 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:49:26.795116 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788943 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:49:26.795116 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788946 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:49:26.795116 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788949 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:49:26.795116 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788952 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:49:26.795116 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788954 2576 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:49:26.795116 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788958 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:49:26.795116 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788961 2576 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:49:26.795116 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788963 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:49:26.795116 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788967 2576 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:49:26.795749 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788970 2576 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:49:26.795749 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788973 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:49:26.795749 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788975 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:49:26.795749 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788978 2576 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:49:26.795749 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788983 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:49:26.795749 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788985 2576 flags.go:64] FLAG: --lock-file="" Oct 30 21:49:26.795749 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788988 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:49:26.795749 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788991 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:49:26.795749 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788994 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:49:26.795749 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.788999 2576 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:49:26.795749 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789002 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:49:26.795749 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789005 2576 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:49:26.795749 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789008 2576 flags.go:64] FLAG: --logging-format="text" Oct 30 21:49:26.795749 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789011 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:49:26.795749 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789014 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:49:26.795749 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789017 2576 flags.go:64] FLAG: --manifest-url="" Oct 30 21:49:26.795749 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789020 2576 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:49:26.795749 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789024 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:49:26.795749 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789027 2576 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:49:26.795749 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789031 2576 flags.go:64] FLAG: --max-pods="110" Oct 30 21:49:26.795749 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789033 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:49:26.795749 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789036 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:49:26.795749 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789040 2576 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:49:26.795749 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789043 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:49:26.795749 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789045 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:49:26.796454 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789048 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:49:26.796454 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789051 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:49:26.796454 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789058 2576 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:49:26.796454 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789061 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:49:26.796454 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789064 2576 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:49:26.796454 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789067 2576 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:49:26.796454 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789070 2576 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:49:26.796454 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789073 2576 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:49:26.796454 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789076 2576 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:49:26.796454 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789079 2576 flags.go:64] FLAG: --port="10250" Oct 30 21:49:26.796454 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789082 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:49:26.796454 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789085 2576 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-03fd6ff48ae5bf048" Oct 30 21:49:26.796454 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789088 2576 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:49:26.796454 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789091 2576 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:49:26.796454 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789094 2576 flags.go:64] FLAG: --register-node="true" Oct 30 21:49:26.796454 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789097 2576 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:49:26.796454 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789103 2576 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:49:26.796454 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789106 2576 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:49:26.796454 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789109 2576 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:49:26.796454 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789112 2576 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:49:26.796454 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789115 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:49:26.796454 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789118 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:49:26.796454 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789121 2576 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:49:26.796454 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789124 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:49:26.796454 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789126 2576 flags.go:64] FLAG: --runonce="false" Oct 30 21:49:26.796454 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789129 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:49:26.797163 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789132 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:49:26.797163 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789135 2576 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:49:26.797163 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789138 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:49:26.797163 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789141 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:49:26.797163 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789144 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:49:26.797163 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789147 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:49:26.797163 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789150 2576 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:49:26.797163 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789152 2576 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:49:26.797163 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789155 2576 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:49:26.797163 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789158 2576 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:49:26.797163 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789161 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:49:26.797163 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789164 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:49:26.797163 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789166 2576 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:49:26.797163 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789171 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:49:26.797163 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789176 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:49:26.797163 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789179 2576 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:49:26.797163 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789182 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:49:26.797163 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789185 2576 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:49:26.797163 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789188 2576 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:49:26.797163 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789191 2576 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:49:26.797163 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789194 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:49:26.797163 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789197 2576 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:49:26.797163 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789200 2576 flags.go:64] FLAG: --v="2" Oct 30 21:49:26.797163 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789204 2576 flags.go:64] FLAG: --version="false" Oct 30 21:49:26.797163 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789208 2576 flags.go:64] FLAG: --vmodule="" Oct 30 21:49:26.797848 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789212 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:49:26.797848 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789215 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:49:26.797848 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789352 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:49:26.797848 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789357 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:49:26.797848 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789361 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:49:26.797848 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789364 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:49:26.797848 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789368 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:49:26.797848 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789371 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:49:26.797848 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789373 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:49:26.797848 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789376 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:49:26.797848 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789379 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:49:26.797848 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789381 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:49:26.797848 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789384 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:49:26.797848 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789387 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:49:26.797848 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789390 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:49:26.797848 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789392 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:49:26.797848 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789396 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:49:26.797848 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789398 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:49:26.797848 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789401 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:49:26.798375 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789403 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:49:26.798375 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789407 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:49:26.798375 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789409 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:49:26.798375 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789412 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:49:26.798375 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789415 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:49:26.798375 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789417 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:49:26.798375 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789420 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:49:26.798375 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789423 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:49:26.798375 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789425 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:49:26.798375 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789428 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:49:26.798375 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789430 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:49:26.798375 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789433 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:49:26.798375 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789435 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:49:26.798375 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789437 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:49:26.798375 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789440 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:49:26.798375 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789442 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:49:26.798375 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789445 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:49:26.798375 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789448 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:49:26.798375 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789450 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:49:26.798915 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789453 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:49:26.798915 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789455 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:49:26.798915 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789458 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:49:26.798915 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789460 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:49:26.798915 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789463 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:49:26.798915 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789465 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:49:26.798915 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789468 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:49:26.798915 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789471 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:49:26.798915 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789473 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:49:26.798915 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789476 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:49:26.798915 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789478 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:49:26.798915 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789481 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:49:26.798915 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789483 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:49:26.798915 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789486 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:49:26.798915 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789490 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:49:26.798915 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789492 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:49:26.798915 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789494 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:49:26.798915 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789498 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:49:26.798915 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789500 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:49:26.799380 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789503 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:49:26.799380 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789505 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:49:26.799380 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789508 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:49:26.799380 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789511 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:49:26.799380 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789513 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:49:26.799380 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789516 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:49:26.799380 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789519 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:49:26.799380 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789521 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:49:26.799380 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789523 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:49:26.799380 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789526 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:49:26.799380 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789528 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:49:26.799380 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789531 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:49:26.799380 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789534 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:49:26.799380 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789538 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:49:26.799380 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789540 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:49:26.799380 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789543 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:49:26.799380 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789545 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:49:26.799380 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789548 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:49:26.799380 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789550 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:49:26.799380 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789553 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:49:26.800014 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789555 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:49:26.800014 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789558 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:49:26.800014 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789562 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:49:26.800014 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789564 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:49:26.800014 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789567 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:49:26.800014 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789569 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:49:26.800014 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789572 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:49:26.800014 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789576 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:49:26.800014 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789578 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:49:26.800014 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789581 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:49:26.800014 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789583 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:49:26.800014 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789585 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:49:26.800014 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789588 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:49:26.800014 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789590 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:49:26.800014 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789593 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:49:26.800014 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789595 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:49:26.800014 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789598 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:49:26.800014 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789600 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:49:26.800014 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789603 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:49:26.800014 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789605 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:49:26.800528 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789607 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:49:26.800528 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789610 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:49:26.800528 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789612 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:49:26.800528 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789614 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:49:26.800528 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789617 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:49:26.800528 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789619 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:49:26.800528 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789621 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:49:26.800528 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789624 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:49:26.800528 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789627 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:49:26.800528 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789629 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:49:26.800528 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789631 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:49:26.800528 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.789657 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:49:26.800528 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789692 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:49:26.800528 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789696 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:49:26.800528 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789701 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:49:26.800528 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789704 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:49:26.800910 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789707 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:49:26.800910 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789710 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:49:26.800910 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789712 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:49:26.800910 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789715 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:49:26.800910 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789718 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:49:26.800910 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789721 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:49:26.800910 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789724 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:49:26.800910 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789726 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:49:26.800910 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789729 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:49:26.800910 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789731 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:49:26.800910 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789734 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:49:26.800910 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789736 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:49:26.800910 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789738 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:49:26.800910 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789741 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:49:26.800910 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789743 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:49:26.800910 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789746 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:49:26.800910 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789748 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:49:26.800910 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789751 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:49:26.800910 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789753 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:49:26.800910 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789756 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:49:26.801385 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789758 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:49:26.801385 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789760 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:49:26.801385 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789764 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:49:26.801385 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789768 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:49:26.801385 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789771 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:49:26.801385 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789773 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:49:26.801385 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789776 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:49:26.801385 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789778 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:49:26.801385 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789781 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:49:26.801385 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789784 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:49:26.801385 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789787 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:49:26.801385 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789790 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:49:26.801385 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789793 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:49:26.801385 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789795 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:49:26.801385 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789798 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:49:26.801385 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789800 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:49:26.801385 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789816 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:49:26.801385 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789819 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:49:26.801385 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789822 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:49:26.801883 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789825 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:49:26.801883 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789827 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:49:26.801883 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789830 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:49:26.801883 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789833 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:49:26.801883 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789835 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:49:26.801883 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789838 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:49:26.801883 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789840 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:49:26.801883 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789843 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:49:26.801883 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789845 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:49:26.801883 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789848 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:49:26.801883 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789850 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:49:26.801883 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789853 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:49:26.801883 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789855 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:49:26.801883 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789858 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:49:26.801883 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789860 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:49:26.801883 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789862 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:49:26.801883 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789865 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:49:26.801883 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789867 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:49:26.801883 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789870 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:49:26.802336 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789872 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:49:26.802336 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789875 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:49:26.802336 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789877 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:49:26.802336 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789880 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:49:26.802336 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789883 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:49:26.802336 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789887 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:49:26.802336 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789890 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:49:26.802336 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789893 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:49:26.802336 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789895 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:49:26.802336 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789898 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:49:26.802336 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789902 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:49:26.802336 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789905 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:49:26.802336 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789908 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:49:26.802336 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789910 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:49:26.802336 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789912 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:49:26.802336 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789915 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:49:26.802336 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789918 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:49:26.802336 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789920 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:49:26.802336 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789923 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:49:26.802794 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789925 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:49:26.802794 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789927 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:49:26.802794 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789930 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:49:26.802794 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789932 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:49:26.802794 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789935 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:49:26.802794 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789937 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:49:26.802794 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789940 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:49:26.802794 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789942 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:49:26.802794 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789945 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:49:26.802794 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789947 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:49:26.802794 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789950 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:49:26.802794 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789953 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:49:26.802794 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789957 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:49:26.802794 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789960 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:49:26.802794 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789962 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:49:26.802794 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789965 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:49:26.802794 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789967 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:49:26.802794 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789971 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:49:26.802794 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789974 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:49:26.802794 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789976 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:49:26.803303 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789979 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:49:26.803303 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789982 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:49:26.803303 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789985 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:49:26.803303 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789989 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:49:26.803303 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.789991 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:49:26.803303 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.797044 2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:49:26.803303 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.797060 2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:49:26.803303 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.797073 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:49:26.803303 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.797077 2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:49:26.803303 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797146 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:49:26.803303 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797151 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:49:26.803303 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797154 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:49:26.803303 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797157 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:49:26.803303 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797159 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:49:26.803303 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797162 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:49:26.803303 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797165 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:49:26.803303 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797168 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:49:26.803303 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797170 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:49:26.803303 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797173 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:49:26.803738 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797176 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:49:26.803738 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797179 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:49:26.803738 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797181 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:49:26.803738 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797184 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:49:26.803738 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797186 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:49:26.803738 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797189 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:49:26.803738 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797191 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:49:26.803738 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797194 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:49:26.803738 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797197 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:49:26.803738 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797199 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:49:26.803738 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797202 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:49:26.803738 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797204 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:49:26.803738 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797207 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:49:26.803738 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797211 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:49:26.803738 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797214 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:49:26.803738 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797217 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:49:26.803738 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797220 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:49:26.803738 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797223 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:49:26.803738 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797226 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:49:26.803738 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797234 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:49:26.804343 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797237 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:49:26.804343 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797240 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:49:26.804343 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797242 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:49:26.804343 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797245 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:49:26.804343 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797247 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:49:26.804343 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797249 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:49:26.804343 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797252 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:49:26.804343 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797254 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:49:26.804343 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797257 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:49:26.804343 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797259 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:49:26.804343 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797262 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:49:26.804343 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797264 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:49:26.804343 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797267 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:49:26.804343 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797269 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:49:26.804343 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797272 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:49:26.804343 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797275 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:49:26.804343 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797277 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:49:26.804343 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797280 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:49:26.804343 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797282 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:49:26.805102 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797284 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:49:26.805102 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797287 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:49:26.805102 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797290 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:49:26.805102 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797292 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:49:26.805102 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797295 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:49:26.805102 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797297 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:49:26.805102 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797300 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:49:26.805102 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797302 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:49:26.805102 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797305 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:49:26.805102 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797307 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:49:26.805102 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797310 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:49:26.805102 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797313 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:49:26.805102 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797316 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:49:26.805102 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797326 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:49:26.805102 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797328 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:49:26.805102 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797331 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:49:26.805102 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797333 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:49:26.805102 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797336 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:49:26.805102 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797338 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:49:26.805851 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797340 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:49:26.805851 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797343 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:49:26.805851 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797345 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:49:26.805851 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797347 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:49:26.805851 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797350 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:49:26.805851 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797353 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:49:26.805851 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797357 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:49:26.805851 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797360 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:49:26.805851 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797363 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:49:26.805851 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797365 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:49:26.805851 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797368 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:49:26.805851 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797370 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:49:26.805851 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797372 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:49:26.805851 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797375 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:49:26.805851 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797377 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:49:26.805851 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797380 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:49:26.805851 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797382 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:49:26.805851 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797385 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:49:26.805851 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797387 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:49:26.806376 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797390 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:49:26.806376 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797392 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:49:26.806376 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797394 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:49:26.806376 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797397 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:49:26.806376 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797400 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:49:26.806376 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797402 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:49:26.806376 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797405 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:49:26.806376 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797408 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:49:26.806376 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797416 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:49:26.806376 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797418 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:49:26.806376 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797421 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:49:26.806376 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797423 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:49:26.806376 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797426 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:49:26.806376 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797428 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:49:26.806376 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797431 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:49:26.806376 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797433 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:49:26.806376 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797435 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:49:26.806376 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797438 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:49:26.806376 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797440 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:49:26.806844 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.797468 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:49:26.806844 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797656 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:49:26.806844 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797661 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:49:26.806844 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797664 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:49:26.806844 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797666 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:49:26.806844 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797669 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:49:26.806844 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797672 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:49:26.806844 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797675 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:49:26.806844 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797678 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:49:26.806844 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797681 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:49:26.806844 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797684 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:49:26.806844 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797687 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:49:26.806844 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797689 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:49:26.806844 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797692 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:49:26.806844 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797694 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:49:26.806844 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797697 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:49:26.807228 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797699 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:49:26.807228 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797702 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:49:26.807228 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797704 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:49:26.807228 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797708 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:49:26.807228 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797710 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:49:26.807228 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797712 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:49:26.807228 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797721 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:49:26.807228 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797724 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:49:26.807228 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797726 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:49:26.807228 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797729 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:49:26.807228 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797731 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:49:26.807228 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797734 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:49:26.807228 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797736 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:49:26.807228 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797738 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:49:26.807228 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797741 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:49:26.807228 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797743 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:49:26.807228 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797746 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:49:26.807228 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797749 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:49:26.807228 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797751 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:49:26.807668 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797754 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:49:26.807668 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797756 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:49:26.807668 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797758 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:49:26.807668 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797761 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:49:26.807668 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797763 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:49:26.807668 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797766 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:49:26.807668 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797768 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:49:26.807668 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797771 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:49:26.807668 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797774 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:49:26.807668 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797776 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:49:26.807668 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797778 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:49:26.807668 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797781 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:49:26.807668 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797783 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:49:26.807668 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797786 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:49:26.807668 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797789 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:49:26.807668 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797791 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:49:26.807668 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797793 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:49:26.807668 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797796 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:49:26.807668 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797798 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:49:26.808201 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797802 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:49:26.808201 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797828 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:49:26.808201 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797831 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:49:26.808201 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797834 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:49:26.808201 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797836 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:49:26.808201 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797839 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:49:26.808201 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797841 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:49:26.808201 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797844 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:49:26.808201 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797846 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:49:26.808201 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797849 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:49:26.808201 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797851 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:49:26.808201 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797854 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:49:26.808201 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797856 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:49:26.808201 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797859 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:49:26.808201 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797861 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:49:26.808201 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797864 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:49:26.808201 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797866 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:49:26.808201 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797868 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:49:26.808201 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797871 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:49:26.808645 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797873 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:49:26.808645 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797876 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:49:26.808645 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797878 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:49:26.808645 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797886 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:49:26.808645 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797890 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:49:26.808645 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797893 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:49:26.808645 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797896 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:49:26.808645 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797898 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:49:26.808645 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797901 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:49:26.808645 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797904 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:49:26.808645 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797906 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:49:26.808645 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797909 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:49:26.808645 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797911 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:49:26.808645 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797913 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:49:26.808645 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797916 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:49:26.808645 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797924 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:49:26.808645 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797927 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:49:26.808645 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797929 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:49:26.808645 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797931 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:49:26.808645 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797934 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:49:26.809142 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797936 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:49:26.809142 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797939 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:49:26.809142 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797941 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:49:26.809142 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797944 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:49:26.809142 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797946 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:49:26.809142 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797948 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:49:26.809142 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797951 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:49:26.809142 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797953 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:49:26.809142 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797956 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:49:26.809142 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797958 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:49:26.809142 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797960 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:49:26.809142 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797963 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:49:26.809142 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797965 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:49:26.809142 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.797968 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:49:26.809142 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.797991 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:49:26.809142 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798028 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:49:26.809508 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798032 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:49:26.809508 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798035 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:49:26.809508 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798037 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:49:26.809508 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798040 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:49:26.809508 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798042 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:49:26.809508 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798047 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:49:26.809508 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798049 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:49:26.809508 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798052 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:49:26.809508 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798054 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:49:26.809508 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798057 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:49:26.809508 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798060 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:49:26.809508 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798063 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:49:26.809508 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798070 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:49:26.809508 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798073 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:49:26.809508 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798076 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:49:26.809508 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798078 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:49:26.809508 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798080 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:49:26.809508 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798083 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:49:26.809508 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798086 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:49:26.809508 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798088 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:49:26.809998 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798091 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:49:26.809998 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798093 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:49:26.809998 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798096 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:49:26.809998 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798098 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:49:26.809998 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798101 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:49:26.809998 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798103 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:49:26.809998 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798105 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:49:26.809998 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798108 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:49:26.809998 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798110 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:49:26.809998 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798113 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:49:26.809998 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798115 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:49:26.809998 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798117 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:49:26.809998 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798120 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:49:26.809998 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798122 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:49:26.809998 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798124 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:49:26.809998 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798127 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:49:26.809998 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798129 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:49:26.809998 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798133 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:49:26.809998 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798135 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:49:26.809998 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798138 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:49:26.810470 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798141 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:49:26.810470 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798143 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:49:26.810470 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798145 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:49:26.810470 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798148 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:49:26.810470 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798150 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:49:26.810470 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798159 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:49:26.810470 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798161 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:49:26.810470 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798164 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:49:26.810470 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798166 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:49:26.810470 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798169 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:49:26.810470 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798171 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:49:26.810470 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798173 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:49:26.810470 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798176 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:49:26.810470 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798178 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:49:26.810470 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798181 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:49:26.810470 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798183 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:49:26.810470 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798186 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:49:26.810470 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798189 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:49:26.810470 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798191 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:49:26.810931 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798193 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:49:26.810931 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798196 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:49:26.810931 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798198 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:49:26.810931 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798200 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:49:26.810931 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798211 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:49:26.810931 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798214 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:49:26.810931 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798217 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:49:26.810931 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798220 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:49:26.810931 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798222 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:49:26.810931 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798225 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:49:26.810931 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798228 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:49:26.810931 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798231 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:49:26.810931 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798234 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:49:26.810931 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798236 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:49:26.810931 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798239 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:49:26.810931 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798241 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:49:26.810931 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798244 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:49:26.810931 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798248 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:49:26.810931 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798252 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:49:26.811414 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798260 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:49:26.811414 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798263 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:49:26.811414 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798265 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:49:26.811414 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798268 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:49:26.811414 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798270 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:49:26.811414 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798272 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:49:26.811414 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798275 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:49:26.811414 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798277 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:49:26.811414 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798280 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:49:26.811414 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798282 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:49:26.811414 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798284 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:49:26.811414 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798287 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:49:26.811414 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798289 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:49:26.811414 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798292 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:49:26.811414 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798294 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:49:26.811414 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798297 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:49:26.811414 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798299 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:49:26.811414 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798301 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:49:26.811414 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798303 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:49:26.811414 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798306 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:49:26.811997 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798308 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:49:26.811997 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798313 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:49:26.811997 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798317 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:49:26.811997 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798321 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:49:26.811997 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798324 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:49:26.811997 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798327 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:49:26.811997 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:26.798329 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:49:26.811997 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.798388 2576 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:49:26.811997 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.800284 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 21:49:26.811997 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.801233 2576 server.go:1014] "Starting client certificate rotation" Oct 30 21:49:26.811997 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.801323 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:49:26.811997 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.802055 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:49:26.811997 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.803122 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:49:26.811997 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.807298 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:49:26.818107 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.818093 2576 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:49:26.821272 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.821260 2576 log.go:25] "Validated CRI v1 image API" Oct 30 21:49:26.822093 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.822066 2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:49:26.825149 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.825129 2576 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c993a38e-afe9-456e-bb91-d227f9a63332:/dev/nvme0n1p4 efbbe2c6-6ea2-412f-a897-a1d2a68314fa:/dev/nvme0n1p3] Oct 30 21:49:26.825200 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.825149 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 21:49:26.829944 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.829841 2576 manager.go:217] Machine: {Timestamp:2026-10-30 21:49:26.828694914 +0000 UTC m=+0.324470796 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097065 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec298e9147d176c257d2d4c9379a8bba SystemUUID:ec298e91-47d1-76c2-57d2-d4c9379a8bba BootID:43e37fc7-77f8-4761-ae62-e22182a891f6 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:51:9a:1c:aa:a7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:51:9a:1c:aa:a7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:3a:57:37:91:10 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:49:26.829944 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.829943 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:49:26.830053 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.830011 2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:49:26.832847 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.832818 2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:49:26.832976 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.832849 2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-81.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:49:26.833019 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.832978 2576 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:49:26.833019 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.832986 2576 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:49:26.833019 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.832997 2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:49:26.833764 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.833753 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:49:26.834517 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.834507 2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:49:26.838059 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.838048 2576 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:49:26.838178 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.838169 2576 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:49:26.838729 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.838720 2576 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:49:26.838761 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.838733 2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:49:26.838761 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.838745 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:49:26.838761 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.838757 2576 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:49:26.838870 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.838765 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:49:26.839133 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.839121 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:49:26.840981 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.840958 2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:49:26.845126 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.845113 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:49:26.845930 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.845917 2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:49:26.845966 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.845948 2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:49:26.846106 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.846094 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:49:26.846161 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.846109 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:49:26.846161 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.846115 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:49:26.846161 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.846122 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:49:26.846161 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.846128 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:49:26.846161 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.846143 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:49:26.846696 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.846686 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:49:26.846696 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.846696 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:49:26.846754 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.846704 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:49:26.846754 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.846710 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:49:26.846754 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.846719 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:49:26.846754 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.846729 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:49:26.847499 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.847490 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:49:26.847499 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.847499 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:49:26.851128 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.851117 2576 server.go:1263] "Started kubelet" Oct 30 21:49:26.851688 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.851236 2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:49:26.851688 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.851579 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:49:26.851836 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.851715 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:49:26.852374 ip-10-0-137-81 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:49:26.854239 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.854216 2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:49:26.856148 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.856131 2576 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:49:26.857090 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:26.857072 2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 21:49:26.858611 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.858591 2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:49:26.858699 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.858610 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:49:26.858786 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:26.858768 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-81.us-west-1.compute.internal\" not found" Oct 30 21:49:26.859422 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.859240 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:49:26.859422 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.859349 2576 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:49:26.859422 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.859365 2576 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:49:26.859577 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.859505 2576 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:49:26.859577 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.859513 2576 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:49:26.860027 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.860011 2576 factory.go:55] Registering systemd factory Oct 30 21:49:26.860027 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.860027 2576 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:49:26.860361 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.860329 2576 factory.go:153] Registering CRI-O factory Oct 30 21:49:26.860361 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.860352 2576 factory.go:223] Registration of the crio container factory successfully Oct 30 21:49:26.860493 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.860409 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:49:26.860493 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.860431 2576 factory.go:103] Registering Raw factory Oct 30 21:49:26.860493 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.860447 2576 manager.go:1199] Started watching for new ooms in manager Oct 30 21:49:26.861345 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.861327 2576 manager.go:319] Starting recovery of all containers Oct 30 21:49:26.871199 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.871182 2576 manager.go:324] Recovery completed Oct 30 21:49:26.872295 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.872268 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:49:26.873075 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:26.873047 2576 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": readdirent /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Oct 30 21:49:26.876548 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.876534 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:49:26.878400 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.878383 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:49:26.878462 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.878416 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:49:26.878462 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.878429 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:49:26.878954 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.878941 2576 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:49:26.878954 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.878951 2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:49:26.879050 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.878967 2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:49:26.879050 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.878989 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:49:26.879753 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.879739 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:49:26.879792 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.879769 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:49:26.879854 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.879801 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:49:26.881263 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.881250 2576 policy_none.go:50] "Start" Oct 30 21:49:26.881313 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.881267 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:49:26.881313 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.881277 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:49:26.883351 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.883340 2576 policy_none.go:44] "Start" Oct 30 21:49:26.922017 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.921834 2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:49:26.925916 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.925901 2576 manager.go:341] "Starting Device Plugin manager" Oct 30 21:49:26.926433 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:26.926410 2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:49:26.926433 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.926430 2576 server.go:87] "Starting device plugin registration server" Oct 30 21:49:26.926633 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.926618 2576 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:49:26.926727 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.926633 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:49:26.926788 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.926752 2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:49:26.926863 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.926855 2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:49:26.926917 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.926865 2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:49:26.927275 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:26.927250 2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:49:26.927373 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:26.927310 2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-81.us-west-1.compute.internal\" not found" Oct 30 21:49:26.986967 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.986946 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:49:26.986967 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.986967 2576 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:49:26.989035 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:26.986986 2576 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:49:26.989035 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:26.987019 2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 21:49:27.001665 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.001639 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-81.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 21:49:27.001665 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:27.001640 2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-81.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 21:49:27.002615 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:27.001634 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-81.us-west-1.compute.internal.18e36d0a1e6c05bd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-81.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-81.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:49:26.851085757 +0000 UTC m=+0.346861639,LastTimestamp:2026-10-30 21:49:26.851085757 +0000 UTC m=+0.346861639,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-81.us-west-1.compute.internal,}" Oct 30 21:49:27.027157 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.027140 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:49:27.027920 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.027897 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:49:27.028000 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.027930 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:49:27.028000 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.027946 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:49:27.028000 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.027981 2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:49:27.085458 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.085397 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:49:27.085458 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.085458 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8qq8q" Oct 30 21:49:27.087457 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.087443 2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal"] Oct 30 21:49:27.087499 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.087491 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:49:27.088283 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.088267 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:49:27.088374 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.088299 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:49:27.088374 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.088314 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:49:27.090514 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.090499 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:49:27.090641 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.090626 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:49:27.090718 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.090655 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:49:27.091290 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.091273 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:49:27.091356 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.091303 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:49:27.091356 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.091315 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:49:27.091356 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.091275 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:49:27.091356 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.091341 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:49:27.091356 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.091354 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:49:27.093363 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.093350 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:49:27.093404 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.093377 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:49:27.094141 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.094124 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:49:27.094210 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.094160 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:49:27.094210 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.094175 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:49:27.125803 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:27.125783 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-81.us-west-1.compute.internal\" not found" node="ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:49:27.132146 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:27.132121 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-81.us-west-1.compute.internal\" not found" node="ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:49:27.160438 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.160408 2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:49:27.161528 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.161510 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ff2bf62cbfb1edc876c4b4c7d0808622-config\") pod \"kube-apiserver-proxy-ip-10-0-137-81.us-west-1.compute.internal\" (UID: \"ff2bf62cbfb1edc876c4b4c7d0808622\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:49:27.161587 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.161542 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8bdaae95a41fc764b37958d5699dbc6f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal\" (UID: \"8bdaae95a41fc764b37958d5699dbc6f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:49:27.161587 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.161561 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bdaae95a41fc764b37958d5699dbc6f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal\" (UID: \"8bdaae95a41fc764b37958d5699dbc6f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:49:27.162222 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.162202 2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:49:27.259739 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.259692 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:49:27.262020 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.262000 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ff2bf62cbfb1edc876c4b4c7d0808622-config\") pod \"kube-apiserver-proxy-ip-10-0-137-81.us-west-1.compute.internal\" (UID: \"ff2bf62cbfb1edc876c4b4c7d0808622\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:49:27.262125 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.262029 2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:49:27.262125 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.262034 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8bdaae95a41fc764b37958d5699dbc6f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal\" (UID: \"8bdaae95a41fc764b37958d5699dbc6f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:49:27.262125 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.262063 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bdaae95a41fc764b37958d5699dbc6f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal\" (UID: \"8bdaae95a41fc764b37958d5699dbc6f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:49:27.262278 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.262127 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bdaae95a41fc764b37958d5699dbc6f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal\" (UID: \"8bdaae95a41fc764b37958d5699dbc6f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:49:27.262278 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.262128 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ff2bf62cbfb1edc876c4b4c7d0808622-config\") pod \"kube-apiserver-proxy-ip-10-0-137-81.us-west-1.compute.internal\" (UID: \"ff2bf62cbfb1edc876c4b4c7d0808622\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:49:27.262278 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.262127 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8bdaae95a41fc764b37958d5699dbc6f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal\" (UID: \"8bdaae95a41fc764b37958d5699dbc6f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:49:27.262278 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.262229 2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:49:27.359061 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.358985 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:49:27.423793 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.423769 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:49:27.423900 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.423890 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:49:27.428118 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.428102 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:49:27.435465 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.435450 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:49:27.483872 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.483850 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:49:27.801868 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.801841 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 21:49:27.802434 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.802021 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 21:49:27.802434 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.802028 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 21:49:27.802434 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:27.801979 2576 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://aa7caa7bfd2874950af3c2a870e3e248-6e00b72b7f2ae867.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-137-81.us-west-1.compute.internal.18e36d0a200d50f8\": read tcp 10.0.137.81:55004->18.227.6.60:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-137-81.us-west-1.compute.internal.18e36d0a200d50f8 default 5409 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-81.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-137-81.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-137-81.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:49:26 +0000 UTC,LastTimestamp:2026-10-30 21:49:27.027951845 +0000 UTC m=+0.523727728,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-81.us-west-1.compute.internal,}" Oct 30 21:49:27.802434 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.802057 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 21:49:27.839831 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.839799 2576 apiserver.go:52] "Watching apiserver" Oct 30 21:49:27.859600 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:27.859581 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 21:49:28.090115 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.090026 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:44:28 +0000 UTC" deadline="2028-03-27 07:38:25.983189532 +0000 UTC" Oct 30 21:49:28.090115 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.090066 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12321h48m57.893127257s" Oct 30 21:49:28.111039 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.111019 2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:49:28.111353 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.111336 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-q8gsv","kube-system/global-pull-secret-syncer-xbwzg","openshift-cluster-node-tuning-operator/tuned-zxdbg","openshift-dns/node-resolver-8qxjv","openshift-image-registry/node-ca-4dqsc","kube-system/konnectivity-agent-hmx87","openshift-multus/multus-additional-cni-plugins-d4fs7","openshift-multus/multus-b7k9p","openshift-network-diagnostics/network-check-target-mg6qm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal","openshift-ovn-kubernetes/ovnkube-node-6th89","kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-west-1.compute.internal","openshift-multus/network-metrics-daemon-h4v24"] Oct 30 21:49:28.113634 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.113618 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-q8gsv" Oct 30 21:49:28.115620 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.115601 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:49:28.115738 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:28.115670 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbwzg" podUID="4630b412-da87-49f7-bbbb-ba3dc820ddd1" Oct 30 21:49:28.117643 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.117627 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.119732 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.119715 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8qxjv" Oct 30 21:49:28.121740 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.121728 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4dqsc" Oct 30 21:49:28.123788 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.123774 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hmx87" Oct 30 21:49:28.125891 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.125878 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:49:28.128084 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.128068 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.130297 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.130281 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:49:28.130355 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:28.130335 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg6qm" podUID="2e5352ac-8295-494d-950a-16c09b09272e" Oct 30 21:49:28.132516 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.132502 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:49:28.134819 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.134790 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.137129 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.137082 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:49:28.137287 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:28.137253 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4v24" podUID="83a9baf0-064d-4544-94b4-a0dd090dbd04" Oct 30 21:49:28.161034 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.161016 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:49:28.166868 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.166843 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/18a7ec6e-13fa-4893-a179-529f13fb2172-konnectivity-ca\") pod \"konnectivity-agent-hmx87\" (UID: \"18a7ec6e-13fa-4893-a179-529f13fb2172\") " pod="kube-system/konnectivity-agent-hmx87" Oct 30 21:49:28.166965 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.166888 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-sys\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.166965 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.166906 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:49:28.166965 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.166911 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-host-var-lib-cni-multus\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.166965 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.166927 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-run-openvswitch\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.167168 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.166971 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-cni-netd\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.167168 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.166995 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-sysctl-d\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.167168 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167018 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4105fd0-16f9-492e-ae10-1f7458524eeb-cnibin\") pod \"multus-additional-cni-plugins-d4fs7\" (UID: \"b4105fd0-16f9-492e-ae10-1f7458524eeb\") " pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:49:28.167168 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167042 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-host-var-lib-cni-bin\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.167168 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167070 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:49:28.167168 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167066 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp9qb\" (UniqueName: \"kubernetes.io/projected/d2226976-1041-4798-b567-6fb6590c0f6f-kube-api-access-pp9qb\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.167168 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167123 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f5d7f625-40b0-46b6-8489-fdd1daa0068e-registration-dir\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:49:28.167168 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167130 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:49:28.167168 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167148 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-var-lib-openvswitch\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.167168 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167130 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qjhgs\"" Oct 30 21:49:28.167168 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167164 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-node-log\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.167663 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167185 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7fd1e2d6-4877-4459-82f9-95b31d098229-ovn-node-metrics-cert\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.167663 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167206 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-var-lib-kubelet\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.167663 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167222 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-host-run-k8s-cni-cncf-io\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.167663 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167236 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e7b50ca6-0f9c-40f8-a09d-004b055105b8-iptables-alerter-script\") pod \"iptables-alerter-q8gsv\" (UID: \"e7b50ca6-0f9c-40f8-a09d-004b055105b8\") " pod="openshift-network-operator/iptables-alerter-q8gsv" Oct 30 21:49:28.167663 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167254 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-kubernetes\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.167663 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167275 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f5d7f625-40b0-46b6-8489-fdd1daa0068e-device-dir\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:49:28.167663 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167296 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-run-netns\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.167663 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167310 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-log-socket\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.167663 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167325 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svfrj\" (UniqueName: \"kubernetes.io/projected/e7b50ca6-0f9c-40f8-a09d-004b055105b8-kube-api-access-svfrj\") pod \"iptables-alerter-q8gsv\" (UID: \"e7b50ca6-0f9c-40f8-a09d-004b055105b8\") " pod="openshift-network-operator/iptables-alerter-q8gsv" Oct 30 21:49:28.167663 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167343 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4630b412-da87-49f7-bbbb-ba3dc820ddd1-kubelet-config\") pod \"global-pull-secret-syncer-xbwzg\" (UID: \"4630b412-da87-49f7-bbbb-ba3dc820ddd1\") " pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:49:28.167663 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167364 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4630b412-da87-49f7-bbbb-ba3dc820ddd1-dbus\") pod \"global-pull-secret-syncer-xbwzg\" (UID: \"4630b412-da87-49f7-bbbb-ba3dc820ddd1\") " pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:49:28.167663 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167381 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6q6v\" (UniqueName: \"kubernetes.io/projected/9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8-kube-api-access-q6q6v\") pod \"node-ca-4dqsc\" (UID: \"9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8\") " pod="openshift-image-registry/node-ca-4dqsc" Oct 30 21:49:28.167663 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167403 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4xbt\" (UniqueName: \"kubernetes.io/projected/b4105fd0-16f9-492e-ae10-1f7458524eeb-kube-api-access-t4xbt\") pod \"multus-additional-cni-plugins-d4fs7\" (UID: \"b4105fd0-16f9-492e-ae10-1f7458524eeb\") " pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:49:28.167663 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167418 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-multus-cni-dir\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.167663 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167437 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkk9k\" (UniqueName: \"kubernetes.io/projected/f5d7f625-40b0-46b6-8489-fdd1daa0068e-kube-api-access-kkk9k\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:49:28.167663 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167452 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7fd1e2d6-4877-4459-82f9-95b31d098229-ovnkube-config\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.168395 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167476 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f5d7f625-40b0-46b6-8489-fdd1daa0068e-sys-fs\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:49:28.168395 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167507 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-slash\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.168395 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167530 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7b50ca6-0f9c-40f8-a09d-004b055105b8-host-slash\") pod \"iptables-alerter-q8gsv\" (UID: \"e7b50ca6-0f9c-40f8-a09d-004b055105b8\") " pod="openshift-network-operator/iptables-alerter-q8gsv" Oct 30 21:49:28.168395 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167543 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-run\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.168395 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167559 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-host\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.168395 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167582 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b4105fd0-16f9-492e-ae10-1f7458524eeb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d4fs7\" (UID: \"b4105fd0-16f9-492e-ae10-1f7458524eeb\") " pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:49:28.168395 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167602 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-system-cni-dir\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.168395 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167615 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-cnibin\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.168395 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167679 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f5d7f625-40b0-46b6-8489-fdd1daa0068e-etc-selinux\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:49:28.168395 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167701 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-systemd-units\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.168395 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167721 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp9j4\" (UniqueName: \"kubernetes.io/projected/83a9baf0-064d-4544-94b4-a0dd090dbd04-kube-api-access-qp9j4\") pod \"network-metrics-daemon-h4v24\" (UID: \"83a9baf0-064d-4544-94b4-a0dd090dbd04\") " pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:49:28.168395 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167738 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d2226976-1041-4798-b567-6fb6590c0f6f-cni-binary-copy\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.168395 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167753 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-host-var-lib-kubelet\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.168395 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167766 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-run-systemd\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.168395 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167787 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/579c3ec8-fe12-472f-8ab0-247bfa8ed8b7-hosts-file\") pod \"node-resolver-8qxjv\" (UID: \"579c3ec8-fe12-472f-8ab0-247bfa8ed8b7\") " pod="openshift-dns/node-resolver-8qxjv" Oct 30 21:49:28.168395 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167818 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj8hr\" (UniqueName: \"kubernetes.io/projected/2e5352ac-8295-494d-950a-16c09b09272e-kube-api-access-dj8hr\") pod \"network-check-target-mg6qm\" (UID: \"2e5352ac-8295-494d-950a-16c09b09272e\") " pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:49:28.168957 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167837 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f5d7f625-40b0-46b6-8489-fdd1daa0068e-socket-dir\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:49:28.168957 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167852 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/18a7ec6e-13fa-4893-a179-529f13fb2172-agent-certs\") pod \"konnectivity-agent-hmx87\" (UID: \"18a7ec6e-13fa-4893-a179-529f13fb2172\") " pod="kube-system/konnectivity-agent-hmx87" Oct 30 21:49:28.168957 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167866 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-modprobe-d\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.168957 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167884 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-lib-modules\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.168957 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167910 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4105fd0-16f9-492e-ae10-1f7458524eeb-system-cni-dir\") pod \"multus-additional-cni-plugins-d4fs7\" (UID: \"b4105fd0-16f9-492e-ae10-1f7458524eeb\") " pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:49:28.168957 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167934 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a9baf0-064d-4544-94b4-a0dd090dbd04-metrics-certs\") pod \"network-metrics-daemon-h4v24\" (UID: \"83a9baf0-064d-4544-94b4-a0dd090dbd04\") " pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:49:28.168957 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167950 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-tuned\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.168957 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.167992 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-host-run-multus-certs\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.168957 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.168019 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-hostroot\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.168957 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.168045 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-multus-conf-dir\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.168957 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.168061 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f5d7f625-40b0-46b6-8489-fdd1daa0068e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:49:28.168957 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.168078 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7fd1e2d6-4877-4459-82f9-95b31d098229-env-overrides\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.168957 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.168098 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfsd9\" (UniqueName: \"kubernetes.io/projected/579c3ec8-fe12-472f-8ab0-247bfa8ed8b7-kube-api-access-tfsd9\") pod \"node-resolver-8qxjv\" (UID: \"579c3ec8-fe12-472f-8ab0-247bfa8ed8b7\") " pod="openshift-dns/node-resolver-8qxjv" Oct 30 21:49:28.168957 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.168132 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8-serviceca\") pod \"node-ca-4dqsc\" (UID: \"9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8\") " pod="openshift-image-registry/node-ca-4dqsc" Oct 30 21:49:28.168957 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.168160 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d962c24e-5855-453e-97d5-78bad4a19fbc-tmp\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.168957 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.168177 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b4105fd0-16f9-492e-ae10-1f7458524eeb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d4fs7\" (UID: \"b4105fd0-16f9-492e-ae10-1f7458524eeb\") " pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:49:28.169379 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.168219 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-etc-kubernetes\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.169379 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.168241 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-etc-openvswitch\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.169379 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.168255 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-run-ovn\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.169379 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.168283 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-run-ovn-kubernetes\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.169379 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.168303 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.169379 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.168327 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-sysconfig\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.169379 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.168342 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-systemd\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.169379 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.168364 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-host-run-netns\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.169379 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.168380 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-cni-bin\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.169379 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.168397 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4630b412-da87-49f7-bbbb-ba3dc820ddd1-original-pull-secret\") pod \"global-pull-secret-syncer-xbwzg\" (UID: \"4630b412-da87-49f7-bbbb-ba3dc820ddd1\") " pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:49:28.169379 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.168412 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/579c3ec8-fe12-472f-8ab0-247bfa8ed8b7-tmp-dir\") pod \"node-resolver-8qxjv\" (UID: \"579c3ec8-fe12-472f-8ab0-247bfa8ed8b7\") " pod="openshift-dns/node-resolver-8qxjv" Oct 30 21:49:28.169379 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.168431 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-sysctl-conf\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.169379 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.168453 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6nx7\" (UniqueName: \"kubernetes.io/projected/d962c24e-5855-453e-97d5-78bad4a19fbc-kube-api-access-h6nx7\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.169379 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.168467 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp4k2\" (UniqueName: \"kubernetes.io/projected/7fd1e2d6-4877-4459-82f9-95b31d098229-kube-api-access-lp4k2\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.169379 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.168510 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d2226976-1041-4798-b567-6fb6590c0f6f-multus-daemon-config\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.169379 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.168550 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8-host\") pod \"node-ca-4dqsc\" (UID: \"9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8\") " pod="openshift-image-registry/node-ca-4dqsc" Oct 30 21:49:28.169819 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.168589 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-multus-socket-dir-parent\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.169819 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.168624 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-kubelet\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.169819 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.168642 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7fd1e2d6-4877-4459-82f9-95b31d098229-ovnkube-script-lib\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.169819 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.168656 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b4105fd0-16f9-492e-ae10-1f7458524eeb-cni-binary-copy\") pod \"multus-additional-cni-plugins-d4fs7\" (UID: \"b4105fd0-16f9-492e-ae10-1f7458524eeb\") " pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:49:28.169819 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.168674 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4105fd0-16f9-492e-ae10-1f7458524eeb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d4fs7\" (UID: \"b4105fd0-16f9-492e-ae10-1f7458524eeb\") " pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:49:28.170027 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.169849 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:49:28.170027 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.169933 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9mdmg\"" Oct 30 21:49:28.170027 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.169990 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:49:28.171757 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.171744 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:49:28.171830 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.171803 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pqcb5\"" Oct 30 21:49:28.171926 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.171915 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:49:28.174039 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.174027 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-k97q2\"" Oct 30 21:49:28.174099 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.174041 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:49:28.174099 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.174059 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:49:28.174099 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.174083 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:49:28.175870 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.175858 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:49:28.175956 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.175943 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:49:28.175994 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.175947 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-f5s29\"" Oct 30 21:49:28.211476 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.211456 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rrcm9\"" Oct 30 21:49:28.211559 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.211458 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-w9pjm\"" Oct 30 21:49:28.212396 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.212285 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:49:28.212396 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.212323 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:49:28.212396 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.212328 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:49:28.212396 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.212340 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:49:28.212396 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.212338 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fs82w\"" Oct 30 21:49:28.212396 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.212353 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:49:28.212396 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.212366 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:49:28.212655 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.212415 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:49:28.212655 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.212425 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:49:28.213678 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.213658 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:49:28.213764 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.213687 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:49:28.213853 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.213840 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wvpps\"" Oct 30 21:49:28.214709 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.214690 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:49:28.214764 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.214698 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:49:28.214764 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.214754 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:49:28.215470 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.215457 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:49:28.216453 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.216430 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:49:28.217965 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.217949 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:49:28.218017 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.218004 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dwnc7" Oct 30 21:49:28.269465 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.269436 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e7b50ca6-0f9c-40f8-a09d-004b055105b8-iptables-alerter-script\") pod \"iptables-alerter-q8gsv\" (UID: \"e7b50ca6-0f9c-40f8-a09d-004b055105b8\") " pod="openshift-network-operator/iptables-alerter-q8gsv" Oct 30 21:49:28.269586 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.269478 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-kubernetes\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.269586 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.269502 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f5d7f625-40b0-46b6-8489-fdd1daa0068e-device-dir\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:49:28.269586 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.269530 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-run-netns\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.269586 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.269554 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-log-socket\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.269586 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.269567 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-kubernetes\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.269586 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.269579 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svfrj\" (UniqueName: \"kubernetes.io/projected/e7b50ca6-0f9c-40f8-a09d-004b055105b8-kube-api-access-svfrj\") pod \"iptables-alerter-q8gsv\" (UID: \"e7b50ca6-0f9c-40f8-a09d-004b055105b8\") " pod="openshift-network-operator/iptables-alerter-q8gsv" Oct 30 21:49:28.269941 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.269604 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4630b412-da87-49f7-bbbb-ba3dc820ddd1-kubelet-config\") pod \"global-pull-secret-syncer-xbwzg\" (UID: \"4630b412-da87-49f7-bbbb-ba3dc820ddd1\") " pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:49:28.269941 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.269629 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4630b412-da87-49f7-bbbb-ba3dc820ddd1-dbus\") pod \"global-pull-secret-syncer-xbwzg\" (UID: \"4630b412-da87-49f7-bbbb-ba3dc820ddd1\") " pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:49:28.269941 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.269632 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f5d7f625-40b0-46b6-8489-fdd1daa0068e-device-dir\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:49:28.269941 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.269651 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6q6v\" (UniqueName: \"kubernetes.io/projected/9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8-kube-api-access-q6q6v\") pod \"node-ca-4dqsc\" (UID: \"9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8\") " pod="openshift-image-registry/node-ca-4dqsc" Oct 30 21:49:28.269941 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.269670 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-log-socket\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.269941 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.269674 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4xbt\" (UniqueName: \"kubernetes.io/projected/b4105fd0-16f9-492e-ae10-1f7458524eeb-kube-api-access-t4xbt\") pod \"multus-additional-cni-plugins-d4fs7\" (UID: \"b4105fd0-16f9-492e-ae10-1f7458524eeb\") " pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:49:28.269941 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.269698 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-multus-cni-dir\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.269941 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.269723 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkk9k\" (UniqueName: \"kubernetes.io/projected/f5d7f625-40b0-46b6-8489-fdd1daa0068e-kube-api-access-kkk9k\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:49:28.269941 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.269749 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7fd1e2d6-4877-4459-82f9-95b31d098229-ovnkube-config\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.269941 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.269771 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f5d7f625-40b0-46b6-8489-fdd1daa0068e-sys-fs\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:49:28.269941 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.269789 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4630b412-da87-49f7-bbbb-ba3dc820ddd1-dbus\") pod \"global-pull-secret-syncer-xbwzg\" (UID: \"4630b412-da87-49f7-bbbb-ba3dc820ddd1\") " pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:49:28.269941 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.269794 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-slash\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.269941 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.269723 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4630b412-da87-49f7-bbbb-ba3dc820ddd1-kubelet-config\") pod \"global-pull-secret-syncer-xbwzg\" (UID: \"4630b412-da87-49f7-bbbb-ba3dc820ddd1\") " pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:49:28.269941 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.269853 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7b50ca6-0f9c-40f8-a09d-004b055105b8-host-slash\") pod \"iptables-alerter-q8gsv\" (UID: \"e7b50ca6-0f9c-40f8-a09d-004b055105b8\") " pod="openshift-network-operator/iptables-alerter-q8gsv" Oct 30 21:49:28.269941 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.269870 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-run\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.269941 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.269884 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-host\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.269941 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.269901 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b4105fd0-16f9-492e-ae10-1f7458524eeb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d4fs7\" (UID: \"b4105fd0-16f9-492e-ae10-1f7458524eeb\") " pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:49:28.270784 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.269927 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-system-cni-dir\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.270784 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.269949 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-cnibin\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.270784 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.269972 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f5d7f625-40b0-46b6-8489-fdd1daa0068e-etc-selinux\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:49:28.270784 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270014 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-systemd-units\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.270784 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270042 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp9j4\" (UniqueName: \"kubernetes.io/projected/83a9baf0-064d-4544-94b4-a0dd090dbd04-kube-api-access-qp9j4\") pod \"network-metrics-daemon-h4v24\" (UID: \"83a9baf0-064d-4544-94b4-a0dd090dbd04\") " pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:49:28.270784 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.269628 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-run-netns\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.270784 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270050 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e7b50ca6-0f9c-40f8-a09d-004b055105b8-iptables-alerter-script\") pod \"iptables-alerter-q8gsv\" (UID: \"e7b50ca6-0f9c-40f8-a09d-004b055105b8\") " pod="openshift-network-operator/iptables-alerter-q8gsv" Oct 30 21:49:28.270784 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270079 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d2226976-1041-4798-b567-6fb6590c0f6f-cni-binary-copy\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.270784 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270106 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-host-var-lib-kubelet\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.270784 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270113 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-multus-cni-dir\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.270784 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270140 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-run-systemd\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.270784 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270166 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/579c3ec8-fe12-472f-8ab0-247bfa8ed8b7-hosts-file\") pod \"node-resolver-8qxjv\" (UID: \"579c3ec8-fe12-472f-8ab0-247bfa8ed8b7\") " pod="openshift-dns/node-resolver-8qxjv" Oct 30 21:49:28.270784 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270201 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj8hr\" (UniqueName: \"kubernetes.io/projected/2e5352ac-8295-494d-950a-16c09b09272e-kube-api-access-dj8hr\") pod \"network-check-target-mg6qm\" (UID: \"2e5352ac-8295-494d-950a-16c09b09272e\") " pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:49:28.270784 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270224 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f5d7f625-40b0-46b6-8489-fdd1daa0068e-socket-dir\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:49:28.270784 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270251 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/18a7ec6e-13fa-4893-a179-529f13fb2172-agent-certs\") pod \"konnectivity-agent-hmx87\" (UID: \"18a7ec6e-13fa-4893-a179-529f13fb2172\") " pod="kube-system/konnectivity-agent-hmx87" Oct 30 21:49:28.270784 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270266 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-modprobe-d\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.270784 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270281 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-lib-modules\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.271338 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270290 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-systemd-units\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.271338 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270295 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4105fd0-16f9-492e-ae10-1f7458524eeb-system-cni-dir\") pod \"multus-additional-cni-plugins-d4fs7\" (UID: \"b4105fd0-16f9-492e-ae10-1f7458524eeb\") " pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:49:28.271338 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270309 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f5d7f625-40b0-46b6-8489-fdd1daa0068e-etc-selinux\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:49:28.271338 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270334 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a9baf0-064d-4544-94b4-a0dd090dbd04-metrics-certs\") pod \"network-metrics-daemon-h4v24\" (UID: \"83a9baf0-064d-4544-94b4-a0dd090dbd04\") " pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:49:28.271338 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270353 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4105fd0-16f9-492e-ae10-1f7458524eeb-system-cni-dir\") pod \"multus-additional-cni-plugins-d4fs7\" (UID: \"b4105fd0-16f9-492e-ae10-1f7458524eeb\") " pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:49:28.271338 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270362 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-tuned\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.271338 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270362 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-system-cni-dir\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.271338 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270387 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-host-run-multus-certs\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.271338 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270405 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-cnibin\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.271338 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270413 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-hostroot\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.271338 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270417 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f5d7f625-40b0-46b6-8489-fdd1daa0068e-sys-fs\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:49:28.271338 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270426 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7b50ca6-0f9c-40f8-a09d-004b055105b8-host-slash\") pod \"iptables-alerter-q8gsv\" (UID: \"e7b50ca6-0f9c-40f8-a09d-004b055105b8\") " pod="openshift-network-operator/iptables-alerter-q8gsv" Oct 30 21:49:28.271338 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270438 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-multus-conf-dir\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.271338 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270464 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f5d7f625-40b0-46b6-8489-fdd1daa0068e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:49:28.271338 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270472 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-host\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.271338 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270478 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7fd1e2d6-4877-4459-82f9-95b31d098229-ovnkube-config\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.271338 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270488 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7fd1e2d6-4877-4459-82f9-95b31d098229-env-overrides\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.271338 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:28.270512 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:49:28.271888 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270527 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-hostroot\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.271888 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270541 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-host-run-multus-certs\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.271888 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270555 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfsd9\" (UniqueName: \"kubernetes.io/projected/579c3ec8-fe12-472f-8ab0-247bfa8ed8b7-kube-api-access-tfsd9\") pod \"node-resolver-8qxjv\" (UID: \"579c3ec8-fe12-472f-8ab0-247bfa8ed8b7\") " pod="openshift-dns/node-resolver-8qxjv" Oct 30 21:49:28.271888 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:28.270585 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83a9baf0-064d-4544-94b4-a0dd090dbd04-metrics-certs podName:83a9baf0-064d-4544-94b4-a0dd090dbd04 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:28.770565361 +0000 UTC m=+2.266341254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/83a9baf0-064d-4544-94b4-a0dd090dbd04-metrics-certs") pod "network-metrics-daemon-h4v24" (UID: "83a9baf0-064d-4544-94b4-a0dd090dbd04") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:49:28.271888 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270604 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8-serviceca\") pod \"node-ca-4dqsc\" (UID: \"9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8\") " pod="openshift-image-registry/node-ca-4dqsc" Oct 30 21:49:28.271888 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270631 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d962c24e-5855-453e-97d5-78bad4a19fbc-tmp\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.271888 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270694 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b4105fd0-16f9-492e-ae10-1f7458524eeb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d4fs7\" (UID: \"b4105fd0-16f9-492e-ae10-1f7458524eeb\") " pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:49:28.271888 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270710 2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:49:28.271888 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270723 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-etc-kubernetes\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.271888 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270745 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d2226976-1041-4798-b567-6fb6590c0f6f-cni-binary-copy\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.271888 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270768 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-etc-openvswitch\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.271888 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270791 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-run-ovn\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.271888 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270849 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-run-ovn-kubernetes\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.271888 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270862 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-run-systemd\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.271888 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270878 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.271888 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270882 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7fd1e2d6-4877-4459-82f9-95b31d098229-env-overrides\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.271888 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270902 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-sysconfig\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.272826 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270910 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/579c3ec8-fe12-472f-8ab0-247bfa8ed8b7-hosts-file\") pod \"node-resolver-8qxjv\" (UID: \"579c3ec8-fe12-472f-8ab0-247bfa8ed8b7\") " pod="openshift-dns/node-resolver-8qxjv" Oct 30 21:49:28.272826 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270925 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-systemd\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.272826 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270932 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-multus-conf-dir\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.272826 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270948 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-host-run-netns\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.272826 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270955 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f5d7f625-40b0-46b6-8489-fdd1daa0068e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:49:28.272826 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270222 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-slash\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.272826 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270975 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-cni-bin\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.272826 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271000 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4630b412-da87-49f7-bbbb-ba3dc820ddd1-original-pull-secret\") pod \"global-pull-secret-syncer-xbwzg\" (UID: \"4630b412-da87-49f7-bbbb-ba3dc820ddd1\") " pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:49:28.272826 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271015 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-etc-openvswitch\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.272826 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270513 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-run\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.272826 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271026 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/579c3ec8-fe12-472f-8ab0-247bfa8ed8b7-tmp-dir\") pod \"node-resolver-8qxjv\" (UID: \"579c3ec8-fe12-472f-8ab0-247bfa8ed8b7\") " pod="openshift-dns/node-resolver-8qxjv" Oct 30 21:49:28.272826 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271039 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f5d7f625-40b0-46b6-8489-fdd1daa0068e-socket-dir\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:49:28.272826 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271072 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-sysctl-conf\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.272826 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271098 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6nx7\" (UniqueName: \"kubernetes.io/projected/d962c24e-5855-453e-97d5-78bad4a19fbc-kube-api-access-h6nx7\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.272826 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271114 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp4k2\" (UniqueName: \"kubernetes.io/projected/7fd1e2d6-4877-4459-82f9-95b31d098229-kube-api-access-lp4k2\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.272826 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271130 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d2226976-1041-4798-b567-6fb6590c0f6f-multus-daemon-config\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.272826 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271145 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8-host\") pod \"node-ca-4dqsc\" (UID: \"9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8\") " pod="openshift-image-registry/node-ca-4dqsc" Oct 30 21:49:28.272826 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271160 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-multus-socket-dir-parent\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.273657 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271176 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-kubelet\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.273657 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271194 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7fd1e2d6-4877-4459-82f9-95b31d098229-ovnkube-script-lib\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.273657 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271210 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b4105fd0-16f9-492e-ae10-1f7458524eeb-cni-binary-copy\") pod \"multus-additional-cni-plugins-d4fs7\" (UID: \"b4105fd0-16f9-492e-ae10-1f7458524eeb\") " pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:49:28.273657 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271228 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4105fd0-16f9-492e-ae10-1f7458524eeb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d4fs7\" (UID: \"b4105fd0-16f9-492e-ae10-1f7458524eeb\") " pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:49:28.273657 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271244 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/18a7ec6e-13fa-4893-a179-529f13fb2172-konnectivity-ca\") pod \"konnectivity-agent-hmx87\" (UID: \"18a7ec6e-13fa-4893-a179-529f13fb2172\") " pod="kube-system/konnectivity-agent-hmx87" Oct 30 21:49:28.273657 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271259 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-sys\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.273657 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271273 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-host-var-lib-cni-multus\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.273657 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271288 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-run-openvswitch\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.273657 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271304 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-cni-netd\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.273657 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271308 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-sysconfig\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.273657 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271320 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-sysctl-d\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.273657 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271351 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-run-ovn\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.273657 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271375 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4105fd0-16f9-492e-ae10-1f7458524eeb-cnibin\") pod \"multus-additional-cni-plugins-d4fs7\" (UID: \"b4105fd0-16f9-492e-ae10-1f7458524eeb\") " pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:49:28.273657 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271396 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-sysctl-d\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.273657 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271397 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-host-var-lib-cni-bin\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.273657 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271419 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp9qb\" (UniqueName: \"kubernetes.io/projected/d2226976-1041-4798-b567-6fb6590c0f6f-kube-api-access-pp9qb\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.273657 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271444 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f5d7f625-40b0-46b6-8489-fdd1daa0068e-registration-dir\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:49:28.274370 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271464 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-var-lib-openvswitch\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.274370 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271485 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-node-log\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.274370 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271489 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-sysctl-conf\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.274370 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271505 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7fd1e2d6-4877-4459-82f9-95b31d098229-ovn-node-metrics-cert\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.274370 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271529 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-var-lib-kubelet\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.274370 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271539 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-systemd\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.274370 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271548 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-host-run-k8s-cni-cncf-io\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.274370 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271614 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-host-run-k8s-cni-cncf-io\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.274370 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.270796 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-host-var-lib-kubelet\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.274370 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271654 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-run-ovn-kubernetes\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.274370 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271676 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/579c3ec8-fe12-472f-8ab0-247bfa8ed8b7-tmp-dir\") pod \"node-resolver-8qxjv\" (UID: \"579c3ec8-fe12-472f-8ab0-247bfa8ed8b7\") " pod="openshift-dns/node-resolver-8qxjv" Oct 30 21:49:28.274370 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271722 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8-host\") pod \"node-ca-4dqsc\" (UID: \"9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8\") " pod="openshift-image-registry/node-ca-4dqsc" Oct 30 21:49:28.274370 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271729 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.274370 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271794 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8-serviceca\") pod \"node-ca-4dqsc\" (UID: \"9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8\") " pod="openshift-image-registry/node-ca-4dqsc" Oct 30 21:49:28.274370 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271798 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4105fd0-16f9-492e-ae10-1f7458524eeb-cnibin\") pod \"multus-additional-cni-plugins-d4fs7\" (UID: \"b4105fd0-16f9-492e-ae10-1f7458524eeb\") " pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:49:28.274370 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271945 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-multus-socket-dir-parent\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.274370 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271980 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-kubelet\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.274370 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271986 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-host-var-lib-cni-bin\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.274914 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.272037 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f5d7f625-40b0-46b6-8489-fdd1daa0068e-registration-dir\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:49:28.274914 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.272068 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-var-lib-openvswitch\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.274914 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.272107 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-node-log\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.274914 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.272149 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d2226976-1041-4798-b567-6fb6590c0f6f-multus-daemon-config\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.274914 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.272214 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-cni-bin\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.274914 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:28.272286 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:49:28.274914 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.272358 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b4105fd0-16f9-492e-ae10-1f7458524eeb-cni-binary-copy\") pod \"multus-additional-cni-plugins-d4fs7\" (UID: \"b4105fd0-16f9-492e-ae10-1f7458524eeb\") " pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:49:28.274914 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:28.272441 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4630b412-da87-49f7-bbbb-ba3dc820ddd1-original-pull-secret podName:4630b412-da87-49f7-bbbb-ba3dc820ddd1 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:28.772316945 +0000 UTC m=+2.268092819 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4630b412-da87-49f7-bbbb-ba3dc820ddd1-original-pull-secret") pod "global-pull-secret-syncer-xbwzg" (UID: "4630b412-da87-49f7-bbbb-ba3dc820ddd1") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:49:28.274914 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271209 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-modprobe-d\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.274914 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.272553 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-var-lib-kubelet\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.274914 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271270 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-lib-modules\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.274914 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.272853 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4105fd0-16f9-492e-ae10-1f7458524eeb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d4fs7\" (UID: \"b4105fd0-16f9-492e-ae10-1f7458524eeb\") " pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:49:28.274914 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.272909 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-run-openvswitch\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.274914 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.272955 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-cni-netd\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.274914 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.271487 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b4105fd0-16f9-492e-ae10-1f7458524eeb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d4fs7\" (UID: \"b4105fd0-16f9-492e-ae10-1f7458524eeb\") " pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:49:28.274914 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.273244 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/18a7ec6e-13fa-4893-a179-529f13fb2172-konnectivity-ca\") pod \"konnectivity-agent-hmx87\" (UID: \"18a7ec6e-13fa-4893-a179-529f13fb2172\") " pod="kube-system/konnectivity-agent-hmx87" Oct 30 21:49:28.274914 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.273317 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-sys\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.275421 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.273364 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-host-var-lib-cni-multus\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.275421 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.273407 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-host-run-netns\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.275421 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.273457 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7fd1e2d6-4877-4459-82f9-95b31d098229-ovnkube-script-lib\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.275421 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.273523 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-etc-kubernetes\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.275421 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.274121 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-tuned\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.275421 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.274177 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b4105fd0-16f9-492e-ae10-1f7458524eeb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d4fs7\" (UID: \"b4105fd0-16f9-492e-ae10-1f7458524eeb\") " pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:49:28.275421 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.274272 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d962c24e-5855-453e-97d5-78bad4a19fbc-tmp\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.275421 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.274427 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/18a7ec6e-13fa-4893-a179-529f13fb2172-agent-certs\") pod \"konnectivity-agent-hmx87\" (UID: \"18a7ec6e-13fa-4893-a179-529f13fb2172\") " pod="kube-system/konnectivity-agent-hmx87" Oct 30 21:49:28.275421 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.274537 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7fd1e2d6-4877-4459-82f9-95b31d098229-ovn-node-metrics-cert\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.330329 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:28.330286 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:49:28.330329 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:28.330317 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:49:28.330329 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:28.330329 2576 projected.go:196] Error preparing data for projected volume kube-api-access-dj8hr for pod openshift-network-diagnostics/network-check-target-mg6qm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:49:28.330758 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:28.330409 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e5352ac-8295-494d-950a-16c09b09272e-kube-api-access-dj8hr podName:2e5352ac-8295-494d-950a-16c09b09272e nodeName:}" failed. No retries permitted until 2026-10-30 21:49:28.830381691 +0000 UTC m=+2.326157574 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dj8hr" (UniqueName: "kubernetes.io/projected/2e5352ac-8295-494d-950a-16c09b09272e-kube-api-access-dj8hr") pod "network-check-target-mg6qm" (UID: "2e5352ac-8295-494d-950a-16c09b09272e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:49:28.332252 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.332217 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6q6v\" (UniqueName: \"kubernetes.io/projected/9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8-kube-api-access-q6q6v\") pod \"node-ca-4dqsc\" (UID: \"9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8\") " pod="openshift-image-registry/node-ca-4dqsc" Oct 30 21:49:28.332892 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.332858 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfsd9\" (UniqueName: \"kubernetes.io/projected/579c3ec8-fe12-472f-8ab0-247bfa8ed8b7-kube-api-access-tfsd9\") pod \"node-resolver-8qxjv\" (UID: \"579c3ec8-fe12-472f-8ab0-247bfa8ed8b7\") " pod="openshift-dns/node-resolver-8qxjv" Oct 30 21:49:28.333253 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.333228 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4xbt\" (UniqueName: \"kubernetes.io/projected/b4105fd0-16f9-492e-ae10-1f7458524eeb-kube-api-access-t4xbt\") pod \"multus-additional-cni-plugins-d4fs7\" (UID: \"b4105fd0-16f9-492e-ae10-1f7458524eeb\") " pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:49:28.333440 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.333417 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkk9k\" (UniqueName: \"kubernetes.io/projected/f5d7f625-40b0-46b6-8489-fdd1daa0068e-kube-api-access-kkk9k\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:49:28.333504 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.333461 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6nx7\" (UniqueName: \"kubernetes.io/projected/d962c24e-5855-453e-97d5-78bad4a19fbc-kube-api-access-h6nx7\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.333605 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.333465 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp9j4\" (UniqueName: \"kubernetes.io/projected/83a9baf0-064d-4544-94b4-a0dd090dbd04-kube-api-access-qp9j4\") pod \"network-metrics-daemon-h4v24\" (UID: \"83a9baf0-064d-4544-94b4-a0dd090dbd04\") " pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:49:28.333699 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.333683 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svfrj\" (UniqueName: \"kubernetes.io/projected/e7b50ca6-0f9c-40f8-a09d-004b055105b8-kube-api-access-svfrj\") pod \"iptables-alerter-q8gsv\" (UID: \"e7b50ca6-0f9c-40f8-a09d-004b055105b8\") " pod="openshift-network-operator/iptables-alerter-q8gsv" Oct 30 21:49:28.336600 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.336579 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp9qb\" (UniqueName: \"kubernetes.io/projected/d2226976-1041-4798-b567-6fb6590c0f6f-kube-api-access-pp9qb\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.337118 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.337100 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp4k2\" (UniqueName: \"kubernetes.io/projected/7fd1e2d6-4877-4459-82f9-95b31d098229-kube-api-access-lp4k2\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.448944 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.448881 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-q8gsv" Oct 30 21:49:28.465438 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.465415 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:49:28.487420 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.487395 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8qxjv" Oct 30 21:49:28.505709 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.505688 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4dqsc" Oct 30 21:49:28.516876 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.516856 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hmx87" Oct 30 21:49:28.530543 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:28.530518 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9478ab3b_4826_4ac9_9b7d_0f907dd4c7f8.slice/crio-239c653cefcebd880c0ec32f20cce418280a8611f3b3a5f0ed35ab1717a315de WatchSource:0}: Error finding container 239c653cefcebd880c0ec32f20cce418280a8611f3b3a5f0ed35ab1717a315de: Status 404 returned error can't find the container with id 239c653cefcebd880c0ec32f20cce418280a8611f3b3a5f0ed35ab1717a315de Oct 30 21:49:28.531388 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:28.531357 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a7ec6e_13fa_4893_a179_529f13fb2172.slice/crio-5b7492cb0c24700e79719b3ff607e612fb7f8b98a8ac52f0f09fe7715d19c97e WatchSource:0}: Error finding container 5b7492cb0c24700e79719b3ff607e612fb7f8b98a8ac52f0f09fe7715d19c97e: Status 404 returned error can't find the container with id 5b7492cb0c24700e79719b3ff607e612fb7f8b98a8ac52f0f09fe7715d19c97e Oct 30 21:49:28.531825 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:28.531786 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bdaae95a41fc764b37958d5699dbc6f.slice/crio-7141d4a04393caf7ededd58a5b768f9fdaeec111305133aaf1ee854c4cc89ee3 WatchSource:0}: Error finding container 7141d4a04393caf7ededd58a5b768f9fdaeec111305133aaf1ee854c4cc89ee3: Status 404 returned error can't find the container with id 7141d4a04393caf7ededd58a5b768f9fdaeec111305133aaf1ee854c4cc89ee3 Oct 30 21:49:28.532407 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:28.532377 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff2bf62cbfb1edc876c4b4c7d0808622.slice/crio-e13d803c6a7f26891ce23e9b9f5eb7c1ec83254184e57ffc124fc1f0b4198939 WatchSource:0}: Error finding container e13d803c6a7f26891ce23e9b9f5eb7c1ec83254184e57ffc124fc1f0b4198939: Status 404 returned error can't find the container with id e13d803c6a7f26891ce23e9b9f5eb7c1ec83254184e57ffc124fc1f0b4198939 Oct 30 21:49:28.537249 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.537226 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:49:28.543690 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.543671 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:49:28.548711 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.548689 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-b7k9p" Oct 30 21:49:28.552486 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:28.552465 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4105fd0_16f9_492e_ae10_1f7458524eeb.slice/crio-75914d1d2d2adc77ca1b238530dda483062212fea9c1fbd92ed4a5e373b89184 WatchSource:0}: Error finding container 75914d1d2d2adc77ca1b238530dda483062212fea9c1fbd92ed4a5e373b89184: Status 404 returned error can't find the container with id 75914d1d2d2adc77ca1b238530dda483062212fea9c1fbd92ed4a5e373b89184 Oct 30 21:49:28.556992 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:28.556964 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2226976_1041_4798_b567_6fb6590c0f6f.slice/crio-579477397ab60402a9ba83f3aa408531328e8e2901dd2d33996f716274408586 WatchSource:0}: Error finding container 579477397ab60402a9ba83f3aa408531328e8e2901dd2d33996f716274408586: Status 404 returned error can't find the container with id 579477397ab60402a9ba83f3aa408531328e8e2901dd2d33996f716274408586 Oct 30 21:49:28.566568 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.566553 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:49:28.575732 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:28.575706 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5d7f625_40b0_46b6_8489_fdd1daa0068e.slice/crio-3748a0779e5a23d6e1222a0447bf6f1fb23cfc4523a9104e0262381b37c677d0 WatchSource:0}: Error finding container 3748a0779e5a23d6e1222a0447bf6f1fb23cfc4523a9104e0262381b37c677d0: Status 404 returned error can't find the container with id 3748a0779e5a23d6e1222a0447bf6f1fb23cfc4523a9104e0262381b37c677d0 Oct 30 21:49:28.584992 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.584977 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:28.593844 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:28.593823 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fd1e2d6_4877_4459_82f9_95b31d098229.slice/crio-8a2e36d351ddb4fe71b03446aa206d5cfb1a975dda8e76eef321bd198f35c791 WatchSource:0}: Error finding container 8a2e36d351ddb4fe71b03446aa206d5cfb1a975dda8e76eef321bd198f35c791: Status 404 returned error can't find the container with id 8a2e36d351ddb4fe71b03446aa206d5cfb1a975dda8e76eef321bd198f35c791 Oct 30 21:49:28.727626 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:28.727602 2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:49:29.996230 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:29.996185 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a9baf0-064d-4544-94b4-a0dd090dbd04-metrics-certs\") pod \"network-metrics-daemon-h4v24\" (UID: \"83a9baf0-064d-4544-94b4-a0dd090dbd04\") " pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:49:29.996350 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:29.996249 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4630b412-da87-49f7-bbbb-ba3dc820ddd1-original-pull-secret\") pod \"global-pull-secret-syncer-xbwzg\" (UID: \"4630b412-da87-49f7-bbbb-ba3dc820ddd1\") " pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:49:29.996350 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:29.996344 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:49:29.996453 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:29.996344 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:49:29.996453 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:29.996404 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4630b412-da87-49f7-bbbb-ba3dc820ddd1-original-pull-secret podName:4630b412-da87-49f7-bbbb-ba3dc820ddd1 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:30.996387541 +0000 UTC m=+3.271756815 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4630b412-da87-49f7-bbbb-ba3dc820ddd1-original-pull-secret") pod "global-pull-secret-syncer-xbwzg" (UID: "4630b412-da87-49f7-bbbb-ba3dc820ddd1") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:49:29.996453 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:29.996421 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83a9baf0-064d-4544-94b4-a0dd090dbd04-metrics-certs podName:83a9baf0-064d-4544-94b4-a0dd090dbd04 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:30.996412574 +0000 UTC m=+3.271781845 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/83a9baf0-064d-4544-94b4-a0dd090dbd04-metrics-certs") pod "network-metrics-daemon-h4v24" (UID: "83a9baf0-064d-4544-94b4-a0dd090dbd04") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:49:30.096654 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:30.096629 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj8hr\" (UniqueName: \"kubernetes.io/projected/2e5352ac-8295-494d-950a-16c09b09272e-kube-api-access-dj8hr\") pod \"network-check-target-mg6qm\" (UID: \"2e5352ac-8295-494d-950a-16c09b09272e\") " pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:49:30.097236 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:30.096786 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:49:30.097236 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:30.096806 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:49:30.097236 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:30.096815 2576 projected.go:196] Error preparing data for projected volume kube-api-access-dj8hr for pod openshift-network-diagnostics/network-check-target-mg6qm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:49:30.097236 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:30.096866 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e5352ac-8295-494d-950a-16c09b09272e-kube-api-access-dj8hr podName:2e5352ac-8295-494d-950a-16c09b09272e nodeName:}" failed. No retries permitted until 2026-10-30 21:49:31.096852553 +0000 UTC m=+3.372221829 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dj8hr" (UniqueName: "kubernetes.io/projected/2e5352ac-8295-494d-950a-16c09b09272e-kube-api-access-dj8hr") pod "network-check-target-mg6qm" (UID: "2e5352ac-8295-494d-950a-16c09b09272e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:49:30.136796 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:30.136767 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:49:30.211155 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:30.211060 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-west-1.compute.internal" event={"ID":"ff2bf62cbfb1edc876c4b4c7d0808622","Type":"ContainerStarted","Data":"e13d803c6a7f26891ce23e9b9f5eb7c1ec83254184e57ffc124fc1f0b4198939"} Oct 30 21:49:30.212158 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:30.212131 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal" event={"ID":"8bdaae95a41fc764b37958d5699dbc6f","Type":"ContainerStarted","Data":"7141d4a04393caf7ededd58a5b768f9fdaeec111305133aaf1ee854c4cc89ee3"} Oct 30 21:49:30.213150 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:30.213124 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4dqsc" event={"ID":"9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8","Type":"ContainerStarted","Data":"239c653cefcebd880c0ec32f20cce418280a8611f3b3a5f0ed35ab1717a315de"} Oct 30 21:49:30.214168 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:30.214151 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6th89" event={"ID":"7fd1e2d6-4877-4459-82f9-95b31d098229","Type":"ContainerStarted","Data":"8a2e36d351ddb4fe71b03446aa206d5cfb1a975dda8e76eef321bd198f35c791"} Oct 30 21:49:30.215161 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:30.215143 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" event={"ID":"f5d7f625-40b0-46b6-8489-fdd1daa0068e","Type":"ContainerStarted","Data":"3748a0779e5a23d6e1222a0447bf6f1fb23cfc4523a9104e0262381b37c677d0"} Oct 30 21:49:30.216187 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:30.216169 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-b7k9p" event={"ID":"d2226976-1041-4798-b567-6fb6590c0f6f","Type":"ContainerStarted","Data":"579477397ab60402a9ba83f3aa408531328e8e2901dd2d33996f716274408586"} Oct 30 21:49:30.217195 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:30.217175 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d4fs7" event={"ID":"b4105fd0-16f9-492e-ae10-1f7458524eeb","Type":"ContainerStarted","Data":"75914d1d2d2adc77ca1b238530dda483062212fea9c1fbd92ed4a5e373b89184"} Oct 30 21:49:30.218274 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:30.218253 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hmx87" event={"ID":"18a7ec6e-13fa-4893-a179-529f13fb2172","Type":"ContainerStarted","Data":"5b7492cb0c24700e79719b3ff607e612fb7f8b98a8ac52f0f09fe7715d19c97e"} Oct 30 21:49:30.436176 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:30.436152 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd962c24e_5855_453e_97d5_78bad4a19fbc.slice/crio-30373c3a0f86eaf3542a7147226d68cbe044e9b164cf997df7fb521805e49090 WatchSource:0}: Error finding container 30373c3a0f86eaf3542a7147226d68cbe044e9b164cf997df7fb521805e49090: Status 404 returned error can't find the container with id 30373c3a0f86eaf3542a7147226d68cbe044e9b164cf997df7fb521805e49090 Oct 30 21:49:30.441371 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:30.441332 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:44:29 +0000 UTC" deadline="2028-07-29 01:28:51.73923005 +0000 UTC" Oct 30 21:49:30.441427 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:30.441372 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15291h39m21.297860793s" Oct 30 21:49:30.470709 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:30.470686 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7b50ca6_0f9c_40f8_a09d_004b055105b8.slice/crio-414bb35e7badab19b1be107dc7b2cf6d0e1164f956de288edb7d68816b26fed9 WatchSource:0}: Error finding container 414bb35e7badab19b1be107dc7b2cf6d0e1164f956de288edb7d68816b26fed9: Status 404 returned error can't find the container with id 414bb35e7badab19b1be107dc7b2cf6d0e1164f956de288edb7d68816b26fed9 Oct 30 21:49:30.524081 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:49:30.524057 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod579c3ec8_fe12_472f_8ab0_247bfa8ed8b7.slice/crio-4a01871d6a7408e50912a0bb509bf6dfe44039eecf78fd92976280b187301f36 WatchSource:0}: Error finding container 4a01871d6a7408e50912a0bb509bf6dfe44039eecf78fd92976280b187301f36: Status 404 returned error can't find the container with id 4a01871d6a7408e50912a0bb509bf6dfe44039eecf78fd92976280b187301f36 Oct 30 21:49:30.535556 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:30.535535 2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:49:31.010894 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:31.010861 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4630b412-da87-49f7-bbbb-ba3dc820ddd1-original-pull-secret\") pod \"global-pull-secret-syncer-xbwzg\" (UID: \"4630b412-da87-49f7-bbbb-ba3dc820ddd1\") " pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:49:31.011074 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:31.010949 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a9baf0-064d-4544-94b4-a0dd090dbd04-metrics-certs\") pod \"network-metrics-daemon-h4v24\" (UID: \"83a9baf0-064d-4544-94b4-a0dd090dbd04\") " pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:49:31.011139 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:31.011082 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:49:31.011193 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:31.011140 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83a9baf0-064d-4544-94b4-a0dd090dbd04-metrics-certs podName:83a9baf0-064d-4544-94b4-a0dd090dbd04 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:33.011122403 +0000 UTC m=+5.286491677 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/83a9baf0-064d-4544-94b4-a0dd090dbd04-metrics-certs") pod "network-metrics-daemon-h4v24" (UID: "83a9baf0-064d-4544-94b4-a0dd090dbd04") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:49:31.011294 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:31.011235 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:49:31.011294 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:31.011274 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4630b412-da87-49f7-bbbb-ba3dc820ddd1-original-pull-secret podName:4630b412-da87-49f7-bbbb-ba3dc820ddd1 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:33.0112616 +0000 UTC m=+5.286630877 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4630b412-da87-49f7-bbbb-ba3dc820ddd1-original-pull-secret") pod "global-pull-secret-syncer-xbwzg" (UID: "4630b412-da87-49f7-bbbb-ba3dc820ddd1") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:49:31.111396 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:31.111358 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj8hr\" (UniqueName: \"kubernetes.io/projected/2e5352ac-8295-494d-950a-16c09b09272e-kube-api-access-dj8hr\") pod \"network-check-target-mg6qm\" (UID: \"2e5352ac-8295-494d-950a-16c09b09272e\") " pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:49:31.111949 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:31.111556 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:49:31.111949 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:31.111574 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:49:31.111949 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:31.111586 2576 projected.go:196] Error preparing data for projected volume kube-api-access-dj8hr for pod openshift-network-diagnostics/network-check-target-mg6qm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:49:31.111949 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:31.111643 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e5352ac-8295-494d-950a-16c09b09272e-kube-api-access-dj8hr podName:2e5352ac-8295-494d-950a-16c09b09272e nodeName:}" failed. No retries permitted until 2026-10-30 21:49:33.111623905 +0000 UTC m=+5.386993185 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dj8hr" (UniqueName: "kubernetes.io/projected/2e5352ac-8295-494d-950a-16c09b09272e-kube-api-access-dj8hr") pod "network-check-target-mg6qm" (UID: "2e5352ac-8295-494d-950a-16c09b09272e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:49:31.207787 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:31.207758 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:49:31.207958 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:31.207882 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4v24" podUID="83a9baf0-064d-4544-94b4-a0dd090dbd04" Oct 30 21:49:31.208022 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:31.207962 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:49:31.208075 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:31.208034 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg6qm" podUID="2e5352ac-8295-494d-950a-16c09b09272e" Oct 30 21:49:31.208136 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:31.208095 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:49:31.208187 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:31.208166 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbwzg" podUID="4630b412-da87-49f7-bbbb-ba3dc820ddd1" Oct 30 21:49:31.229888 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:31.229854 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8qxjv" event={"ID":"579c3ec8-fe12-472f-8ab0-247bfa8ed8b7","Type":"ContainerStarted","Data":"4a01871d6a7408e50912a0bb509bf6dfe44039eecf78fd92976280b187301f36"} Oct 30 21:49:31.233776 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:31.233742 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-q8gsv" event={"ID":"e7b50ca6-0f9c-40f8-a09d-004b055105b8","Type":"ContainerStarted","Data":"414bb35e7badab19b1be107dc7b2cf6d0e1164f956de288edb7d68816b26fed9"} Oct 30 21:49:31.246105 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:31.246078 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" event={"ID":"d962c24e-5855-453e-97d5-78bad4a19fbc","Type":"ContainerStarted","Data":"30373c3a0f86eaf3542a7147226d68cbe044e9b164cf997df7fb521805e49090"} Oct 30 21:49:31.441887 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:31.441845 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:44:29 +0000 UTC" deadline="2028-03-29 19:36:58.105874429 +0000 UTC" Oct 30 21:49:31.441887 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:31.441886 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12381h47m26.663993092s" Oct 30 21:49:33.027327 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:33.027290 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a9baf0-064d-4544-94b4-a0dd090dbd04-metrics-certs\") pod \"network-metrics-daemon-h4v24\" (UID: \"83a9baf0-064d-4544-94b4-a0dd090dbd04\") " pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:49:33.027753 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:33.027357 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4630b412-da87-49f7-bbbb-ba3dc820ddd1-original-pull-secret\") pod \"global-pull-secret-syncer-xbwzg\" (UID: \"4630b412-da87-49f7-bbbb-ba3dc820ddd1\") " pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:49:33.027753 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:33.027480 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:49:33.027753 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:33.027549 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83a9baf0-064d-4544-94b4-a0dd090dbd04-metrics-certs podName:83a9baf0-064d-4544-94b4-a0dd090dbd04 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:37.027530986 +0000 UTC m=+9.302900263 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/83a9baf0-064d-4544-94b4-a0dd090dbd04-metrics-certs") pod "network-metrics-daemon-h4v24" (UID: "83a9baf0-064d-4544-94b4-a0dd090dbd04") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:49:33.027753 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:33.027573 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:49:33.027753 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:33.027635 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4630b412-da87-49f7-bbbb-ba3dc820ddd1-original-pull-secret podName:4630b412-da87-49f7-bbbb-ba3dc820ddd1 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:37.027620522 +0000 UTC m=+9.302989799 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4630b412-da87-49f7-bbbb-ba3dc820ddd1-original-pull-secret") pod "global-pull-secret-syncer-xbwzg" (UID: "4630b412-da87-49f7-bbbb-ba3dc820ddd1") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:49:33.128796 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:33.128719 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj8hr\" (UniqueName: \"kubernetes.io/projected/2e5352ac-8295-494d-950a-16c09b09272e-kube-api-access-dj8hr\") pod \"network-check-target-mg6qm\" (UID: \"2e5352ac-8295-494d-950a-16c09b09272e\") " pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:49:33.128951 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:33.128899 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:49:33.128951 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:33.128919 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:49:33.128951 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:33.128930 2576 projected.go:196] Error preparing data for projected volume kube-api-access-dj8hr for pod openshift-network-diagnostics/network-check-target-mg6qm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:49:33.129104 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:33.128984 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e5352ac-8295-494d-950a-16c09b09272e-kube-api-access-dj8hr podName:2e5352ac-8295-494d-950a-16c09b09272e nodeName:}" failed. No retries permitted until 2026-10-30 21:49:37.128967222 +0000 UTC m=+9.404336499 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dj8hr" (UniqueName: "kubernetes.io/projected/2e5352ac-8295-494d-950a-16c09b09272e-kube-api-access-dj8hr") pod "network-check-target-mg6qm" (UID: "2e5352ac-8295-494d-950a-16c09b09272e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:49:33.207729 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:33.207692 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:49:33.207899 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:33.207836 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg6qm" podUID="2e5352ac-8295-494d-950a-16c09b09272e" Oct 30 21:49:33.207956 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:33.207934 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:49:33.208024 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:33.208006 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbwzg" podUID="4630b412-da87-49f7-bbbb-ba3dc820ddd1" Oct 30 21:49:33.208081 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:33.208058 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:49:33.208136 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:33.208108 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4v24" podUID="83a9baf0-064d-4544-94b4-a0dd090dbd04" Oct 30 21:49:34.272460 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:34.271385 2576 generic.go:356] "Generic (PLEG): container finished" podID="b4105fd0-16f9-492e-ae10-1f7458524eeb" containerID="171726a64cfc861e06ebb78fab3019699d041756721df0a0d0b517a71041c046" exitCode=0 Oct 30 21:49:34.272460 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:34.271621 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d4fs7" event={"ID":"b4105fd0-16f9-492e-ae10-1f7458524eeb","Type":"ContainerDied","Data":"171726a64cfc861e06ebb78fab3019699d041756721df0a0d0b517a71041c046"} Oct 30 21:49:35.207984 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:35.207945 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:49:35.208161 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:35.208097 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4v24" podUID="83a9baf0-064d-4544-94b4-a0dd090dbd04" Oct 30 21:49:35.208298 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:35.208170 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:49:35.208369 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:35.208295 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg6qm" podUID="2e5352ac-8295-494d-950a-16c09b09272e" Oct 30 21:49:35.208369 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:35.208350 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:49:35.208472 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:35.208426 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbwzg" podUID="4630b412-da87-49f7-bbbb-ba3dc820ddd1" Oct 30 21:49:35.280665 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:35.280628 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hmx87" event={"ID":"18a7ec6e-13fa-4893-a179-529f13fb2172","Type":"ContainerStarted","Data":"aefc6dd0bb5949d412bd687caf7495c3358d26326e3b8b82895cfcc0d29028a4"} Oct 30 21:49:35.285560 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:35.285502 2576 generic.go:356] "Generic (PLEG): container finished" podID="8bdaae95a41fc764b37958d5699dbc6f" containerID="e1ec7bd724076009f96580dd30dd373e2022054d34721e211874664b85440e9b" exitCode=0 Oct 30 21:49:35.285700 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:35.285595 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal" event={"ID":"8bdaae95a41fc764b37958d5699dbc6f","Type":"ContainerDied","Data":"e1ec7bd724076009f96580dd30dd373e2022054d34721e211874664b85440e9b"} Oct 30 21:49:35.291479 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:35.291451 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-west-1.compute.internal" event={"ID":"ff2bf62cbfb1edc876c4b4c7d0808622","Type":"ContainerStarted","Data":"858cbf4cc2fc8d7295ab25c152c6d8eba107b132ae05f8c7f554d680f37e646b"} Oct 30 21:49:35.402525 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:35.402438 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hmx87" podStartSLOduration=2.587075593 podStartE2EDuration="7.402419003s" totalImagesPullingTime="4.81534341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:35.398575316 +0000 UTC m=+7.673944601" watchObservedRunningTime="2026-10-30 21:49:35.402419003 +0000 UTC m=+7.677788277" Oct 30 21:49:35.637274 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:35.637053 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-west-1.compute.internal" podStartSLOduration=7.637032725 podStartE2EDuration="7.637032725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:35.520729232 +0000 UTC m=+7.796098516" watchObservedRunningTime="2026-10-30 21:49:35.637032725 +0000 UTC m=+7.912402008" Oct 30 21:49:36.296849 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:36.296809 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal" event={"ID":"8bdaae95a41fc764b37958d5699dbc6f","Type":"ContainerStarted","Data":"42b975907e04084f274c85de36330639352f755c619906e4f37740af4b3ca3fb"} Oct 30 21:49:36.311239 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:36.309977 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4dqsc" event={"ID":"9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8","Type":"ContainerStarted","Data":"71a4012f311831bb9857c87236679d61df02a3e0753b23965cb3e0e7c0801a72"} Oct 30 21:49:36.414209 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:36.414157 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal" podStartSLOduration=8.414141569 podStartE2EDuration="8.414141569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:36.413397394 +0000 UTC m=+8.688766686" watchObservedRunningTime="2026-10-30 21:49:36.414141569 +0000 UTC m=+8.689510856" Oct 30 21:49:37.066493 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:37.066451 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a9baf0-064d-4544-94b4-a0dd090dbd04-metrics-certs\") pod \"network-metrics-daemon-h4v24\" (UID: \"83a9baf0-064d-4544-94b4-a0dd090dbd04\") " pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:49:37.066663 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:37.066510 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4630b412-da87-49f7-bbbb-ba3dc820ddd1-original-pull-secret\") pod \"global-pull-secret-syncer-xbwzg\" (UID: \"4630b412-da87-49f7-bbbb-ba3dc820ddd1\") " pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:49:37.066663 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:37.066642 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:49:37.066741 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:37.066707 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4630b412-da87-49f7-bbbb-ba3dc820ddd1-original-pull-secret podName:4630b412-da87-49f7-bbbb-ba3dc820ddd1 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:45.066688689 +0000 UTC m=+17.342057951 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4630b412-da87-49f7-bbbb-ba3dc820ddd1-original-pull-secret") pod "global-pull-secret-syncer-xbwzg" (UID: "4630b412-da87-49f7-bbbb-ba3dc820ddd1") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:49:37.066797 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:37.066777 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:49:37.066837 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:37.066818 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83a9baf0-064d-4544-94b4-a0dd090dbd04-metrics-certs podName:83a9baf0-064d-4544-94b4-a0dd090dbd04 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:45.066806668 +0000 UTC m=+17.342175943 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/83a9baf0-064d-4544-94b4-a0dd090dbd04-metrics-certs") pod "network-metrics-daemon-h4v24" (UID: "83a9baf0-064d-4544-94b4-a0dd090dbd04") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:49:37.167818 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:37.167737 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj8hr\" (UniqueName: \"kubernetes.io/projected/2e5352ac-8295-494d-950a-16c09b09272e-kube-api-access-dj8hr\") pod \"network-check-target-mg6qm\" (UID: \"2e5352ac-8295-494d-950a-16c09b09272e\") " pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:49:37.167969 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:37.167938 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:49:37.167969 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:37.167961 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:49:37.168082 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:37.167975 2576 projected.go:196] Error preparing data for projected volume kube-api-access-dj8hr for pod openshift-network-diagnostics/network-check-target-mg6qm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:49:37.168082 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:37.168025 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e5352ac-8295-494d-950a-16c09b09272e-kube-api-access-dj8hr podName:2e5352ac-8295-494d-950a-16c09b09272e nodeName:}" failed. No retries permitted until 2026-10-30 21:49:45.168012558 +0000 UTC m=+17.443381817 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dj8hr" (UniqueName: "kubernetes.io/projected/2e5352ac-8295-494d-950a-16c09b09272e-kube-api-access-dj8hr") pod "network-check-target-mg6qm" (UID: "2e5352ac-8295-494d-950a-16c09b09272e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:49:37.208344 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:37.208307 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:49:37.208344 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:37.208326 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:49:37.208578 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:37.208303 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:49:37.208578 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:37.208440 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbwzg" podUID="4630b412-da87-49f7-bbbb-ba3dc820ddd1" Oct 30 21:49:37.208578 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:37.208544 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4v24" podUID="83a9baf0-064d-4544-94b4-a0dd090dbd04" Oct 30 21:49:37.208730 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:37.208619 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg6qm" podUID="2e5352ac-8295-494d-950a-16c09b09272e" Oct 30 21:49:37.314519 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:37.314487 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" event={"ID":"f5d7f625-40b0-46b6-8489-fdd1daa0068e","Type":"ContainerStarted","Data":"24c312aac4e272430408f9450279fba8dc1d81a78f27587aabbb0b3bcc2d330d"} Oct 30 21:49:38.320787 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:38.320742 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-b7k9p" event={"ID":"d2226976-1041-4798-b567-6fb6590c0f6f","Type":"ContainerStarted","Data":"12176bea6d9d8a75a629abe8ebc32e74405e4d272b433e9f14a984bdd5dae27e"} Oct 30 21:49:38.442813 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:38.442752 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4dqsc" podStartSLOduration=4.306802483 podStartE2EDuration="10.442732205s" totalImagesPullingTime="6.135929722s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:36.52948571 +0000 UTC m=+8.804854994" watchObservedRunningTime="2026-10-30 21:49:38.442732205 +0000 UTC m=+10.718101489" Oct 30 21:49:38.443060 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:38.443023 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-b7k9p" podStartSLOduration=2.113076643 podStartE2EDuration="10.443013652s" totalImagesPullingTime="8.329937009s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:38.442049124 +0000 UTC m=+10.717418406" watchObservedRunningTime="2026-10-30 21:49:38.443013652 +0000 UTC m=+10.718382972" Oct 30 21:49:38.552276 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:38.551911 2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:49:38.649199 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:38.648883 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hmx87" Oct 30 21:49:38.650206 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:38.650024 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hmx87" Oct 30 21:49:39.154567 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:39.154239 2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:49:38.551929495Z","UUID":"a4a81f27-b99d-4fe3-921d-c543cd1d496f","Handler":null,"Name":"","Endpoint":""} Oct 30 21:49:39.157075 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:39.156974 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:49:39.157075 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:39.157008 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:49:39.208272 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:39.208173 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:49:39.208420 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:39.208349 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg6qm" podUID="2e5352ac-8295-494d-950a-16c09b09272e" Oct 30 21:49:39.208467 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:39.208444 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:49:39.208563 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:39.208530 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbwzg" podUID="4630b412-da87-49f7-bbbb-ba3dc820ddd1" Oct 30 21:49:39.208610 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:39.208602 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:49:39.208699 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:39.208682 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4v24" podUID="83a9baf0-064d-4544-94b4-a0dd090dbd04" Oct 30 21:49:39.324917 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:39.324881 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" event={"ID":"f5d7f625-40b0-46b6-8489-fdd1daa0068e","Type":"ContainerStarted","Data":"b35e72e0211b6bf85c45e8a85011759941912b898f9f04cd3ed07593efd9bace"} Oct 30 21:49:39.327206 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:39.327178 2576 generic.go:356] "Generic (PLEG): container finished" podID="b4105fd0-16f9-492e-ae10-1f7458524eeb" containerID="74ba2e46e24ff5a83c78a6b30e6f62b704c65774c5811a80debd3cc86b37f5e6" exitCode=0 Oct 30 21:49:39.327344 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:39.327236 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d4fs7" event={"ID":"b4105fd0-16f9-492e-ae10-1f7458524eeb","Type":"ContainerDied","Data":"74ba2e46e24ff5a83c78a6b30e6f62b704c65774c5811a80debd3cc86b37f5e6"} Oct 30 21:49:39.328977 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:39.328949 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8qxjv" event={"ID":"579c3ec8-fe12-472f-8ab0-247bfa8ed8b7","Type":"ContainerStarted","Data":"3c4c284a0175f47c71b211f585fbfee8e6d8dc504787955d14b34ff68c292ce7"} Oct 30 21:49:39.573723 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:39.573626 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-8qxjv" podStartSLOduration=3.14935757 podStartE2EDuration="11.573609054s" totalImagesPullingTime="8.424251484s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:39.572841351 +0000 UTC m=+11.848210635" watchObservedRunningTime="2026-10-30 21:49:39.573609054 +0000 UTC m=+11.848978334" Oct 30 21:49:40.334163 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:40.334135 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" event={"ID":"f5d7f625-40b0-46b6-8489-fdd1daa0068e","Type":"ContainerStarted","Data":"2f72c6a193acb4e514f0395976f3b25b7859af6699851057c70f0a87c06caf7d"} Oct 30 21:49:40.336336 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:40.336305 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-q8gsv" event={"ID":"e7b50ca6-0f9c-40f8-a09d-004b055105b8","Type":"ContainerStarted","Data":"ff7d87c9b9f7970a1c8ecb1c390eca57e352e4e450afb144c4291848e0d5dd87"} Oct 30 21:49:40.577188 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:40.577084 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" podStartSLOduration=2.663421154 podStartE2EDuration="12.577064334s" totalImagesPullingTime="9.91364318s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:40.461424052 +0000 UTC m=+12.736793333" watchObservedRunningTime="2026-10-30 21:49:40.577064334 +0000 UTC m=+12.852433655" Oct 30 21:49:40.577565 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:40.577525 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-q8gsv" podStartSLOduration=4.360301122 podStartE2EDuration="12.577516164s" totalImagesPullingTime="8.217215042s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:40.576906694 +0000 UTC m=+12.852275974" watchObservedRunningTime="2026-10-30 21:49:40.577516164 +0000 UTC m=+12.852885445" Oct 30 21:49:41.207919 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:41.207890 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:49:41.208039 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:41.208017 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbwzg" podUID="4630b412-da87-49f7-bbbb-ba3dc820ddd1" Oct 30 21:49:41.208098 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:41.208089 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:49:41.208187 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:41.208156 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg6qm" podUID="2e5352ac-8295-494d-950a-16c09b09272e" Oct 30 21:49:41.208262 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:41.208238 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:49:41.208354 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:41.208323 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4v24" podUID="83a9baf0-064d-4544-94b4-a0dd090dbd04" Oct 30 21:49:41.339862 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:41.339825 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" event={"ID":"d962c24e-5855-453e-97d5-78bad4a19fbc","Type":"ContainerStarted","Data":"713e2c5c518e28230ed4610e917ad629f6f9155cbea55d9374ba505f3fe8810e"} Oct 30 21:49:41.342699 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:41.342672 2576 generic.go:356] "Generic (PLEG): container finished" podID="b4105fd0-16f9-492e-ae10-1f7458524eeb" containerID="eac26567ad4fb6633d1a3b2cca0d87f6444761e1778a963c5a3cea7c3ac12024" exitCode=0 Oct 30 21:49:41.342976 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:41.342953 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d4fs7" event={"ID":"b4105fd0-16f9-492e-ae10-1f7458524eeb","Type":"ContainerDied","Data":"eac26567ad4fb6633d1a3b2cca0d87f6444761e1778a963c5a3cea7c3ac12024"} Oct 30 21:49:41.465715 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:41.465653 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" podStartSLOduration=2.766594363 podStartE2EDuration="13.465634917s" totalImagesPullingTime="10.699040554s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:41.465071641 +0000 UTC m=+13.740440922" watchObservedRunningTime="2026-10-30 21:49:41.465634917 +0000 UTC m=+13.741004199" Oct 30 21:49:43.208353 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:43.208108 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:49:43.208936 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:43.208135 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:49:43.208936 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:43.208458 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg6qm" podUID="2e5352ac-8295-494d-950a-16c09b09272e" Oct 30 21:49:43.208936 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:43.208135 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:49:43.208936 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:43.208551 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbwzg" podUID="4630b412-da87-49f7-bbbb-ba3dc820ddd1" Oct 30 21:49:43.208936 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:43.208616 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4v24" podUID="83a9baf0-064d-4544-94b4-a0dd090dbd04" Oct 30 21:49:43.350953 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:43.350914 2576 generic.go:356] "Generic (PLEG): container finished" podID="b4105fd0-16f9-492e-ae10-1f7458524eeb" containerID="e1a04b5ad21206e5b7e9319e80537b4f5e37fea736731c8a1548397883cd83d8" exitCode=0 Oct 30 21:49:43.351119 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:43.350977 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d4fs7" event={"ID":"b4105fd0-16f9-492e-ae10-1f7458524eeb","Type":"ContainerDied","Data":"e1a04b5ad21206e5b7e9319e80537b4f5e37fea736731c8a1548397883cd83d8"} Oct 30 21:49:45.130899 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:45.130855 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a9baf0-064d-4544-94b4-a0dd090dbd04-metrics-certs\") pod \"network-metrics-daemon-h4v24\" (UID: \"83a9baf0-064d-4544-94b4-a0dd090dbd04\") " pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:49:45.131342 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:45.130911 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4630b412-da87-49f7-bbbb-ba3dc820ddd1-original-pull-secret\") pod \"global-pull-secret-syncer-xbwzg\" (UID: \"4630b412-da87-49f7-bbbb-ba3dc820ddd1\") " pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:49:45.131342 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:45.131019 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:49:45.131342 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:45.131083 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4630b412-da87-49f7-bbbb-ba3dc820ddd1-original-pull-secret podName:4630b412-da87-49f7-bbbb-ba3dc820ddd1 nodeName:}" failed. No retries permitted until 2026-10-30 21:50:01.131070693 +0000 UTC m=+33.406439966 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4630b412-da87-49f7-bbbb-ba3dc820ddd1-original-pull-secret") pod "global-pull-secret-syncer-xbwzg" (UID: "4630b412-da87-49f7-bbbb-ba3dc820ddd1") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:49:45.131512 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:45.131494 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:49:45.131568 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:45.131547 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83a9baf0-064d-4544-94b4-a0dd090dbd04-metrics-certs podName:83a9baf0-064d-4544-94b4-a0dd090dbd04 nodeName:}" failed. No retries permitted until 2026-10-30 21:50:01.13153092 +0000 UTC m=+33.406900179 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/83a9baf0-064d-4544-94b4-a0dd090dbd04-metrics-certs") pod "network-metrics-daemon-h4v24" (UID: "83a9baf0-064d-4544-94b4-a0dd090dbd04") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:49:45.208235 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:45.208191 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:49:45.208407 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:45.208241 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:49:45.208407 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:45.208241 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:49:45.208407 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:45.208360 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4v24" podUID="83a9baf0-064d-4544-94b4-a0dd090dbd04" Oct 30 21:49:45.208556 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:45.208459 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbwzg" podUID="4630b412-da87-49f7-bbbb-ba3dc820ddd1" Oct 30 21:49:45.208556 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:45.208539 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg6qm" podUID="2e5352ac-8295-494d-950a-16c09b09272e" Oct 30 21:49:45.232064 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:45.232034 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj8hr\" (UniqueName: \"kubernetes.io/projected/2e5352ac-8295-494d-950a-16c09b09272e-kube-api-access-dj8hr\") pod \"network-check-target-mg6qm\" (UID: \"2e5352ac-8295-494d-950a-16c09b09272e\") " pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:49:45.232245 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:45.232204 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:49:45.232245 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:45.232238 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:49:45.232366 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:45.232251 2576 projected.go:196] Error preparing data for projected volume kube-api-access-dj8hr for pod openshift-network-diagnostics/network-check-target-mg6qm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:49:45.232366 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:45.232292 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e5352ac-8295-494d-950a-16c09b09272e-kube-api-access-dj8hr podName:2e5352ac-8295-494d-950a-16c09b09272e nodeName:}" failed. No retries permitted until 2026-10-30 21:50:01.232280313 +0000 UTC m=+33.507649586 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dj8hr" (UniqueName: "kubernetes.io/projected/2e5352ac-8295-494d-950a-16c09b09272e-kube-api-access-dj8hr") pod "network-check-target-mg6qm" (UID: "2e5352ac-8295-494d-950a-16c09b09272e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:49:46.360548 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:46.360369 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6th89" event={"ID":"7fd1e2d6-4877-4459-82f9-95b31d098229","Type":"ContainerStarted","Data":"88953bb43c964b83d04ef381aa7d3f99ba63e466ed8d9f4f7efcda0e891ed059"} Oct 30 21:49:46.360878 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:46.360608 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6th89" event={"ID":"7fd1e2d6-4877-4459-82f9-95b31d098229","Type":"ContainerStarted","Data":"4b935cc196d4a3bc3457941693d969c96fd901d156d0d182b6a8c177efb46f52"} Oct 30 21:49:46.360878 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:46.360654 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6th89" event={"ID":"7fd1e2d6-4877-4459-82f9-95b31d098229","Type":"ContainerStarted","Data":"6adc3e3bead9fef6d7e73eec3fb23e92462546f8d2d008be478f6f1e0c6178cf"} Oct 30 21:49:46.360878 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:46.360683 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6th89" event={"ID":"7fd1e2d6-4877-4459-82f9-95b31d098229","Type":"ContainerStarted","Data":"a160ef0d96ff9236ec6776b0d54f6d5e19b985524050fd6e4bfe5c8510c8a8b3"} Oct 30 21:49:46.360878 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:46.360710 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6th89" event={"ID":"7fd1e2d6-4877-4459-82f9-95b31d098229","Type":"ContainerStarted","Data":"9e6d6adb9a83f63897226adaa956447a6d5e347bc24041fe4cc5d911fafc1385"} Oct 30 21:49:46.363607 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:46.363582 2576 generic.go:356] "Generic (PLEG): container finished" podID="b4105fd0-16f9-492e-ae10-1f7458524eeb" containerID="df3ab91aec64f7a669bf65cae46a3c31abfb020db6e17d622ed8ca3ce8771b47" exitCode=0 Oct 30 21:49:46.363711 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:46.363626 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d4fs7" event={"ID":"b4105fd0-16f9-492e-ae10-1f7458524eeb","Type":"ContainerDied","Data":"df3ab91aec64f7a669bf65cae46a3c31abfb020db6e17d622ed8ca3ce8771b47"} Oct 30 21:49:47.207790 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:47.207760 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:49:47.207790 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:47.207787 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:49:47.207979 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:47.207866 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4v24" podUID="83a9baf0-064d-4544-94b4-a0dd090dbd04" Oct 30 21:49:47.207979 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:47.207909 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:49:47.208036 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:47.207981 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg6qm" podUID="2e5352ac-8295-494d-950a-16c09b09272e" Oct 30 21:49:47.208036 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:47.207995 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbwzg" podUID="4630b412-da87-49f7-bbbb-ba3dc820ddd1" Oct 30 21:49:47.368558 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:47.368528 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6th89" event={"ID":"7fd1e2d6-4877-4459-82f9-95b31d098229","Type":"ContainerStarted","Data":"cf7ad77860289a435be640f01ffe731b39464e6dde1361a5399096fd2aed0054"} Oct 30 21:49:47.371451 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:47.371426 2576 generic.go:356] "Generic (PLEG): container finished" podID="b4105fd0-16f9-492e-ae10-1f7458524eeb" containerID="88213f3653a0b8bd50d75209b8503b1d8b3af276ad7312bee50dd1558d5ade5f" exitCode=0 Oct 30 21:49:47.371558 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:47.371457 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d4fs7" event={"ID":"b4105fd0-16f9-492e-ae10-1f7458524eeb","Type":"ContainerDied","Data":"88213f3653a0b8bd50d75209b8503b1d8b3af276ad7312bee50dd1558d5ade5f"} Oct 30 21:49:48.375874 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:48.375838 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d4fs7" event={"ID":"b4105fd0-16f9-492e-ae10-1f7458524eeb","Type":"ContainerStarted","Data":"7d7155f7246e65d436d63893fb6908977077a5e8c46e7b8eba8da411b736f27e"} Oct 30 21:49:48.514321 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:48.514281 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-d4fs7" podStartSLOduration=7.031518487 podStartE2EDuration="20.51426885s" totalImagesPullingTime="13.482750363s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:48.513741844 +0000 UTC m=+20.789111125" watchObservedRunningTime="2026-10-30 21:49:48.51426885 +0000 UTC m=+20.789638130" Oct 30 21:49:49.134767 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:49.134731 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hmx87_18a7ec6e-13fa-4893-a179-529f13fb2172/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:49.135121 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:49.135098 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8qxjv_579c3ec8-fe12-472f-8ab0-247bfa8ed8b7/dns-node-resolver/0.log" Oct 30 21:49:49.208421 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:49.208388 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:49:49.208421 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:49.208408 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:49:49.208570 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:49.208495 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:49:49.208570 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:49.208506 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4v24" podUID="83a9baf0-064d-4544-94b4-a0dd090dbd04" Oct 30 21:49:49.208642 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:49.208574 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg6qm" podUID="2e5352ac-8295-494d-950a-16c09b09272e" Oct 30 21:49:49.208642 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:49.208626 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbwzg" podUID="4630b412-da87-49f7-bbbb-ba3dc820ddd1" Oct 30 21:49:49.381276 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:49.381244 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6th89" event={"ID":"7fd1e2d6-4877-4459-82f9-95b31d098229","Type":"ContainerStarted","Data":"5359d47b8ebdb943740f259654be62ffc64e96284ed770224633ac30389853f9"} Oct 30 21:49:49.559210 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:49.559179 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hmx87_18a7ec6e-13fa-4893-a179-529f13fb2172/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:50.136075 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:50.136051 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4dqsc_9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8/node-ca/0.log" Oct 30 21:49:51.208616 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:51.208591 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:49:51.209255 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:51.208594 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:49:51.209255 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:51.208692 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4v24" podUID="83a9baf0-064d-4544-94b4-a0dd090dbd04" Oct 30 21:49:51.209255 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:51.208733 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:49:51.209255 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:51.208829 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg6qm" podUID="2e5352ac-8295-494d-950a-16c09b09272e" Oct 30 21:49:51.209255 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:51.208908 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbwzg" podUID="4630b412-da87-49f7-bbbb-ba3dc820ddd1" Oct 30 21:49:51.389290 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:51.389251 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6th89" event={"ID":"7fd1e2d6-4877-4459-82f9-95b31d098229","Type":"ContainerStarted","Data":"830dc4f4621171bb40af86a232f6064a1f5a70382a243975743f4384dae79ac4"} Oct 30 21:49:51.389555 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:51.389528 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:51.389675 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:51.389602 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:51.405150 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:51.405131 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:51.670176 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:51.670090 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6th89" podStartSLOduration=7.444588029 podStartE2EDuration="23.670075643s" totalImagesPullingTime="16.225487614s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:49:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:51.620779111 +0000 UTC m=+23.896148403" watchObservedRunningTime="2026-10-30 21:49:51.670075643 +0000 UTC m=+23.945444923" Oct 30 21:49:52.392198 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:52.392167 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:52.407100 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:52.407075 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:49:53.207962 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:53.207932 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:49:53.208163 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:53.207973 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:49:53.208163 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:53.208086 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg6qm" podUID="2e5352ac-8295-494d-950a-16c09b09272e" Oct 30 21:49:53.208163 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:53.208129 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:49:53.208353 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:53.208198 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbwzg" podUID="4630b412-da87-49f7-bbbb-ba3dc820ddd1" Oct 30 21:49:53.208353 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:53.208277 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4v24" podUID="83a9baf0-064d-4544-94b4-a0dd090dbd04" Oct 30 21:49:55.208208 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:55.208175 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:49:55.208208 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:55.208202 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:49:55.208637 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:55.208200 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:49:55.208637 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:55.208299 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4v24" podUID="83a9baf0-064d-4544-94b4-a0dd090dbd04" Oct 30 21:49:55.208637 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:55.208364 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbwzg" podUID="4630b412-da87-49f7-bbbb-ba3dc820ddd1" Oct 30 21:49:55.208637 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:55.208424 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg6qm" podUID="2e5352ac-8295-494d-950a-16c09b09272e" Oct 30 21:49:55.804899 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:55.804872 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hmx87" Oct 30 21:49:55.805049 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:55.804987 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:49:55.805549 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:55.805529 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hmx87" Oct 30 21:49:57.208698 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:57.208527 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:49:57.209060 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:57.208568 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:49:57.209060 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:57.208776 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbwzg" podUID="4630b412-da87-49f7-bbbb-ba3dc820ddd1" Oct 30 21:49:57.209060 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:57.208831 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg6qm" podUID="2e5352ac-8295-494d-950a-16c09b09272e" Oct 30 21:49:57.209060 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:57.208593 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:49:57.209060 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:57.208896 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4v24" podUID="83a9baf0-064d-4544-94b4-a0dd090dbd04" Oct 30 21:49:59.208076 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:59.208036 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:49:59.208076 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:59.208063 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:49:59.208076 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:49:59.208077 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:49:59.208634 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:59.208155 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg6qm" podUID="2e5352ac-8295-494d-950a-16c09b09272e" Oct 30 21:49:59.208634 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:59.208206 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbwzg" podUID="4630b412-da87-49f7-bbbb-ba3dc820ddd1" Oct 30 21:49:59.208634 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:49:59.208299 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4v24" podUID="83a9baf0-064d-4544-94b4-a0dd090dbd04" Oct 30 21:50:01.150526 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:01.150492 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a9baf0-064d-4544-94b4-a0dd090dbd04-metrics-certs\") pod \"network-metrics-daemon-h4v24\" (UID: \"83a9baf0-064d-4544-94b4-a0dd090dbd04\") " pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:50:01.150526 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:01.150531 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4630b412-da87-49f7-bbbb-ba3dc820ddd1-original-pull-secret\") pod \"global-pull-secret-syncer-xbwzg\" (UID: \"4630b412-da87-49f7-bbbb-ba3dc820ddd1\") " pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:50:01.150960 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:50:01.150632 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:50:01.150960 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:50:01.150640 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:50:01.150960 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:50:01.150688 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4630b412-da87-49f7-bbbb-ba3dc820ddd1-original-pull-secret podName:4630b412-da87-49f7-bbbb-ba3dc820ddd1 nodeName:}" failed. No retries permitted until 2026-10-30 21:50:33.150674555 +0000 UTC m=+65.426043826 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4630b412-da87-49f7-bbbb-ba3dc820ddd1-original-pull-secret") pod "global-pull-secret-syncer-xbwzg" (UID: "4630b412-da87-49f7-bbbb-ba3dc820ddd1") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:50:01.150960 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:50:01.150703 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83a9baf0-064d-4544-94b4-a0dd090dbd04-metrics-certs podName:83a9baf0-064d-4544-94b4-a0dd090dbd04 nodeName:}" failed. No retries permitted until 2026-10-30 21:50:33.15069588 +0000 UTC m=+65.426065140 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/83a9baf0-064d-4544-94b4-a0dd090dbd04-metrics-certs") pod "network-metrics-daemon-h4v24" (UID: "83a9baf0-064d-4544-94b4-a0dd090dbd04") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:50:01.208465 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:01.208431 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:50:01.208695 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:01.208539 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:50:01.208695 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:50:01.208546 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4v24" podUID="83a9baf0-064d-4544-94b4-a0dd090dbd04" Oct 30 21:50:01.208695 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:01.208574 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:50:01.208695 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:50:01.208661 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg6qm" podUID="2e5352ac-8295-494d-950a-16c09b09272e" Oct 30 21:50:01.208880 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:50:01.208743 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbwzg" podUID="4630b412-da87-49f7-bbbb-ba3dc820ddd1" Oct 30 21:50:01.251254 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:01.251226 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj8hr\" (UniqueName: \"kubernetes.io/projected/2e5352ac-8295-494d-950a-16c09b09272e-kube-api-access-dj8hr\") pod \"network-check-target-mg6qm\" (UID: \"2e5352ac-8295-494d-950a-16c09b09272e\") " pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:50:01.251419 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:50:01.251375 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:50:01.251419 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:50:01.251395 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:50:01.251419 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:50:01.251408 2576 projected.go:196] Error preparing data for projected volume kube-api-access-dj8hr for pod openshift-network-diagnostics/network-check-target-mg6qm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:50:01.251508 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:50:01.251455 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e5352ac-8295-494d-950a-16c09b09272e-kube-api-access-dj8hr podName:2e5352ac-8295-494d-950a-16c09b09272e nodeName:}" failed. No retries permitted until 2026-10-30 21:50:33.251442114 +0000 UTC m=+65.526811390 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dj8hr" (UniqueName: "kubernetes.io/projected/2e5352ac-8295-494d-950a-16c09b09272e-kube-api-access-dj8hr") pod "network-check-target-mg6qm" (UID: "2e5352ac-8295-494d-950a-16c09b09272e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:50:03.208550 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:03.208519 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:50:03.209033 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:03.208519 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:50:03.209033 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:50:03.208621 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg6qm" podUID="2e5352ac-8295-494d-950a-16c09b09272e" Oct 30 21:50:03.209033 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:03.208519 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:50:03.209033 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:50:03.208708 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4v24" podUID="83a9baf0-064d-4544-94b4-a0dd090dbd04" Oct 30 21:50:03.209033 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:50:03.208759 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbwzg" podUID="4630b412-da87-49f7-bbbb-ba3dc820ddd1" Oct 30 21:50:05.208162 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:05.208128 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:50:05.208652 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:05.208238 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:50:05.208652 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:50:05.208350 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbwzg" podUID="4630b412-da87-49f7-bbbb-ba3dc820ddd1" Oct 30 21:50:05.208652 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:50:05.208238 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg6qm" podUID="2e5352ac-8295-494d-950a-16c09b09272e" Oct 30 21:50:05.208652 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:05.208398 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:50:05.208652 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:50:05.208485 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4v24" podUID="83a9baf0-064d-4544-94b4-a0dd090dbd04" Oct 30 21:50:07.208267 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:07.208238 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:50:07.208267 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:07.208262 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:50:07.208684 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:07.208237 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:50:07.208684 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:50:07.208345 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbwzg" podUID="4630b412-da87-49f7-bbbb-ba3dc820ddd1" Oct 30 21:50:07.208684 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:50:07.208412 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4v24" podUID="83a9baf0-064d-4544-94b4-a0dd090dbd04" Oct 30 21:50:07.208684 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:50:07.208464 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg6qm" podUID="2e5352ac-8295-494d-950a-16c09b09272e" Oct 30 21:50:09.208351 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:09.208310 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:50:09.208351 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:09.208351 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:50:09.208731 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:09.208425 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:50:09.208731 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:50:09.208444 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbwzg" podUID="4630b412-da87-49f7-bbbb-ba3dc820ddd1" Oct 30 21:50:09.208731 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:50:09.208495 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg6qm" podUID="2e5352ac-8295-494d-950a-16c09b09272e" Oct 30 21:50:09.208731 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:50:09.208561 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4v24" podUID="83a9baf0-064d-4544-94b4-a0dd090dbd04" Oct 30 21:50:11.207757 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:11.207722 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:50:11.207757 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:11.207750 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:50:11.208170 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:50:11.207818 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg6qm" podUID="2e5352ac-8295-494d-950a-16c09b09272e" Oct 30 21:50:11.208170 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:11.207883 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:50:11.208170 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:50:11.207960 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbwzg" podUID="4630b412-da87-49f7-bbbb-ba3dc820ddd1" Oct 30 21:50:11.208170 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:50:11.208009 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4v24" podUID="83a9baf0-064d-4544-94b4-a0dd090dbd04" Oct 30 21:50:12.383383 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:12.383348 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xbwzg"] Oct 30 21:50:12.383834 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:12.383482 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:50:12.383834 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:50:12.383603 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbwzg" podUID="4630b412-da87-49f7-bbbb-ba3dc820ddd1" Oct 30 21:50:12.386437 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:12.386411 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mg6qm"] Oct 30 21:50:12.386555 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:12.386504 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:50:12.386621 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:50:12.386600 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg6qm" podUID="2e5352ac-8295-494d-950a-16c09b09272e" Oct 30 21:50:12.394034 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:12.392032 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h4v24"] Oct 30 21:50:12.394034 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:12.392121 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:50:12.394034 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:50:12.392204 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4v24" podUID="83a9baf0-064d-4544-94b4-a0dd090dbd04" Oct 30 21:50:14.208133 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:14.208094 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:50:14.208595 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:14.208110 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:50:14.208595 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:50:14.208204 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg6qm" podUID="2e5352ac-8295-494d-950a-16c09b09272e" Oct 30 21:50:14.208595 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:14.208231 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:50:14.208595 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:50:14.208337 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4v24" podUID="83a9baf0-064d-4544-94b4-a0dd090dbd04" Oct 30 21:50:14.208595 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:50:14.208428 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbwzg" podUID="4630b412-da87-49f7-bbbb-ba3dc820ddd1" Oct 30 21:50:16.208466 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:16.208430 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:50:16.208923 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:16.208431 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:50:16.208923 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:50:16.208573 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h4v24" podUID="83a9baf0-064d-4544-94b4-a0dd090dbd04" Oct 30 21:50:16.208923 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:16.208447 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:50:16.208923 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:50:16.208605 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg6qm" podUID="2e5352ac-8295-494d-950a-16c09b09272e" Oct 30 21:50:16.208923 ip-10-0-137-81 kubenswrapper[2576]: E1030 21:50:16.208683 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbwzg" podUID="4630b412-da87-49f7-bbbb-ba3dc820ddd1" Oct 30 21:50:16.969754 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:16.969729 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-west-1.compute.internal" event="NodeReady" Oct 30 21:50:16.969906 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:16.969865 2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 21:50:17.126136 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.126047 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7khqw"] Oct 30 21:50:17.127780 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.127763 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-v6wts"] Oct 30 21:50:17.127920 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.127900 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7khqw" Oct 30 21:50:17.129708 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.129690 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v6wts" Oct 30 21:50:17.147180 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.147156 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zf4n4"] Oct 30 21:50:17.150682 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.149461 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zf4n4" Oct 30 21:50:17.163743 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.163725 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e1f2183b-a823-43ae-b7c8-c4b24dd3b403-crio-socket\") pod \"insights-runtime-extractor-v6wts\" (UID: \"e1f2183b-a823-43ae-b7c8-c4b24dd3b403\") " pod="openshift-insights/insights-runtime-extractor-v6wts" Oct 30 21:50:17.163824 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.163755 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e1f2183b-a823-43ae-b7c8-c4b24dd3b403-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v6wts\" (UID: \"e1f2183b-a823-43ae-b7c8-c4b24dd3b403\") " pod="openshift-insights/insights-runtime-extractor-v6wts" Oct 30 21:50:17.163824 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.163771 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad6cfef-ff22-4eb5-8910-a761537d737f-config-volume\") pod \"dns-default-zf4n4\" (UID: \"0ad6cfef-ff22-4eb5-8910-a761537d737f\") " pod="openshift-dns/dns-default-zf4n4" Oct 30 21:50:17.163824 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.163788 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7163574d-9b90-4a4c-83c5-5ca2d95a6275-cert\") pod \"ingress-canary-7khqw\" (UID: \"7163574d-9b90-4a4c-83c5-5ca2d95a6275\") " pod="openshift-ingress-canary/ingress-canary-7khqw" Oct 30 21:50:17.163957 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.163842 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ad6cfef-ff22-4eb5-8910-a761537d737f-metrics-tls\") pod \"dns-default-zf4n4\" (UID: \"0ad6cfef-ff22-4eb5-8910-a761537d737f\") " pod="openshift-dns/dns-default-zf4n4" Oct 30 21:50:17.163957 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.163917 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1f2183b-a823-43ae-b7c8-c4b24dd3b403-data-volume\") pod \"insights-runtime-extractor-v6wts\" (UID: \"e1f2183b-a823-43ae-b7c8-c4b24dd3b403\") " pod="openshift-insights/insights-runtime-extractor-v6wts" Oct 30 21:50:17.163957 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.163938 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntp69\" (UniqueName: \"kubernetes.io/projected/0ad6cfef-ff22-4eb5-8910-a761537d737f-kube-api-access-ntp69\") pod \"dns-default-zf4n4\" (UID: \"0ad6cfef-ff22-4eb5-8910-a761537d737f\") " pod="openshift-dns/dns-default-zf4n4" Oct 30 21:50:17.164097 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.164023 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ad6cfef-ff22-4eb5-8910-a761537d737f-tmp-dir\") pod \"dns-default-zf4n4\" (UID: \"0ad6cfef-ff22-4eb5-8910-a761537d737f\") " pod="openshift-dns/dns-default-zf4n4" Oct 30 21:50:17.164148 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.164094 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctmq5\" (UniqueName: \"kubernetes.io/projected/e1f2183b-a823-43ae-b7c8-c4b24dd3b403-kube-api-access-ctmq5\") pod \"insights-runtime-extractor-v6wts\" (UID: \"e1f2183b-a823-43ae-b7c8-c4b24dd3b403\") " pod="openshift-insights/insights-runtime-extractor-v6wts" Oct 30 21:50:17.164148 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.164127 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd8ns\" (UniqueName: \"kubernetes.io/projected/7163574d-9b90-4a4c-83c5-5ca2d95a6275-kube-api-access-zd8ns\") pod \"ingress-canary-7khqw\" (UID: \"7163574d-9b90-4a4c-83c5-5ca2d95a6275\") " pod="openshift-ingress-canary/ingress-canary-7khqw" Oct 30 21:50:17.164268 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.164152 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e1f2183b-a823-43ae-b7c8-c4b24dd3b403-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v6wts\" (UID: \"e1f2183b-a823-43ae-b7c8-c4b24dd3b403\") " pod="openshift-insights/insights-runtime-extractor-v6wts" Oct 30 21:50:17.185428 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.185401 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:50:17.185428 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.185412 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:50:17.185570 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.185424 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:50:17.185628 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.185615 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:50:17.185662 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.185615 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-vjtf6\"" Oct 30 21:50:17.186823 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.186809 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:50:17.187108 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.187096 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lwnsl\"" Oct 30 21:50:17.187571 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.187559 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:50:17.187571 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.187567 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:50:17.200965 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.200948 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7khqw"] Oct 30 21:50:17.203972 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.203952 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v6wts"] Oct 30 21:50:17.234554 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.234536 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zf4n4"] Oct 30 21:50:17.265284 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.265251 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e1f2183b-a823-43ae-b7c8-c4b24dd3b403-crio-socket\") pod \"insights-runtime-extractor-v6wts\" (UID: \"e1f2183b-a823-43ae-b7c8-c4b24dd3b403\") " pod="openshift-insights/insights-runtime-extractor-v6wts" Oct 30 21:50:17.265412 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.265297 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e1f2183b-a823-43ae-b7c8-c4b24dd3b403-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v6wts\" (UID: \"e1f2183b-a823-43ae-b7c8-c4b24dd3b403\") " pod="openshift-insights/insights-runtime-extractor-v6wts" Oct 30 21:50:17.265412 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.265323 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad6cfef-ff22-4eb5-8910-a761537d737f-config-volume\") pod \"dns-default-zf4n4\" (UID: \"0ad6cfef-ff22-4eb5-8910-a761537d737f\") " pod="openshift-dns/dns-default-zf4n4" Oct 30 21:50:17.265412 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.265347 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7163574d-9b90-4a4c-83c5-5ca2d95a6275-cert\") pod \"ingress-canary-7khqw\" (UID: \"7163574d-9b90-4a4c-83c5-5ca2d95a6275\") " pod="openshift-ingress-canary/ingress-canary-7khqw" Oct 30 21:50:17.265412 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.265370 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ad6cfef-ff22-4eb5-8910-a761537d737f-metrics-tls\") pod \"dns-default-zf4n4\" (UID: \"0ad6cfef-ff22-4eb5-8910-a761537d737f\") " pod="openshift-dns/dns-default-zf4n4" Oct 30 21:50:17.265412 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.265399 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1f2183b-a823-43ae-b7c8-c4b24dd3b403-data-volume\") pod \"insights-runtime-extractor-v6wts\" (UID: \"e1f2183b-a823-43ae-b7c8-c4b24dd3b403\") " pod="openshift-insights/insights-runtime-extractor-v6wts" Oct 30 21:50:17.265642 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.265425 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntp69\" (UniqueName: \"kubernetes.io/projected/0ad6cfef-ff22-4eb5-8910-a761537d737f-kube-api-access-ntp69\") pod \"dns-default-zf4n4\" (UID: \"0ad6cfef-ff22-4eb5-8910-a761537d737f\") " pod="openshift-dns/dns-default-zf4n4" Oct 30 21:50:17.265642 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.265488 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e1f2183b-a823-43ae-b7c8-c4b24dd3b403-crio-socket\") pod \"insights-runtime-extractor-v6wts\" (UID: \"e1f2183b-a823-43ae-b7c8-c4b24dd3b403\") " pod="openshift-insights/insights-runtime-extractor-v6wts" Oct 30 21:50:17.265745 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.265695 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ad6cfef-ff22-4eb5-8910-a761537d737f-tmp-dir\") pod \"dns-default-zf4n4\" (UID: \"0ad6cfef-ff22-4eb5-8910-a761537d737f\") " pod="openshift-dns/dns-default-zf4n4" Oct 30 21:50:17.265798 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.265767 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctmq5\" (UniqueName: \"kubernetes.io/projected/e1f2183b-a823-43ae-b7c8-c4b24dd3b403-kube-api-access-ctmq5\") pod \"insights-runtime-extractor-v6wts\" (UID: \"e1f2183b-a823-43ae-b7c8-c4b24dd3b403\") " pod="openshift-insights/insights-runtime-extractor-v6wts" Oct 30 21:50:17.265833 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.265802 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd8ns\" (UniqueName: \"kubernetes.io/projected/7163574d-9b90-4a4c-83c5-5ca2d95a6275-kube-api-access-zd8ns\") pod \"ingress-canary-7khqw\" (UID: \"7163574d-9b90-4a4c-83c5-5ca2d95a6275\") " pod="openshift-ingress-canary/ingress-canary-7khqw" Oct 30 21:50:17.265833 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.265765 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1f2183b-a823-43ae-b7c8-c4b24dd3b403-data-volume\") pod \"insights-runtime-extractor-v6wts\" (UID: \"e1f2183b-a823-43ae-b7c8-c4b24dd3b403\") " pod="openshift-insights/insights-runtime-extractor-v6wts" Oct 30 21:50:17.265833 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.265827 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e1f2183b-a823-43ae-b7c8-c4b24dd3b403-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v6wts\" (UID: \"e1f2183b-a823-43ae-b7c8-c4b24dd3b403\") " pod="openshift-insights/insights-runtime-extractor-v6wts" Oct 30 21:50:17.266082 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.266060 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ad6cfef-ff22-4eb5-8910-a761537d737f-tmp-dir\") pod \"dns-default-zf4n4\" (UID: \"0ad6cfef-ff22-4eb5-8910-a761537d737f\") " pod="openshift-dns/dns-default-zf4n4" Oct 30 21:50:17.266394 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.266371 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e1f2183b-a823-43ae-b7c8-c4b24dd3b403-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v6wts\" (UID: \"e1f2183b-a823-43ae-b7c8-c4b24dd3b403\") " pod="openshift-insights/insights-runtime-extractor-v6wts" Oct 30 21:50:17.269193 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.269174 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e1f2183b-a823-43ae-b7c8-c4b24dd3b403-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v6wts\" (UID: \"e1f2183b-a823-43ae-b7c8-c4b24dd3b403\") " pod="openshift-insights/insights-runtime-extractor-v6wts" Oct 30 21:50:17.269263 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.269185 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7163574d-9b90-4a4c-83c5-5ca2d95a6275-cert\") pod \"ingress-canary-7khqw\" (UID: \"7163574d-9b90-4a4c-83c5-5ca2d95a6275\") " pod="openshift-ingress-canary/ingress-canary-7khqw" Oct 30 21:50:17.285473 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.285449 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:50:17.285564 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.285482 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:50:17.285642 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.285626 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-c6mrn\"" Oct 30 21:50:17.288654 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.288640 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ad6cfef-ff22-4eb5-8910-a761537d737f-metrics-tls\") pod \"dns-default-zf4n4\" (UID: \"0ad6cfef-ff22-4eb5-8910-a761537d737f\") " pod="openshift-dns/dns-default-zf4n4" Oct 30 21:50:17.295886 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.295868 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad6cfef-ff22-4eb5-8910-a761537d737f-config-volume\") pod \"dns-default-zf4n4\" (UID: \"0ad6cfef-ff22-4eb5-8910-a761537d737f\") " pod="openshift-dns/dns-default-zf4n4" Oct 30 21:50:17.329064 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.329041 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctmq5\" (UniqueName: \"kubernetes.io/projected/e1f2183b-a823-43ae-b7c8-c4b24dd3b403-kube-api-access-ctmq5\") pod \"insights-runtime-extractor-v6wts\" (UID: \"e1f2183b-a823-43ae-b7c8-c4b24dd3b403\") " pod="openshift-insights/insights-runtime-extractor-v6wts" Oct 30 21:50:17.329343 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.329326 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd8ns\" (UniqueName: \"kubernetes.io/projected/7163574d-9b90-4a4c-83c5-5ca2d95a6275-kube-api-access-zd8ns\") pod \"ingress-canary-7khqw\" (UID: \"7163574d-9b90-4a4c-83c5-5ca2d95a6275\") " pod="openshift-ingress-canary/ingress-canary-7khqw" Oct 30 21:50:17.329732 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.329711 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntp69\" (UniqueName: \"kubernetes.io/projected/0ad6cfef-ff22-4eb5-8910-a761537d737f-kube-api-access-ntp69\") pod \"dns-default-zf4n4\" (UID: \"0ad6cfef-ff22-4eb5-8910-a761537d737f\") " pod="openshift-dns/dns-default-zf4n4" Oct 30 21:50:17.439771 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.439748 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7khqw" Oct 30 21:50:17.446316 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.446289 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v6wts" Oct 30 21:50:17.460026 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.460002 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zf4n4" Oct 30 21:50:17.862732 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.862649 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7khqw"] Oct 30 21:50:17.866743 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:50:17.866712 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7163574d_9b90_4a4c_83c5_5ca2d95a6275.slice/crio-fb3e60f0f46a4d1992a0924c8dec57c15e8f16ff740e92fadeaf6f22455e54e6 WatchSource:0}: Error finding container fb3e60f0f46a4d1992a0924c8dec57c15e8f16ff740e92fadeaf6f22455e54e6: Status 404 returned error can't find the container with id fb3e60f0f46a4d1992a0924c8dec57c15e8f16ff740e92fadeaf6f22455e54e6 Oct 30 21:50:17.881742 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.881720 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v6wts"] Oct 30 21:50:17.884357 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:17.884337 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zf4n4"] Oct 30 21:50:17.884807 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:50:17.884785 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1f2183b_a823_43ae_b7c8_c4b24dd3b403.slice/crio-03c8dd34da7ccc342a64c88427c9faa1d368d5aa84977dfe9abde48b44a5d7ee WatchSource:0}: Error finding container 03c8dd34da7ccc342a64c88427c9faa1d368d5aa84977dfe9abde48b44a5d7ee: Status 404 returned error can't find the container with id 03c8dd34da7ccc342a64c88427c9faa1d368d5aa84977dfe9abde48b44a5d7ee Oct 30 21:50:17.887564 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:50:17.887541 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad6cfef_ff22_4eb5_8910_a761537d737f.slice/crio-bcd76e95362cbb32470158be7f72ebad66a49c3885bb19b9e1c292850d12b328 WatchSource:0}: Error finding container bcd76e95362cbb32470158be7f72ebad66a49c3885bb19b9e1c292850d12b328: Status 404 returned error can't find the container with id bcd76e95362cbb32470158be7f72ebad66a49c3885bb19b9e1c292850d12b328 Oct 30 21:50:18.210365 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:18.210342 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:50:18.210534 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:18.210451 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:50:18.210608 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:18.210593 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:50:18.264090 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:18.264066 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:50:18.264090 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:18.264066 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:50:18.264977 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:18.264942 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-gmxqg\"" Oct 30 21:50:18.265084 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:18.264956 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r6nvz\"" Oct 30 21:50:18.265636 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:18.265614 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:50:18.265825 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:18.265797 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:50:18.265933 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:18.265916 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jjx9q\"" Oct 30 21:50:18.447501 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:18.447466 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zf4n4" event={"ID":"0ad6cfef-ff22-4eb5-8910-a761537d737f","Type":"ContainerStarted","Data":"bcd76e95362cbb32470158be7f72ebad66a49c3885bb19b9e1c292850d12b328"} Oct 30 21:50:18.449001 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:18.448975 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v6wts" event={"ID":"e1f2183b-a823-43ae-b7c8-c4b24dd3b403","Type":"ContainerStarted","Data":"4b3e8fe76746a43f7ee70c17bd1a6944c478566e3cf53d24afb59f25fd8fc2f8"} Oct 30 21:50:18.449113 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:18.449024 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v6wts" event={"ID":"e1f2183b-a823-43ae-b7c8-c4b24dd3b403","Type":"ContainerStarted","Data":"03c8dd34da7ccc342a64c88427c9faa1d368d5aa84977dfe9abde48b44a5d7ee"} Oct 30 21:50:18.450071 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:18.450045 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7khqw" event={"ID":"7163574d-9b90-4a4c-83c5-5ca2d95a6275","Type":"ContainerStarted","Data":"fb3e60f0f46a4d1992a0924c8dec57c15e8f16ff740e92fadeaf6f22455e54e6"} Oct 30 21:50:19.454095 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:19.454020 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v6wts" event={"ID":"e1f2183b-a823-43ae-b7c8-c4b24dd3b403","Type":"ContainerStarted","Data":"30ea68d217d1dafa9eeedfec551e99c4cc9f9f8cbb9f1cababdbec8627591554"} Oct 30 21:50:20.682877 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.682848 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-x796z"] Oct 30 21:50:20.685251 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.685209 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x796z" Oct 30 21:50:20.731668 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.731628 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-q5c65"] Oct 30 21:50:20.733860 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.733837 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:20.739857 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.739839 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:50:20.740063 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.740040 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 21:50:20.740135 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.740091 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:50:20.784734 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.781457 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 21:50:20.784734 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.781939 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-78zg5\"" Oct 30 21:50:20.784734 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.782270 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 21:50:20.788742 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.788720 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-x796z"] Oct 30 21:50:20.794484 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.794465 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-node-exporter-tls\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:20.794594 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.794492 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-node-exporter-accelerators-collector-config\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:20.794594 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.794528 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-node-exporter-wtmp\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:20.794594 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.794551 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c199354-31a1-45b4-8c60-a485fd65f1e2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-x796z\" (UID: \"3c199354-31a1-45b4-8c60-a485fd65f1e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x796z" Oct 30 21:50:20.794759 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.794621 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4lcd\" (UniqueName: \"kubernetes.io/projected/3c199354-31a1-45b4-8c60-a485fd65f1e2-kube-api-access-j4lcd\") pod \"openshift-state-metrics-7bd7489957-x796z\" (UID: \"3c199354-31a1-45b4-8c60-a485fd65f1e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x796z" Oct 30 21:50:20.794759 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.794703 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-node-exporter-textfile\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:20.794759 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.794744 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c199354-31a1-45b4-8c60-a485fd65f1e2-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-x796z\" (UID: \"3c199354-31a1-45b4-8c60-a485fd65f1e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x796z" Oct 30 21:50:20.794907 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.794824 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3c199354-31a1-45b4-8c60-a485fd65f1e2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-x796z\" (UID: \"3c199354-31a1-45b4-8c60-a485fd65f1e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x796z" Oct 30 21:50:20.794907 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.794885 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-root\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:20.795008 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.794909 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n67nc\" (UniqueName: \"kubernetes.io/projected/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-kube-api-access-n67nc\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:20.795008 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.794941 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-sys\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:20.795008 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.794965 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:20.795150 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.795005 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-metrics-client-ca\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:20.826985 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.826445 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4"] Oct 30 21:50:20.829293 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.829267 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4" Oct 30 21:50:20.840932 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.840846 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 21:50:20.840932 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.840900 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 21:50:20.841122 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.840996 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-v5k6j\"" Oct 30 21:50:20.841122 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.841072 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 21:50:20.892320 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.892287 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4"] Oct 30 21:50:20.895545 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.895515 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4lcd\" (UniqueName: \"kubernetes.io/projected/3c199354-31a1-45b4-8c60-a485fd65f1e2-kube-api-access-j4lcd\") pod \"openshift-state-metrics-7bd7489957-x796z\" (UID: \"3c199354-31a1-45b4-8c60-a485fd65f1e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x796z" Oct 30 21:50:20.895727 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.895558 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-node-exporter-textfile\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:20.895727 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.895587 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c199354-31a1-45b4-8c60-a485fd65f1e2-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-x796z\" (UID: \"3c199354-31a1-45b4-8c60-a485fd65f1e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x796z" Oct 30 21:50:20.895727 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.895617 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1e76444f-1a5b-460c-b42f-2197b7512574-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-lbjd4\" (UID: \"1e76444f-1a5b-460c-b42f-2197b7512574\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4" Oct 30 21:50:20.895727 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.895647 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3c199354-31a1-45b4-8c60-a485fd65f1e2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-x796z\" (UID: \"3c199354-31a1-45b4-8c60-a485fd65f1e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x796z" Oct 30 21:50:20.895727 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.895680 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1e76444f-1a5b-460c-b42f-2197b7512574-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-lbjd4\" (UID: \"1e76444f-1a5b-460c-b42f-2197b7512574\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4" Oct 30 21:50:20.895727 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.895710 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-root\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:20.896011 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.895735 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e76444f-1a5b-460c-b42f-2197b7512574-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-lbjd4\" (UID: \"1e76444f-1a5b-460c-b42f-2197b7512574\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4" Oct 30 21:50:20.896011 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.895762 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-sys\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:20.896011 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.895813 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-node-exporter-tls\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:20.896011 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.895855 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-node-exporter-wtmp\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:20.896011 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.895878 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e76444f-1a5b-460c-b42f-2197b7512574-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-lbjd4\" (UID: \"1e76444f-1a5b-460c-b42f-2197b7512574\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4" Oct 30 21:50:20.896011 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.895906 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c199354-31a1-45b4-8c60-a485fd65f1e2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-x796z\" (UID: \"3c199354-31a1-45b4-8c60-a485fd65f1e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x796z" Oct 30 21:50:20.896011 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.895932 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf4tt\" (UniqueName: \"kubernetes.io/projected/1e76444f-1a5b-460c-b42f-2197b7512574-kube-api-access-mf4tt\") pod \"kube-state-metrics-5dc8d7d89b-lbjd4\" (UID: \"1e76444f-1a5b-460c-b42f-2197b7512574\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4" Oct 30 21:50:20.896011 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.895963 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-node-exporter-accelerators-collector-config\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:20.896011 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.895998 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n67nc\" (UniqueName: \"kubernetes.io/projected/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-kube-api-access-n67nc\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:20.896405 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.896024 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:20.896405 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.896040 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-metrics-client-ca\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:20.896405 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.896060 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e76444f-1a5b-460c-b42f-2197b7512574-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-lbjd4\" (UID: \"1e76444f-1a5b-460c-b42f-2197b7512574\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4" Oct 30 21:50:20.897506 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.896749 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-node-exporter-textfile\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:20.897506 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.897070 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-node-exporter-wtmp\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:20.897506 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.897473 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-node-exporter-accelerators-collector-config\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:20.897874 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.897848 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-sys\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:20.897955 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.897857 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-root\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:20.899439 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.898741 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c199354-31a1-45b4-8c60-a485fd65f1e2-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-x796z\" (UID: \"3c199354-31a1-45b4-8c60-a485fd65f1e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x796z" Oct 30 21:50:20.899439 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.899158 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-metrics-client-ca\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:20.900480 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.900416 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:20.900811 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.900790 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3c199354-31a1-45b4-8c60-a485fd65f1e2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-x796z\" (UID: \"3c199354-31a1-45b4-8c60-a485fd65f1e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x796z" Oct 30 21:50:20.901794 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.901771 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-node-exporter-tls\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:20.905372 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.905311 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c199354-31a1-45b4-8c60-a485fd65f1e2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-x796z\" (UID: \"3c199354-31a1-45b4-8c60-a485fd65f1e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x796z" Oct 30 21:50:20.940553 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.940519 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 21:50:20.940553 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.940542 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-mcdr9\"" Oct 30 21:50:20.940776 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.940678 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 21:50:20.940776 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.940763 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 21:50:20.959518 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.959490 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4lcd\" (UniqueName: \"kubernetes.io/projected/3c199354-31a1-45b4-8c60-a485fd65f1e2-kube-api-access-j4lcd\") pod \"openshift-state-metrics-7bd7489957-x796z\" (UID: \"3c199354-31a1-45b4-8c60-a485fd65f1e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x796z" Oct 30 21:50:20.967498 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.967467 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n67nc\" (UniqueName: \"kubernetes.io/projected/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-kube-api-access-n67nc\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:20.996938 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.996913 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e76444f-1a5b-460c-b42f-2197b7512574-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-lbjd4\" (UID: \"1e76444f-1a5b-460c-b42f-2197b7512574\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4" Oct 30 21:50:20.997087 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.996945 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf4tt\" (UniqueName: \"kubernetes.io/projected/1e76444f-1a5b-460c-b42f-2197b7512574-kube-api-access-mf4tt\") pod \"kube-state-metrics-5dc8d7d89b-lbjd4\" (UID: \"1e76444f-1a5b-460c-b42f-2197b7512574\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4" Oct 30 21:50:20.997087 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.996992 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e76444f-1a5b-460c-b42f-2197b7512574-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-lbjd4\" (UID: \"1e76444f-1a5b-460c-b42f-2197b7512574\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4" Oct 30 21:50:20.997087 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.997038 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1e76444f-1a5b-460c-b42f-2197b7512574-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-lbjd4\" (UID: \"1e76444f-1a5b-460c-b42f-2197b7512574\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4" Oct 30 21:50:20.997087 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.997069 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1e76444f-1a5b-460c-b42f-2197b7512574-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-lbjd4\" (UID: \"1e76444f-1a5b-460c-b42f-2197b7512574\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4" Oct 30 21:50:20.997307 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.997101 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e76444f-1a5b-460c-b42f-2197b7512574-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-lbjd4\" (UID: \"1e76444f-1a5b-460c-b42f-2197b7512574\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4" Oct 30 21:50:20.998020 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.997988 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1e76444f-1a5b-460c-b42f-2197b7512574-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-lbjd4\" (UID: \"1e76444f-1a5b-460c-b42f-2197b7512574\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4" Oct 30 21:50:20.998126 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.998033 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1e76444f-1a5b-460c-b42f-2197b7512574-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-lbjd4\" (UID: \"1e76444f-1a5b-460c-b42f-2197b7512574\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4" Oct 30 21:50:20.998126 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.998036 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e76444f-1a5b-460c-b42f-2197b7512574-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-lbjd4\" (UID: \"1e76444f-1a5b-460c-b42f-2197b7512574\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4" Oct 30 21:50:20.999448 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.999424 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x796z" Oct 30 21:50:20.999745 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.999725 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e76444f-1a5b-460c-b42f-2197b7512574-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-lbjd4\" (UID: \"1e76444f-1a5b-460c-b42f-2197b7512574\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4" Oct 30 21:50:20.999824 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:20.999761 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e76444f-1a5b-460c-b42f-2197b7512574-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-lbjd4\" (UID: \"1e76444f-1a5b-460c-b42f-2197b7512574\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4" Oct 30 21:50:21.047868 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:21.047591 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:21.066244 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:21.066092 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf4tt\" (UniqueName: \"kubernetes.io/projected/1e76444f-1a5b-460c-b42f-2197b7512574-kube-api-access-mf4tt\") pod \"kube-state-metrics-5dc8d7d89b-lbjd4\" (UID: \"1e76444f-1a5b-460c-b42f-2197b7512574\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4" Oct 30 21:50:21.074249 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:50:21.074196 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a7edc22_1eb1_4458_8f8c_6c46a2bd583c.slice/crio-fde0502c1ae08a6c80f38283d5a6c31b3c9c9835c5fc48a33c0e90a5b21cc559 WatchSource:0}: Error finding container fde0502c1ae08a6c80f38283d5a6c31b3c9c9835c5fc48a33c0e90a5b21cc559: Status 404 returned error can't find the container with id fde0502c1ae08a6c80f38283d5a6c31b3c9c9835c5fc48a33c0e90a5b21cc559 Oct 30 21:50:21.144899 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:21.144863 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4" Oct 30 21:50:21.433778 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:21.433746 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-x796z"] Oct 30 21:50:21.438115 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:50:21.438087 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c199354_31a1_45b4_8c60_a485fd65f1e2.slice/crio-67dc0f160b11ce4f01ae7bbcbce36724376f58941b510f9809856eadbbe77b76 WatchSource:0}: Error finding container 67dc0f160b11ce4f01ae7bbcbce36724376f58941b510f9809856eadbbe77b76: Status 404 returned error can't find the container with id 67dc0f160b11ce4f01ae7bbcbce36724376f58941b510f9809856eadbbe77b76 Oct 30 21:50:21.460992 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:21.460933 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v6wts" event={"ID":"e1f2183b-a823-43ae-b7c8-c4b24dd3b403","Type":"ContainerStarted","Data":"a2486378dc9d64da2f6b095cfd1020f8405d653b3b7980422c73b547f90f240b"} Oct 30 21:50:21.462396 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:21.462363 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q5c65" event={"ID":"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c","Type":"ContainerStarted","Data":"fde0502c1ae08a6c80f38283d5a6c31b3c9c9835c5fc48a33c0e90a5b21cc559"} Oct 30 21:50:21.463728 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:21.463695 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x796z" event={"ID":"3c199354-31a1-45b4-8c60-a485fd65f1e2","Type":"ContainerStarted","Data":"67dc0f160b11ce4f01ae7bbcbce36724376f58941b510f9809856eadbbe77b76"} Oct 30 21:50:21.465690 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:21.465668 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zf4n4" event={"ID":"0ad6cfef-ff22-4eb5-8910-a761537d737f","Type":"ContainerStarted","Data":"9cabdb497687b1bb3925f05bb27ba1626dbe75265707a578f09906fef62eccc0"} Oct 30 21:50:21.465828 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:21.465711 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zf4n4" event={"ID":"0ad6cfef-ff22-4eb5-8910-a761537d737f","Type":"ContainerStarted","Data":"074076cb0869663058c650d44e3289c991eea98bc2ecd8cb41fccebcfd170d7d"} Oct 30 21:50:21.465828 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:21.465755 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zf4n4" Oct 30 21:50:21.532135 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:21.532108 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4"] Oct 30 21:50:21.536527 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:50:21.536495 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e76444f_1a5b_460c_b42f_2197b7512574.slice/crio-b54267304a3fa7233171f6499fc3bcdf39e09bfacd448f597e4b0840df316420 WatchSource:0}: Error finding container b54267304a3fa7233171f6499fc3bcdf39e09bfacd448f597e4b0840df316420: Status 404 returned error can't find the container with id b54267304a3fa7233171f6499fc3bcdf39e09bfacd448f597e4b0840df316420 Oct 30 21:50:21.702565 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:21.702514 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-zf4n4" podStartSLOduration=2.110225416 podStartE2EDuration="4.70250003s" totalImagesPullingTime="2.592274614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:21.701679104 +0000 UTC m=+53.977048397" watchObservedRunningTime="2026-10-30 21:50:21.70250003 +0000 UTC m=+53.977869336" Oct 30 21:50:21.702974 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:21.702659 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-v6wts" podStartSLOduration=1.473771554 podStartE2EDuration="4.702651015s" totalImagesPullingTime="3.228879461s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:50:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:21.58219549 +0000 UTC m=+53.857564783" watchObservedRunningTime="2026-10-30 21:50:21.702651015 +0000 UTC m=+53.978020296" Oct 30 21:50:22.472278 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:22.472234 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x796z" event={"ID":"3c199354-31a1-45b4-8c60-a485fd65f1e2","Type":"ContainerStarted","Data":"a6d29fe99c0de41f49f8d70ca0644670c055946fcff9774ceb65d256d9b932f8"} Oct 30 21:50:22.472464 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:22.472311 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x796z" event={"ID":"3c199354-31a1-45b4-8c60-a485fd65f1e2","Type":"ContainerStarted","Data":"c3f3318d545820c9f9647f6ce2db6c21a5cd331846e315fb5652f3850c7704c0"} Oct 30 21:50:22.473885 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:22.473851 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4" event={"ID":"1e76444f-1a5b-460c-b42f-2197b7512574","Type":"ContainerStarted","Data":"b54267304a3fa7233171f6499fc3bcdf39e09bfacd448f597e4b0840df316420"} Oct 30 21:50:22.475523 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:22.475497 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7khqw" event={"ID":"7163574d-9b90-4a4c-83c5-5ca2d95a6275","Type":"ContainerStarted","Data":"0c6dded4a69f40e09c3d33cbd11cab8d68394596171ae26f809abe0bd35fc8da"} Oct 30 21:50:22.595204 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:22.595150 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7khqw" podStartSLOduration=1.7704974020000002 podStartE2EDuration="5.595132408s" totalImagesPullingTime="3.824635006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:22.594775498 +0000 UTC m=+54.870145058" watchObservedRunningTime="2026-10-30 21:50:22.595132408 +0000 UTC m=+54.870501691" Oct 30 21:50:23.047780 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.047693 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 21:50:23.050690 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.050664 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.111609 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.111576 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 21:50:23.111609 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.111583 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 21:50:23.111808 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.111588 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 21:50:23.111808 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.111588 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 21:50:23.111808 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.111654 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 21:50:23.111808 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.111779 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 21:50:23.111957 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.111883 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-dhvch\"" Oct 30 21:50:23.115039 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.115011 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-config-volume\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.115172 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.115051 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.115172 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.115085 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.115300 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.115207 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.115300 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.115270 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.115378 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.115310 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-web-config\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.115378 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.115343 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-config-out\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.115378 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.115358 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.115378 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.115376 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g25nc\" (UniqueName: \"kubernetes.io/projected/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-kube-api-access-g25nc\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.115566 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.115396 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.115566 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.115517 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.115566 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.115547 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.115691 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.115598 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.213498 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.210788 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 21:50:23.213498 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.212389 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 21:50:23.214950 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.214925 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 21:50:23.215155 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.215133 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 21:50:23.216399 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.216378 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.216496 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.216433 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.216496 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.216453 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.216496 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.216469 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-web-config\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.216633 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.216507 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-config-out\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.216633 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.216530 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.216633 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.216552 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g25nc\" (UniqueName: \"kubernetes.io/projected/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-kube-api-access-g25nc\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.216633 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.216578 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.216633 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.216623 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.217328 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.217062 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.217328 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.217292 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.220967 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.217368 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.220967 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.218284 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-config-volume\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.220967 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.218506 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.220967 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.220432 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.220967 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.220531 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.222802 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.221622 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.224888 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.224004 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.224888 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.224005 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.224888 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.224168 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-config-out\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.224888 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.224499 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.229352 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.229303 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-web-config\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.233653 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.229576 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.233653 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.229987 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.234298 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.234253 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-config-volume\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.278825 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.278781 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g25nc\" (UniqueName: \"kubernetes.io/projected/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-kube-api-access-g25nc\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.369588 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.369554 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:23.480270 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.480206 2576 generic.go:356] "Generic (PLEG): container finished" podID="4a7edc22-1eb1-4458-8f8c-6c46a2bd583c" containerID="8b8e2ef3d207dd9acbf3a214fec4b521041f5ebd3986fe535ca44813a98e918c" exitCode=0 Oct 30 21:50:23.480419 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.480307 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q5c65" event={"ID":"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c","Type":"ContainerDied","Data":"8b8e2ef3d207dd9acbf3a214fec4b521041f5ebd3986fe535ca44813a98e918c"} Oct 30 21:50:23.482673 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.482643 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x796z" event={"ID":"3c199354-31a1-45b4-8c60-a485fd65f1e2","Type":"ContainerStarted","Data":"8e7dfcd31bc11b2306a8326d6de36024ee690b237633528b5c4833b1c14dab04"} Oct 30 21:50:23.484733 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.484710 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4" event={"ID":"1e76444f-1a5b-460c-b42f-2197b7512574","Type":"ContainerStarted","Data":"f6fdea9781b7f9ea53a0fef6dfde2ddf03aab9ae3c3af1246515bb346d878d14"} Oct 30 21:50:23.484847 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.484784 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4" event={"ID":"1e76444f-1a5b-460c-b42f-2197b7512574","Type":"ContainerStarted","Data":"c651ac93342f573cc12ce8acd9b20ed71948dad66e546a6af26e07b5cc803729"} Oct 30 21:50:23.484847 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.484815 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4" event={"ID":"1e76444f-1a5b-460c-b42f-2197b7512574","Type":"ContainerStarted","Data":"8bb691f3053dbc9ed494e20fc3f164b22f60024b0c638816bfdd89a9a94f2aee"} Oct 30 21:50:23.518708 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.518689 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-796d66f799-qtc4t"] Oct 30 21:50:23.522053 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.522034 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:23.575643 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.575609 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 21:50:23.575643 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.575628 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 21:50:23.575643 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.575639 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-esign78lrs486\"" Oct 30 21:50:23.575805 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.575650 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 21:50:23.575968 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.575949 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-45gdr\"" Oct 30 21:50:23.576053 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.575973 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 21:50:23.576053 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.575996 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 21:50:23.586416 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.586394 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-796d66f799-qtc4t"] Oct 30 21:50:23.622647 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.621960 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:23.622647 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.622045 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-tmp\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:23.622647 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.622078 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-metrics-client-ca\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:23.622647 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.622111 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-secret-thanos-querier-tls\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:23.622647 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.622134 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:23.622647 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.622168 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgqbt\" (UniqueName: \"kubernetes.io/projected/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-kube-api-access-sgqbt\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:23.622647 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.622204 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:23.622647 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.622270 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-secret-grpc-tls\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:23.622647 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.622297 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:23.723165 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.722544 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x796z" podStartSLOduration=2.134909534 podStartE2EDuration="3.722524281s" totalImagesPullingTime="1.587614747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:23.72171461 +0000 UTC m=+55.997083885" watchObservedRunningTime="2026-10-30 21:50:23.722524281 +0000 UTC m=+55.997893563" Oct 30 21:50:23.723886 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.723859 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:23.724490 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.724464 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-secret-grpc-tls\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:23.724588 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.724498 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:23.724588 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.724558 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:23.724705 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.724614 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-tmp\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:23.724705 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.724649 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-metrics-client-ca\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:23.724705 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.724682 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-secret-thanos-querier-tls\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:23.724842 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.724711 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:23.724842 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.724750 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgqbt\" (UniqueName: \"kubernetes.io/projected/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-kube-api-access-sgqbt\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:23.725175 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.725153 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-tmp\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:23.725875 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.725851 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-metrics-client-ca\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:23.727177 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.727152 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:23.730084 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.728785 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-secret-thanos-querier-tls\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:23.730084 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.728861 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-secret-grpc-tls\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:23.730084 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.729449 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:23.730084 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.729975 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:23.731079 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.731051 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:23.786881 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.786848 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgqbt\" (UniqueName: \"kubernetes.io/projected/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-kube-api-access-sgqbt\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:23.807361 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.807278 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 21:50:23.810617 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:50:23.810590 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e62c7d9_71b5_4f9b_a587_aaf06c7b0735.slice/crio-f5e4b161dde81879deee20f513c5a60b7623438253853280a93b681288a12fb8 WatchSource:0}: Error finding container f5e4b161dde81879deee20f513c5a60b7623438253853280a93b681288a12fb8: Status 404 returned error can't find the container with id f5e4b161dde81879deee20f513c5a60b7623438253853280a93b681288a12fb8 Oct 30 21:50:23.845835 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.845806 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:23.968209 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:23.968157 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4" podStartSLOduration=2.3727406 podStartE2EDuration="3.968141541s" totalImagesPullingTime="1.595400941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:23.967508393 +0000 UTC m=+56.242877698" watchObservedRunningTime="2026-10-30 21:50:23.968141541 +0000 UTC m=+56.243510821" Oct 30 21:50:24.236485 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:24.236459 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-796d66f799-qtc4t"] Oct 30 21:50:24.237986 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:50:24.237961 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4925cf0_04b5_4ca4_b284_c0355e63b4ca.slice/crio-e31f54543dce236fa066ea1688a07b45980761a3231c0c804cc51d8f9e30fd2d WatchSource:0}: Error finding container e31f54543dce236fa066ea1688a07b45980761a3231c0c804cc51d8f9e30fd2d: Status 404 returned error can't find the container with id e31f54543dce236fa066ea1688a07b45980761a3231c0c804cc51d8f9e30fd2d Oct 30 21:50:24.407773 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:24.407748 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:24.489974 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:24.489873 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q5c65" event={"ID":"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c","Type":"ContainerStarted","Data":"33e30037343eca35ed2336b1d1831ad4fd704108e61eefc46979706562c99947"} Oct 30 21:50:24.489974 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:24.489944 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q5c65" event={"ID":"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c","Type":"ContainerStarted","Data":"361a7cc24ed43b2ecd69fcdb4dfde2e30d97d7c8b206d8162d035d2a9fa705de"} Oct 30 21:50:24.491088 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:24.491058 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735","Type":"ContainerStarted","Data":"f5e4b161dde81879deee20f513c5a60b7623438253853280a93b681288a12fb8"} Oct 30 21:50:24.492252 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:24.492226 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" event={"ID":"f4925cf0-04b5-4ca4-b284-c0355e63b4ca","Type":"ContainerStarted","Data":"e31f54543dce236fa066ea1688a07b45980761a3231c0c804cc51d8f9e30fd2d"} Oct 30 21:50:25.410923 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.409207 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-q5c65" podStartSLOduration=3.811643567 podStartE2EDuration="5.40918512s" totalImagesPullingTime="1.511248794s" totalInitContainerRuntime="86.292759ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:50:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:24.663266293 +0000 UTC m=+56.938635575" watchObservedRunningTime="2026-10-30 21:50:25.40918512 +0000 UTC m=+57.684554403" Oct 30 21:50:25.410923 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.409973 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-dd5t2"] Oct 30 21:50:25.412405 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.411990 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dd5t2" Oct 30 21:50:25.442235 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.442190 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58wwh\" (UniqueName: \"kubernetes.io/projected/290eb94c-6ae5-40e8-8ba3-dbe017243d9d-kube-api-access-58wwh\") pod \"monitoring-plugin-77f4f76594-dd5t2\" (UID: \"290eb94c-6ae5-40e8-8ba3-dbe017243d9d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dd5t2" Oct 30 21:50:25.442435 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.442411 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/290eb94c-6ae5-40e8-8ba3-dbe017243d9d-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-dd5t2\" (UID: \"290eb94c-6ae5-40e8-8ba3-dbe017243d9d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dd5t2" Oct 30 21:50:25.466027 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.466003 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 21:50:25.466027 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.466018 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-4k7qj\"" Oct 30 21:50:25.473682 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.473618 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-dd5t2"] Oct 30 21:50:25.497634 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.497604 2576 generic.go:356] "Generic (PLEG): container finished" podID="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" containerID="b7029e8c3647fc8cf124a1d4e669b174d5c966d763a6a2cbc18befc88e3d1ffc" exitCode=0 Oct 30 21:50:25.497772 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.497685 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735","Type":"ContainerDied","Data":"b7029e8c3647fc8cf124a1d4e669b174d5c966d763a6a2cbc18befc88e3d1ffc"} Oct 30 21:50:25.543085 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.543046 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58wwh\" (UniqueName: \"kubernetes.io/projected/290eb94c-6ae5-40e8-8ba3-dbe017243d9d-kube-api-access-58wwh\") pod \"monitoring-plugin-77f4f76594-dd5t2\" (UID: \"290eb94c-6ae5-40e8-8ba3-dbe017243d9d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dd5t2" Oct 30 21:50:25.543534 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.543507 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/290eb94c-6ae5-40e8-8ba3-dbe017243d9d-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-dd5t2\" (UID: \"290eb94c-6ae5-40e8-8ba3-dbe017243d9d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dd5t2" Oct 30 21:50:25.546649 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.546623 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/290eb94c-6ae5-40e8-8ba3-dbe017243d9d-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-dd5t2\" (UID: \"290eb94c-6ae5-40e8-8ba3-dbe017243d9d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dd5t2" Oct 30 21:50:25.605899 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.605851 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58wwh\" (UniqueName: \"kubernetes.io/projected/290eb94c-6ae5-40e8-8ba3-dbe017243d9d-kube-api-access-58wwh\") pod \"monitoring-plugin-77f4f76594-dd5t2\" (UID: \"290eb94c-6ae5-40e8-8ba3-dbe017243d9d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dd5t2" Oct 30 21:50:25.723988 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.723953 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dd5t2" Oct 30 21:50:25.773434 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.773401 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-d6fb658f8-zvqw4"] Oct 30 21:50:25.777632 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.776506 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" Oct 30 21:50:25.830389 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.830360 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 21:50:25.830525 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.830457 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 21:50:25.830525 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.830458 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 21:50:25.830639 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.830540 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 21:50:25.830639 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.830597 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-3a3p8gb9pmm2\"" Oct 30 21:50:25.830745 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.830665 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-9vsls\"" Oct 30 21:50:25.837061 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.837038 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-d6fb658f8-zvqw4"] Oct 30 21:50:25.846798 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.846769 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d6fb658f8-zvqw4\" (UID: \"e321c1b1-e9e2-46fc-9a35-14a3d6bed233\") " pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" Oct 30 21:50:25.846920 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.846817 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7nrk\" (UniqueName: \"kubernetes.io/projected/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-kube-api-access-q7nrk\") pod \"metrics-server-d6fb658f8-zvqw4\" (UID: \"e321c1b1-e9e2-46fc-9a35-14a3d6bed233\") " pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" Oct 30 21:50:25.846920 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.846870 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-secret-metrics-server-tls\") pod \"metrics-server-d6fb658f8-zvqw4\" (UID: \"e321c1b1-e9e2-46fc-9a35-14a3d6bed233\") " pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" Oct 30 21:50:25.847006 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.846937 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-audit-log\") pod \"metrics-server-d6fb658f8-zvqw4\" (UID: \"e321c1b1-e9e2-46fc-9a35-14a3d6bed233\") " pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" Oct 30 21:50:25.847006 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.846973 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-secret-metrics-server-client-certs\") pod \"metrics-server-d6fb658f8-zvqw4\" (UID: \"e321c1b1-e9e2-46fc-9a35-14a3d6bed233\") " pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" Oct 30 21:50:25.847080 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.847062 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-client-ca-bundle\") pod \"metrics-server-d6fb658f8-zvqw4\" (UID: \"e321c1b1-e9e2-46fc-9a35-14a3d6bed233\") " pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" Oct 30 21:50:25.847180 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.847148 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-metrics-server-audit-profiles\") pod \"metrics-server-d6fb658f8-zvqw4\" (UID: \"e321c1b1-e9e2-46fc-9a35-14a3d6bed233\") " pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" Oct 30 21:50:25.947687 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.947652 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-audit-log\") pod \"metrics-server-d6fb658f8-zvqw4\" (UID: \"e321c1b1-e9e2-46fc-9a35-14a3d6bed233\") " pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" Oct 30 21:50:25.947845 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.947701 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-secret-metrics-server-client-certs\") pod \"metrics-server-d6fb658f8-zvqw4\" (UID: \"e321c1b1-e9e2-46fc-9a35-14a3d6bed233\") " pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" Oct 30 21:50:25.947845 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.947753 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-client-ca-bundle\") pod \"metrics-server-d6fb658f8-zvqw4\" (UID: \"e321c1b1-e9e2-46fc-9a35-14a3d6bed233\") " pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" Oct 30 21:50:25.947845 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.947796 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-metrics-server-audit-profiles\") pod \"metrics-server-d6fb658f8-zvqw4\" (UID: \"e321c1b1-e9e2-46fc-9a35-14a3d6bed233\") " pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" Oct 30 21:50:25.948322 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.948282 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-audit-log\") pod \"metrics-server-d6fb658f8-zvqw4\" (UID: \"e321c1b1-e9e2-46fc-9a35-14a3d6bed233\") " pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" Oct 30 21:50:25.948687 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.948326 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d6fb658f8-zvqw4\" (UID: \"e321c1b1-e9e2-46fc-9a35-14a3d6bed233\") " pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" Oct 30 21:50:25.948687 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.948365 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7nrk\" (UniqueName: \"kubernetes.io/projected/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-kube-api-access-q7nrk\") pod \"metrics-server-d6fb658f8-zvqw4\" (UID: \"e321c1b1-e9e2-46fc-9a35-14a3d6bed233\") " pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" Oct 30 21:50:25.948687 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.948405 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-secret-metrics-server-tls\") pod \"metrics-server-d6fb658f8-zvqw4\" (UID: \"e321c1b1-e9e2-46fc-9a35-14a3d6bed233\") " pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" Oct 30 21:50:25.948873 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.948734 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-metrics-server-audit-profiles\") pod \"metrics-server-d6fb658f8-zvqw4\" (UID: \"e321c1b1-e9e2-46fc-9a35-14a3d6bed233\") " pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" Oct 30 21:50:25.949233 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.949188 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d6fb658f8-zvqw4\" (UID: \"e321c1b1-e9e2-46fc-9a35-14a3d6bed233\") " pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" Oct 30 21:50:25.950389 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.950371 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-secret-metrics-server-client-certs\") pod \"metrics-server-d6fb658f8-zvqw4\" (UID: \"e321c1b1-e9e2-46fc-9a35-14a3d6bed233\") " pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" Oct 30 21:50:25.950619 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.950601 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-client-ca-bundle\") pod \"metrics-server-d6fb658f8-zvqw4\" (UID: \"e321c1b1-e9e2-46fc-9a35-14a3d6bed233\") " pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" Oct 30 21:50:25.950829 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:25.950792 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-secret-metrics-server-tls\") pod \"metrics-server-d6fb658f8-zvqw4\" (UID: \"e321c1b1-e9e2-46fc-9a35-14a3d6bed233\") " pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" Oct 30 21:50:26.011447 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.011378 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7nrk\" (UniqueName: \"kubernetes.io/projected/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-kube-api-access-q7nrk\") pod \"metrics-server-d6fb658f8-zvqw4\" (UID: \"e321c1b1-e9e2-46fc-9a35-14a3d6bed233\") " pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" Oct 30 21:50:26.093257 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.093192 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" Oct 30 21:50:26.101349 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.101324 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-dd5t2"] Oct 30 21:50:26.104812 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:50:26.104780 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod290eb94c_6ae5_40e8_8ba3_dbe017243d9d.slice/crio-06ccadcde28a65df7d812878f515cd5553b31b449a576308afe8ed61d2926346 WatchSource:0}: Error finding container 06ccadcde28a65df7d812878f515cd5553b31b449a576308afe8ed61d2926346: Status 404 returned error can't find the container with id 06ccadcde28a65df7d812878f515cd5553b31b449a576308afe8ed61d2926346 Oct 30 21:50:26.478321 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:50:26.478280 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode321c1b1_e9e2_46fc_9a35_14a3d6bed233.slice/crio-a603d78eeec3bba0ac01e9d232d39fca7877b42828bd2138a4a3b4e06fbbcf23 WatchSource:0}: Error finding container a603d78eeec3bba0ac01e9d232d39fca7877b42828bd2138a4a3b4e06fbbcf23: Status 404 returned error can't find the container with id a603d78eeec3bba0ac01e9d232d39fca7877b42828bd2138a4a3b4e06fbbcf23 Oct 30 21:50:26.493150 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.493078 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-d6fb658f8-zvqw4"] Oct 30 21:50:26.496128 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.496100 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-89884678b-sxtjs"] Oct 30 21:50:26.499655 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.498902 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:26.518570 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.518488 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" event={"ID":"e321c1b1-e9e2-46fc-9a35-14a3d6bed233","Type":"ContainerStarted","Data":"a603d78eeec3bba0ac01e9d232d39fca7877b42828bd2138a4a3b4e06fbbcf23"} Oct 30 21:50:26.520476 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.520396 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dd5t2" event={"ID":"290eb94c-6ae5-40e8-8ba3-dbe017243d9d","Type":"ContainerStarted","Data":"06ccadcde28a65df7d812878f515cd5553b31b449a576308afe8ed61d2926346"} Oct 30 21:50:26.552516 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.552459 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 21:50:26.552516 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.552476 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 21:50:26.552516 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.552458 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 21:50:26.552992 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.552972 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a4d3550-acb2-4cf1-b192-adf694522695-serving-certs-ca-bundle\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:26.553054 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.553027 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8a4d3550-acb2-4cf1-b192-adf694522695-telemeter-client-tls\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:26.553130 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.553115 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a4d3550-acb2-4cf1-b192-adf694522695-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:26.553182 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.553140 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a4d3550-acb2-4cf1-b192-adf694522695-metrics-client-ca\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:26.553273 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.553249 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a4d3550-acb2-4cf1-b192-adf694522695-telemeter-trusted-ca-bundle\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:26.553336 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.553287 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k48m\" (UniqueName: \"kubernetes.io/projected/8a4d3550-acb2-4cf1-b192-adf694522695-kube-api-access-5k48m\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:26.553388 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.553368 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8a4d3550-acb2-4cf1-b192-adf694522695-federate-client-tls\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:26.553436 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.553403 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8a4d3550-acb2-4cf1-b192-adf694522695-secret-telemeter-client\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:26.602757 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.602731 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 21:50:26.602906 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.602831 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 21:50:26.602906 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.602895 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 21:50:26.603020 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.602980 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-bmzgm\"" Oct 30 21:50:26.604430 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.604407 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-89884678b-sxtjs"] Oct 30 21:50:26.654364 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.654326 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a4d3550-acb2-4cf1-b192-adf694522695-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:26.654547 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.654380 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a4d3550-acb2-4cf1-b192-adf694522695-metrics-client-ca\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:26.654547 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.654409 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a4d3550-acb2-4cf1-b192-adf694522695-telemeter-trusted-ca-bundle\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:26.654547 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.654435 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k48m\" (UniqueName: \"kubernetes.io/projected/8a4d3550-acb2-4cf1-b192-adf694522695-kube-api-access-5k48m\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:26.654547 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.654479 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8a4d3550-acb2-4cf1-b192-adf694522695-federate-client-tls\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:26.654547 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.654515 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8a4d3550-acb2-4cf1-b192-adf694522695-secret-telemeter-client\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:26.654807 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.654593 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a4d3550-acb2-4cf1-b192-adf694522695-serving-certs-ca-bundle\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:26.654807 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.654657 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8a4d3550-acb2-4cf1-b192-adf694522695-telemeter-client-tls\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:26.655378 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.655350 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a4d3550-acb2-4cf1-b192-adf694522695-metrics-client-ca\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:26.656552 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.655684 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a4d3550-acb2-4cf1-b192-adf694522695-telemeter-trusted-ca-bundle\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:26.656552 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.656479 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a4d3550-acb2-4cf1-b192-adf694522695-serving-certs-ca-bundle\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:26.657644 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.657615 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a4d3550-acb2-4cf1-b192-adf694522695-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:26.657959 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.657934 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8a4d3550-acb2-4cf1-b192-adf694522695-telemeter-client-tls\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:26.658204 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.658177 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8a4d3550-acb2-4cf1-b192-adf694522695-secret-telemeter-client\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:26.658901 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.658862 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8a4d3550-acb2-4cf1-b192-adf694522695-federate-client-tls\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:26.717033 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.717003 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k48m\" (UniqueName: \"kubernetes.io/projected/8a4d3550-acb2-4cf1-b192-adf694522695-kube-api-access-5k48m\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:26.820128 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:26.820104 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:27.257100 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:27.257035 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-89884678b-sxtjs"] Oct 30 21:50:27.259778 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:50:27.259742 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a4d3550_acb2_4cf1_b192_adf694522695.slice/crio-032ce38740007e36580833395e1f6cb5af67f15993db39dec68faf5e16309b8c WatchSource:0}: Error finding container 032ce38740007e36580833395e1f6cb5af67f15993db39dec68faf5e16309b8c: Status 404 returned error can't find the container with id 032ce38740007e36580833395e1f6cb5af67f15993db39dec68faf5e16309b8c Oct 30 21:50:27.527899 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:27.527815 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" event={"ID":"f4925cf0-04b5-4ca4-b284-c0355e63b4ca","Type":"ContainerStarted","Data":"cdd16fd4bcf67cb874962d0d7111facd7bf5813044352e76906d5c39803c4206"} Oct 30 21:50:27.528303 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:27.527903 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" event={"ID":"f4925cf0-04b5-4ca4-b284-c0355e63b4ca","Type":"ContainerStarted","Data":"1c7c9620e02a19af273298ca8215cb9dd0ee8794801a3f539d12fe1d451672ed"} Oct 30 21:50:27.528303 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:27.527944 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" event={"ID":"f4925cf0-04b5-4ca4-b284-c0355e63b4ca","Type":"ContainerStarted","Data":"70738f1f70a6f00a981f86c1b303a961683f5146be5dc4c65a4c806504ab1c69"} Oct 30 21:50:27.530244 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:27.530198 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" event={"ID":"8a4d3550-acb2-4cf1-b192-adf694522695","Type":"ContainerStarted","Data":"032ce38740007e36580833395e1f6cb5af67f15993db39dec68faf5e16309b8c"} Oct 30 21:50:29.548888 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:29.548845 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735","Type":"ContainerStarted","Data":"37b4de03de908b697a0b88f5f516f1218143ae6f9cf5322c8245d4d3ab69948a"} Oct 30 21:50:29.549502 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:29.548986 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735","Type":"ContainerStarted","Data":"6834d44b6c689c55ee085cf407a92ddb84232029fb7ed209e68358efa2240328"} Oct 30 21:50:29.549502 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:29.549041 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735","Type":"ContainerStarted","Data":"5105322e8d9ba68c630906594410279a6ffa306bdf34adbcbf9d72aa63c2d5a2"} Oct 30 21:50:29.549502 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:29.549091 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735","Type":"ContainerStarted","Data":"235067a08860012a63a021cead0fdc6bc25c7fe1b7fe749b46b504ae96a7e6cd"} Oct 30 21:50:29.549502 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:29.549134 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735","Type":"ContainerStarted","Data":"080d93f93f7820f56af52a80a4e3d83c84dac6d82b71772824d68d0457d5bef7"} Oct 30 21:50:29.549502 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:29.549178 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735","Type":"ContainerStarted","Data":"ce58dbf55668090f8fd32f2003f91b73cf879f6b4b2c2073b1f40f9c401d6978"} Oct 30 21:50:29.551693 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:29.551659 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" event={"ID":"e321c1b1-e9e2-46fc-9a35-14a3d6bed233","Type":"ContainerStarted","Data":"a63c59c48af9d9696b8ac82c020f45730659163d6b78160beb39494c1ba9411c"} Oct 30 21:50:29.553794 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:29.553765 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dd5t2" event={"ID":"290eb94c-6ae5-40e8-8ba3-dbe017243d9d","Type":"ContainerStarted","Data":"48b7ced4de7a5a034a9d4dff31c2091642732554e8905e0246e02a7c267f741c"} Oct 30 21:50:29.553970 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:29.553950 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dd5t2" Oct 30 21:50:29.556988 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:29.556967 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" event={"ID":"f4925cf0-04b5-4ca4-b284-c0355e63b4ca","Type":"ContainerStarted","Data":"b0af6da087b36a47601a2910a433296d707c1e03d58cbd3ea98e7260ccc52678"} Oct 30 21:50:29.557066 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:29.557013 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" event={"ID":"f4925cf0-04b5-4ca4-b284-c0355e63b4ca","Type":"ContainerStarted","Data":"aa1a3b987fd3b593941fe90088398c3f1f30e0da82230d48e9ca78c2c7ee2683"} Oct 30 21:50:29.557066 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:29.557046 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" event={"ID":"f4925cf0-04b5-4ca4-b284-c0355e63b4ca","Type":"ContainerStarted","Data":"f2109840d822d5d0cf4b2ff6be7448cf4931f8b7259a063e9a7af2c675cc48ae"} Oct 30 21:50:29.557134 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:29.557090 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:29.558741 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:29.558723 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dd5t2" Oct 30 21:50:29.761039 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:29.760992 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.0768678879999998 podStartE2EDuration="6.760979266s" totalImagesPullingTime="4.672514837s" totalInitContainerRuntime="11.596541ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:29.738771933 +0000 UTC m=+62.014141238" watchObservedRunningTime="2026-10-30 21:50:29.760979266 +0000 UTC m=+62.036348546" Oct 30 21:50:29.856273 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:29.856152 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dd5t2" podStartSLOduration=2.513852144 podStartE2EDuration="4.856137485s" totalImagesPullingTime="2.342285341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:29.855760398 +0000 UTC m=+62.131129707" watchObservedRunningTime="2026-10-30 21:50:29.856137485 +0000 UTC m=+62.131506767" Oct 30 21:50:29.976010 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:29.975956 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" podStartSLOduration=2.266967254 podStartE2EDuration="4.975939854s" totalImagesPullingTime="2.7089726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:29.974908448 +0000 UTC m=+62.250277740" watchObservedRunningTime="2026-10-30 21:50:29.975939854 +0000 UTC m=+62.251309135" Oct 30 21:50:30.100580 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.100516 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" podStartSLOduration=3.077868985 podStartE2EDuration="7.100500457s" totalImagesPullingTime="4.022631472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:30.099986464 +0000 UTC m=+62.375355762" watchObservedRunningTime="2026-10-30 21:50:30.100500457 +0000 UTC m=+62.375869738" Oct 30 21:50:30.242394 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.242360 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:50:30.246703 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.246684 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.286876 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.286851 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.287037 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.286887 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.287037 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.286918 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.287037 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.287017 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.287183 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.287052 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.287183 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.287074 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.287183 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.287173 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/736951e5-1596-4606-94d9-80d8b50d35c3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.287330 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.287231 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/736951e5-1596-4606-94d9-80d8b50d35c3-config-out\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.287330 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.287261 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.287384 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.287324 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.287415 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.287391 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.287464 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.287427 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.287464 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.287457 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.287774 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.287580 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.287774 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.287649 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.287774 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.287693 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-config\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.287774 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.287723 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.287774 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.287747 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nd2d\" (UniqueName: \"kubernetes.io/projected/736951e5-1596-4606-94d9-80d8b50d35c3-kube-api-access-6nd2d\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.287982 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.287821 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.287982 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.287854 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-web-config\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.300718 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.300696 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 21:50:30.300831 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.300696 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 21:50:30.300831 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.300760 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 21:50:30.300831 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.300768 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 21:50:30.300831 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.300778 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 21:50:30.301017 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.300831 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 21:50:30.301017 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.300869 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 21:50:30.301017 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.300905 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 21:50:30.301017 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.300923 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 21:50:30.301017 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.300926 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 21:50:30.301449 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.301426 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 21:50:30.301449 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.301429 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9jiopb23p179d\"" Oct 30 21:50:30.348848 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.348823 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-t5jsh\"" Oct 30 21:50:30.349099 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.349079 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 21:50:30.388569 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.388538 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.388703 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.388586 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-config\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.388703 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.388605 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.388703 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.388631 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nd2d\" (UniqueName: \"kubernetes.io/projected/736951e5-1596-4606-94d9-80d8b50d35c3-kube-api-access-6nd2d\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.388703 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.388655 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.388703 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.388672 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-web-config\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.389008 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.388985 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.389070 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.389042 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.389125 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.389104 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.389174 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.389135 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.389174 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.389162 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.389316 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.389193 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.389367 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.389314 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/736951e5-1596-4606-94d9-80d8b50d35c3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.389428 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.389371 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/736951e5-1596-4606-94d9-80d8b50d35c3-config-out\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.389428 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.389405 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.389518 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.389436 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.389518 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.389487 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.389518 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.389490 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.389654 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.389517 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.389654 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.389553 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.389654 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.389613 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.390490 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.390465 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.391181 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.391146 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.391441 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.391424 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.392075 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.391984 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.392522 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.392453 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.392522 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.392501 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.393081 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.393055 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.394048 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.393693 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.394048 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.393970 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.394048 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.394004 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-config\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.394282 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.394261 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.394924 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.394906 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.395435 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.395402 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-web-config\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.395543 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.395454 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.395543 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.395518 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/736951e5-1596-4606-94d9-80d8b50d35c3-config-out\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.395735 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.395712 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/736951e5-1596-4606-94d9-80d8b50d35c3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.398493 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.398470 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 21:50:30.398606 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.398578 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 21:50:30.400500 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.400471 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:50:30.402952 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.402886 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.405243 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.404872 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.451045 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.451007 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nd2d\" (UniqueName: \"kubernetes.io/projected/736951e5-1596-4606-94d9-80d8b50d35c3-kube-api-access-6nd2d\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:30.573754 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:30.573675 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:31.068259 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:31.068231 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:50:31.120747 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:50:31.120720 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod736951e5_1596_4606_94d9_80d8b50d35c3.slice/crio-f9fae51e3797f359f826f7d5c8436433cd1dae5fa3210c0f63881d21e1a27259 WatchSource:0}: Error finding container f9fae51e3797f359f826f7d5c8436433cd1dae5fa3210c0f63881d21e1a27259: Status 404 returned error can't find the container with id f9fae51e3797f359f826f7d5c8436433cd1dae5fa3210c0f63881d21e1a27259 Oct 30 21:50:31.477916 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:31.477838 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zf4n4" Oct 30 21:50:31.564864 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:31.564832 2576 generic.go:356] "Generic (PLEG): container finished" podID="736951e5-1596-4606-94d9-80d8b50d35c3" containerID="6ecd79a7c6d356f8e64253560432de55058bb4c2da2408ddf9e73b11af211256" exitCode=0 Oct 30 21:50:31.565016 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:31.564922 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"736951e5-1596-4606-94d9-80d8b50d35c3","Type":"ContainerDied","Data":"6ecd79a7c6d356f8e64253560432de55058bb4c2da2408ddf9e73b11af211256"} Oct 30 21:50:31.565016 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:31.564984 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"736951e5-1596-4606-94d9-80d8b50d35c3","Type":"ContainerStarted","Data":"f9fae51e3797f359f826f7d5c8436433cd1dae5fa3210c0f63881d21e1a27259"} Oct 30 21:50:31.567161 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:31.567139 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" event={"ID":"8a4d3550-acb2-4cf1-b192-adf694522695","Type":"ContainerStarted","Data":"6520450bf6c5867426116153cb4d1d875cdb3467eb96945ef15d2e0047dcad8e"} Oct 30 21:50:31.567265 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:31.567198 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" event={"ID":"8a4d3550-acb2-4cf1-b192-adf694522695","Type":"ContainerStarted","Data":"27dcb2414dbb0ac5a04a9ec234cee1ae661f855e196add64c9cc40a2d084bffa"} Oct 30 21:50:31.567265 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:31.567254 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" event={"ID":"8a4d3550-acb2-4cf1-b192-adf694522695","Type":"ContainerStarted","Data":"d68b47392ea677c7688361ef4faea04f3fc4109d247f10a97646275b8d752015"} Oct 30 21:50:31.719847 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:31.719783 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" podStartSLOduration=2.06201423 podStartE2EDuration="5.719765357s" totalImagesPullingTime="3.657751127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:31.719244011 +0000 UTC m=+63.994613316" watchObservedRunningTime="2026-10-30 21:50:31.719765357 +0000 UTC m=+63.995134640" Oct 30 21:50:33.218291 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:33.218251 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a9baf0-064d-4544-94b4-a0dd090dbd04-metrics-certs\") pod \"network-metrics-daemon-h4v24\" (UID: \"83a9baf0-064d-4544-94b4-a0dd090dbd04\") " pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:50:33.218779 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:33.218381 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4630b412-da87-49f7-bbbb-ba3dc820ddd1-original-pull-secret\") pod \"global-pull-secret-syncer-xbwzg\" (UID: \"4630b412-da87-49f7-bbbb-ba3dc820ddd1\") " pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:50:33.269460 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:33.269427 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:50:33.269460 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:33.269428 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:50:33.281270 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:33.281241 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a9baf0-064d-4544-94b4-a0dd090dbd04-metrics-certs\") pod \"network-metrics-daemon-h4v24\" (UID: \"83a9baf0-064d-4544-94b4-a0dd090dbd04\") " pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:50:33.281614 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:33.281594 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4630b412-da87-49f7-bbbb-ba3dc820ddd1-original-pull-secret\") pod \"global-pull-secret-syncer-xbwzg\" (UID: \"4630b412-da87-49f7-bbbb-ba3dc820ddd1\") " pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:50:33.318865 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:33.318832 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj8hr\" (UniqueName: \"kubernetes.io/projected/2e5352ac-8295-494d-950a-16c09b09272e-kube-api-access-dj8hr\") pod \"network-check-target-mg6qm\" (UID: \"2e5352ac-8295-494d-950a-16c09b09272e\") " pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:50:33.370013 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:33.369985 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:50:33.430158 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:33.430120 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:50:33.442932 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:33.442898 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj8hr\" (UniqueName: \"kubernetes.io/projected/2e5352ac-8295-494d-950a-16c09b09272e-kube-api-access-dj8hr\") pod \"network-check-target-mg6qm\" (UID: \"2e5352ac-8295-494d-950a-16c09b09272e\") " pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:50:33.572034 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:33.571961 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jjx9q\"" Oct 30 21:50:33.577935 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:33.577903 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-gmxqg\"" Oct 30 21:50:33.581991 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:33.581972 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:50:33.583265 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:33.583247 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r6nvz\"" Oct 30 21:50:33.588074 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:33.588057 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:50:33.593008 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:33.592987 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:50:33.965429 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:33.965395 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mg6qm"] Oct 30 21:50:34.006110 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:50:34.006069 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e5352ac_8295_494d_950a_16c09b09272e.slice/crio-c9fd11b3c3ba987ba67e1f854aba8a3c1dc65f84d19575984dbff4e5d0cec644 WatchSource:0}: Error finding container c9fd11b3c3ba987ba67e1f854aba8a3c1dc65f84d19575984dbff4e5d0cec644: Status 404 returned error can't find the container with id c9fd11b3c3ba987ba67e1f854aba8a3c1dc65f84d19575984dbff4e5d0cec644 Oct 30 21:50:34.017153 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:34.017129 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h4v24"] Oct 30 21:50:34.019977 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:34.019935 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xbwzg"] Oct 30 21:50:34.020041 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:50:34.020004 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a9baf0_064d_4544_94b4_a0dd090dbd04.slice/crio-928f9029a8d5765f6d966c38c9db05323be5c91e8183a8819bb902cd6a3e1658 WatchSource:0}: Error finding container 928f9029a8d5765f6d966c38c9db05323be5c91e8183a8819bb902cd6a3e1658: Status 404 returned error can't find the container with id 928f9029a8d5765f6d966c38c9db05323be5c91e8183a8819bb902cd6a3e1658 Oct 30 21:50:34.022719 ip-10-0-137-81 kubenswrapper[2576]: W1030 21:50:34.022684 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4630b412_da87_49f7_bbbb_ba3dc820ddd1.slice/crio-00df8434490defdffffbc4706d440e2f00ab97169b574c117a9fc6d6f1aec695 WatchSource:0}: Error finding container 00df8434490defdffffbc4706d440e2f00ab97169b574c117a9fc6d6f1aec695: Status 404 returned error can't find the container with id 00df8434490defdffffbc4706d440e2f00ab97169b574c117a9fc6d6f1aec695 Oct 30 21:50:34.580043 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:34.580005 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h4v24" event={"ID":"83a9baf0-064d-4544-94b4-a0dd090dbd04","Type":"ContainerStarted","Data":"928f9029a8d5765f6d966c38c9db05323be5c91e8183a8819bb902cd6a3e1658"} Oct 30 21:50:34.581529 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:34.581502 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mg6qm" event={"ID":"2e5352ac-8295-494d-950a-16c09b09272e","Type":"ContainerStarted","Data":"c9fd11b3c3ba987ba67e1f854aba8a3c1dc65f84d19575984dbff4e5d0cec644"} Oct 30 21:50:34.582901 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:34.582873 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xbwzg" event={"ID":"4630b412-da87-49f7-bbbb-ba3dc820ddd1","Type":"ContainerStarted","Data":"00df8434490defdffffbc4706d440e2f00ab97169b574c117a9fc6d6f1aec695"} Oct 30 21:50:35.568013 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:35.567920 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:35.596108 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:35.594902 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"736951e5-1596-4606-94d9-80d8b50d35c3","Type":"ContainerStarted","Data":"6e673397a1f3c2825a68224eb4b922c2f0a5f03c6051241bba2c74b99abb1f6d"} Oct 30 21:50:35.596108 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:35.594991 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"736951e5-1596-4606-94d9-80d8b50d35c3","Type":"ContainerStarted","Data":"b1ddaa6943a6a9fa826e141df1d6d73be3a699d6cdbd5c118deb80cd2f007ee3"} Oct 30 21:50:35.596108 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:35.595036 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"736951e5-1596-4606-94d9-80d8b50d35c3","Type":"ContainerStarted","Data":"4d7dc66cbc297b493c2282a3d67bdb66bdeabbe7484d47c4b8d8c6c5ca9d70f1"} Oct 30 21:50:35.596108 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:35.595076 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"736951e5-1596-4606-94d9-80d8b50d35c3","Type":"ContainerStarted","Data":"c436cf179edf23782012024b058c39f01184ff419de08fd9c33ba0cf17321ed0"} Oct 30 21:50:35.596108 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:35.595112 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"736951e5-1596-4606-94d9-80d8b50d35c3","Type":"ContainerStarted","Data":"7f6a9182c1c60364249dd1d140565cfb4a5f1960e32a8bf97d90413b5212beb1"} Oct 30 21:50:35.596108 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:35.595148 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"736951e5-1596-4606-94d9-80d8b50d35c3","Type":"ContainerStarted","Data":"03bf42adb14c6eafce0516556870e7dd844847b9277150c44db7514cc4eb96cd"} Oct 30 21:50:35.878477 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:35.878407 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.410386588 podStartE2EDuration="5.878386149s" totalImagesPullingTime="3.449398074s" totalInitContainerRuntime="18.601487ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:35.877003525 +0000 UTC m=+68.152372867" watchObservedRunningTime="2026-10-30 21:50:35.878386149 +0000 UTC m=+68.153755433" Oct 30 21:50:36.603513 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:36.602945 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h4v24" event={"ID":"83a9baf0-064d-4544-94b4-a0dd090dbd04","Type":"ContainerStarted","Data":"9254103961ae1ecd958658d0a3da2074d752950fde6873245e1050f4742af8bc"} Oct 30 21:50:36.603513 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:36.603017 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h4v24" event={"ID":"83a9baf0-064d-4544-94b4-a0dd090dbd04","Type":"ContainerStarted","Data":"639e801b075fc0052d6677c481ad23728f1fbb900a9f3c70ad46f0d7a1f31444"} Oct 30 21:50:38.614748 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:38.614709 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mg6qm" event={"ID":"2e5352ac-8295-494d-950a-16c09b09272e","Type":"ContainerStarted","Data":"2a5915df8569954f00560d25bfb753270988ae90ff73288b88b0fbb3c6494539"} Oct 30 21:50:38.615179 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:38.614851 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:50:38.732649 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:38.732594 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-h4v24" podStartSLOduration=68.910596669 podStartE2EDuration="1m10.732577483s" totalImagesPullingTime="1.821980814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:36.722868365 +0000 UTC m=+68.998237665" watchObservedRunningTime="2026-10-30 21:50:38.732577483 +0000 UTC m=+71.007946799" Oct 30 21:50:39.619965 ip-10-0-137-81 kubenswrapper[2576]: I1030 21:50:39.619934 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xbwzg" event={"ID":"4630b412-da87-49f7-bbbb-ba3dc820ddd1","Type":"ContainerStarted","Data":"894a0354c843c313cd4445fcceadb69501d15e327722943d87360a98a84eff85"} Oct 30 21:50:39.671370 ip-10-0-137-81 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 21:50:39.686697 ip-10-0-137-81 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 21:50:39.686905 ip-10-0-137-81 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 21:50:39.686967 ip-10-0-137-81 systemd[1]: kubelet.service: Consumed 4.605s CPU time. Oct 30 21:50:39.694963 ip-10-0-137-81 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:50:39.759803 ip-10-0-137-81 kubenswrapper[5918]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:50:39.759803 ip-10-0-137-81 kubenswrapper[5918]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:50:39.759803 ip-10-0-137-81 kubenswrapper[5918]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:50:39.759803 ip-10-0-137-81 kubenswrapper[5918]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:50:39.761962 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.761884 5918 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:50:39.761962 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.761962 5918 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:50:39.762029 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.761966 5918 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:50:39.762029 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.761970 5918 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:50:39.762029 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.761973 5918 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:50:39.762029 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.761976 5918 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:50:39.762029 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.761980 5918 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:50:39.762029 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.761982 5918 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:50:39.762029 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.761985 5918 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:50:39.762029 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.761988 5918 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:50:39.762029 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.761990 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:50:39.762029 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.761993 5918 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:50:39.762029 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.761995 5918 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:50:39.762029 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.761998 5918 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:50:39.762029 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762001 5918 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:50:39.762029 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762004 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:50:39.762029 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762007 5918 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:50:39.762029 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762010 5918 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:50:39.762029 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762013 5918 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:50:39.762029 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762015 5918 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:50:39.762029 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762018 5918 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:50:39.762527 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762021 5918 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:50:39.762527 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762023 5918 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:50:39.762527 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762026 5918 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:50:39.762527 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762028 5918 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:50:39.762527 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762031 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:50:39.762527 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762034 5918 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:50:39.762527 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762040 5918 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:50:39.762527 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762043 5918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:50:39.762527 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762046 5918 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:50:39.762527 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762048 5918 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:50:39.762527 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762051 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:50:39.762527 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762053 5918 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:50:39.762527 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762056 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:50:39.762527 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762058 5918 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:50:39.762527 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762061 5918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:50:39.762527 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762064 5918 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:50:39.762527 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762068 5918 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:50:39.762527 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762072 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:50:39.762527 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762076 5918 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:50:39.762971 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762079 5918 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:50:39.762971 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762082 5918 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:50:39.762971 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762085 5918 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:50:39.762971 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762087 5918 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:50:39.762971 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762090 5918 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:50:39.762971 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762092 5918 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:50:39.762971 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762095 5918 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:50:39.762971 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762098 5918 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:50:39.762971 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762100 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:50:39.762971 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762103 5918 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:50:39.762971 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762106 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:50:39.762971 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762108 5918 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:50:39.762971 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762111 5918 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:50:39.762971 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762113 5918 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:50:39.762971 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762116 5918 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:50:39.762971 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762119 5918 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:50:39.762971 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762122 5918 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:50:39.762971 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762124 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:50:39.762971 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762127 5918 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:50:39.762971 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762130 5918 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:50:39.763460 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762132 5918 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:50:39.763460 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762135 5918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:50:39.763460 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762138 5918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:50:39.763460 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762141 5918 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:50:39.763460 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762143 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:50:39.763460 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762146 5918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:50:39.763460 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762148 5918 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:50:39.763460 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762151 5918 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:50:39.763460 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762153 5918 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:50:39.763460 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762157 5918 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:50:39.763460 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762161 5918 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:50:39.763460 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762164 5918 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:50:39.763460 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762167 5918 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:50:39.763460 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762170 5918 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:50:39.763460 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762172 5918 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:50:39.763460 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762174 5918 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:50:39.763460 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762177 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:50:39.763460 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762180 5918 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:50:39.763460 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762182 5918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:50:39.763939 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762185 5918 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:50:39.763939 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762187 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:50:39.763939 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762189 5918 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:50:39.763939 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762192 5918 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:50:39.763939 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762195 5918 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:50:39.763939 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762197 5918 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:50:39.763939 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762200 5918 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:50:39.763939 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762203 5918 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:50:39.763939 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762206 5918 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:50:39.763939 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762208 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:50:39.763939 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762243 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:50:39.763939 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762246 5918 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:50:39.763939 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762249 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:50:39.763939 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762253 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:50:39.763939 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762255 5918 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:50:39.763939 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762258 5918 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:50:39.763939 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762260 5918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:50:39.763939 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762263 5918 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:50:39.763939 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762265 5918 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:50:39.763939 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762268 5918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:50:39.764409 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762270 5918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:50:39.764409 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762273 5918 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:50:39.764409 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762275 5918 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:50:39.764409 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762277 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:50:39.764409 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762280 5918 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:50:39.764409 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762282 5918 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:50:39.764409 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.762285 5918 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:50:39.764409 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763050 5918 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:50:39.764409 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763057 5918 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:50:39.764409 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763060 5918 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:50:39.764409 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763063 5918 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:50:39.764409 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763065 5918 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:50:39.764409 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763068 5918 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:50:39.764409 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763072 5918 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:50:39.764409 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763074 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:50:39.764409 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763078 5918 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:50:39.764409 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763080 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:50:39.764409 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763084 5918 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:50:39.764409 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763088 5918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:50:39.764863 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763091 5918 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:50:39.764863 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763093 5918 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:50:39.764863 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763096 5918 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:50:39.764863 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763099 5918 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:50:39.764863 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763102 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:50:39.764863 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763104 5918 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:50:39.764863 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763106 5918 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:50:39.764863 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763109 5918 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:50:39.764863 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763112 5918 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:50:39.764863 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763115 5918 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:50:39.764863 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763117 5918 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:50:39.764863 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763120 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:50:39.764863 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763122 5918 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:50:39.764863 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763125 5918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:50:39.764863 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763127 5918 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:50:39.764863 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763129 5918 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:50:39.764863 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763132 5918 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:50:39.764863 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763134 5918 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:50:39.764863 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763137 5918 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:50:39.764863 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763139 5918 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:50:39.765347 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763142 5918 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:50:39.765347 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763144 5918 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:50:39.765347 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763147 5918 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:50:39.765347 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763149 5918 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:50:39.765347 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763152 5918 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:50:39.765347 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763155 5918 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:50:39.765347 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763158 5918 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:50:39.765347 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763161 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:50:39.765347 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763163 5918 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:50:39.765347 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763166 5918 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:50:39.765347 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763168 5918 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:50:39.765347 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763171 5918 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:50:39.765347 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763174 5918 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:50:39.765347 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763176 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:50:39.765347 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763179 5918 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:50:39.765347 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763182 5918 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:50:39.765347 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763184 5918 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:50:39.765347 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763187 5918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:50:39.765347 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763189 5918 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:50:39.765347 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763191 5918 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:50:39.765803 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763194 5918 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:50:39.765803 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763197 5918 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:50:39.765803 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763199 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:50:39.765803 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763202 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:50:39.765803 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763205 5918 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:50:39.765803 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763208 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:50:39.765803 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763226 5918 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:50:39.765803 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763229 5918 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:50:39.765803 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763232 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:50:39.765803 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763234 5918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:50:39.765803 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763237 5918 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:50:39.765803 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763239 5918 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:50:39.765803 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763242 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:50:39.765803 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763244 5918 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:50:39.765803 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763247 5918 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:50:39.765803 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763250 5918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:50:39.765803 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763253 5918 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:50:39.765803 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763257 5918 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:50:39.765803 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763261 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:50:39.766246 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763265 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:50:39.766246 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763268 5918 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:50:39.766246 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763270 5918 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:50:39.766246 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763273 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:50:39.766246 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763276 5918 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:50:39.766246 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763279 5918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:50:39.766246 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763282 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:50:39.766246 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763285 5918 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:50:39.766246 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763287 5918 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:50:39.766246 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763290 5918 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:50:39.766246 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763293 5918 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:50:39.766246 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763295 5918 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:50:39.766246 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763302 5918 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:50:39.766246 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763305 5918 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:50:39.766246 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763308 5918 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:50:39.766246 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763311 5918 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:50:39.766246 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763313 5918 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:50:39.766246 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763316 5918 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:50:39.766246 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763318 5918 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:50:39.766246 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763321 5918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:50:39.766724 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763324 5918 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:50:39.766724 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763326 5918 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:50:39.766724 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763329 5918 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:50:39.766724 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763331 5918 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:50:39.766724 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763334 5918 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:50:39.766724 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763336 5918 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:50:39.766724 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763339 5918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:50:39.766724 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763342 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:50:39.766724 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763344 5918 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:50:39.766724 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763347 5918 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:50:39.766724 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763349 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:50:39.766724 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763353 5918 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:50:39.766724 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763355 5918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:50:39.766724 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763358 5918 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:50:39.766724 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763363 5918 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:50:39.766724 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763845 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:50:39.766724 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763850 5918 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:50:39.766724 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763853 5918 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:50:39.766724 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763856 5918 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:50:39.767157 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763859 5918 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:50:39.767157 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763861 5918 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:50:39.767157 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763864 5918 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:50:39.767157 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763866 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:50:39.767157 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763869 5918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:50:39.767157 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763873 5918 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:50:39.767157 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763876 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:50:39.767157 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763880 5918 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:50:39.767157 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763882 5918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:50:39.767157 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763885 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:50:39.767157 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763887 5918 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:50:39.767157 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763890 5918 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:50:39.767157 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763893 5918 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:50:39.767157 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763896 5918 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:50:39.767157 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763898 5918 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:50:39.767157 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763901 5918 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:50:39.767157 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763903 5918 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:50:39.767157 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763906 5918 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:50:39.767157 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763910 5918 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:50:39.767613 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763914 5918 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:50:39.767613 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763917 5918 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:50:39.767613 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763920 5918 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:50:39.767613 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763924 5918 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:50:39.767613 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763928 5918 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:50:39.767613 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763930 5918 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:50:39.767613 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763933 5918 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:50:39.767613 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763936 5918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:50:39.767613 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763940 5918 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:50:39.767613 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763943 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:50:39.767613 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763946 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:50:39.767613 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763948 5918 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:50:39.767613 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763951 5918 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:50:39.767613 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763954 5918 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:50:39.767613 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763957 5918 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:50:39.767613 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763960 5918 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:50:39.767613 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763962 5918 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:50:39.767613 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763965 5918 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:50:39.767613 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763969 5918 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:50:39.768054 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763971 5918 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:50:39.768054 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763974 5918 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:50:39.768054 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763977 5918 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:50:39.768054 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763980 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:50:39.768054 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763982 5918 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:50:39.768054 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763985 5918 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:50:39.768054 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763987 5918 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:50:39.768054 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763989 5918 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:50:39.768054 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763992 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:50:39.768054 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763994 5918 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:50:39.768054 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763997 5918 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:50:39.768054 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.763999 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:50:39.768054 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764002 5918 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:50:39.768054 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764004 5918 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:50:39.768054 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764006 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:50:39.768054 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764009 5918 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:50:39.768054 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764011 5918 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:50:39.768054 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764014 5918 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:50:39.768054 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764016 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:50:39.768054 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764018 5918 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:50:39.768533 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764021 5918 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:50:39.768533 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764025 5918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:50:39.768533 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764028 5918 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:50:39.768533 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764030 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:50:39.768533 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764033 5918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:50:39.768533 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764035 5918 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:50:39.768533 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764038 5918 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:50:39.768533 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764041 5918 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:50:39.768533 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764043 5918 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:50:39.768533 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764046 5918 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:50:39.768533 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764048 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:50:39.768533 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764052 5918 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:50:39.768533 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764055 5918 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:50:39.768533 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764057 5918 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:50:39.768533 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764059 5918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:50:39.768533 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764065 5918 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:50:39.768533 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764068 5918 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:50:39.768533 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764070 5918 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:50:39.768533 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764073 5918 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:50:39.768533 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764075 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:50:39.769020 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764078 5918 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:50:39.769020 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764080 5918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:50:39.769020 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764083 5918 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:50:39.769020 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764085 5918 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:50:39.769020 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764087 5918 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:50:39.769020 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764090 5918 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:50:39.769020 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764092 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:50:39.769020 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764094 5918 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:50:39.769020 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764097 5918 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:50:39.769020 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764099 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:50:39.769020 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764102 5918 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:50:39.769020 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764104 5918 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:50:39.769020 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764107 5918 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:50:39.769020 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764112 5918 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:50:39.769020 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764115 5918 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:50:39.769020 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764117 5918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:50:39.769020 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764120 5918 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:50:39.769020 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764122 5918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:50:39.769020 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764125 5918 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:50:39.769473 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764127 5918 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:50:39.769473 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764130 5918 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:50:39.769473 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764132 5918 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:50:39.769473 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764135 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:50:39.769473 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764138 5918 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:50:39.769473 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764269 5918 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:50:39.769473 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764277 5918 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:50:39.769473 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764283 5918 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:50:39.769473 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764287 5918 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:50:39.769473 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764299 5918 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:50:39.769473 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764302 5918 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:50:39.769473 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764307 5918 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:50:39.769473 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764311 5918 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:50:39.769473 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764314 5918 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:50:39.769473 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764317 5918 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:50:39.769473 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764320 5918 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:50:39.769473 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764324 5918 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:50:39.769473 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764327 5918 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:50:39.769473 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764329 5918 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:50:39.769473 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764332 5918 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:50:39.769473 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764335 5918 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:50:39.769473 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764338 5918 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:50:39.769473 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764341 5918 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:50:39.770065 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764348 5918 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:50:39.770065 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764351 5918 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:50:39.770065 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764354 5918 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:50:39.770065 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764359 5918 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:50:39.770065 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764362 5918 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:50:39.770065 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764366 5918 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:50:39.770065 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764369 5918 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:50:39.770065 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764372 5918 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:50:39.770065 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764375 5918 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:50:39.770065 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764378 5918 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:50:39.770065 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764381 5918 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:50:39.770065 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764384 5918 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:50:39.770065 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764387 5918 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:50:39.770065 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764391 5918 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:50:39.770065 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764395 5918 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:50:39.770065 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764398 5918 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:50:39.770065 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764401 5918 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:50:39.770065 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764404 5918 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:50:39.770065 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764406 5918 flags.go:64] FLAG: --enable-server="true" Oct 30 21:50:39.770065 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764415 5918 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:50:39.770065 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764422 5918 flags.go:64] FLAG: --event-burst="100" Oct 30 21:50:39.770065 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764425 5918 flags.go:64] FLAG: --event-qps="50" Oct 30 21:50:39.770065 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764428 5918 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:50:39.770065 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764431 5918 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:50:39.770995 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764433 5918 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:50:39.770995 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764437 5918 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:50:39.770995 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764440 5918 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:50:39.770995 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764443 5918 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:50:39.770995 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764446 5918 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:50:39.770995 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764449 5918 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:50:39.770995 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764452 5918 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:50:39.770995 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764455 5918 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:50:39.770995 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764457 5918 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:50:39.770995 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764460 5918 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:50:39.770995 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764463 5918 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:50:39.770995 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764467 5918 flags.go:64] FLAG: --feature-gates="" Oct 30 21:50:39.770995 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764471 5918 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:50:39.770995 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764474 5918 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:50:39.770995 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764477 5918 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:50:39.770995 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764480 5918 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:50:39.770995 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764483 5918 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:50:39.770995 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764486 5918 flags.go:64] FLAG: --help="false" Oct 30 21:50:39.770995 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764489 5918 flags.go:64] FLAG: --hostname-override="ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:50:39.770995 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764492 5918 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:50:39.770995 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764495 5918 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:50:39.770995 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764499 5918 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:50:39.770995 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764503 5918 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:50:39.771607 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764506 5918 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:50:39.771607 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764509 5918 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:50:39.771607 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764512 5918 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:50:39.771607 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764514 5918 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:50:39.771607 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764517 5918 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:50:39.771607 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764525 5918 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:50:39.771607 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764528 5918 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:50:39.771607 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764531 5918 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:50:39.771607 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764534 5918 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:50:39.771607 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764537 5918 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:50:39.771607 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764540 5918 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:50:39.771607 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764542 5918 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:50:39.771607 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764545 5918 flags.go:64] FLAG: --lock-file="" Oct 30 21:50:39.771607 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764548 5918 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:50:39.771607 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764551 5918 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:50:39.771607 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764554 5918 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:50:39.771607 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764559 5918 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:50:39.771607 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764562 5918 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:50:39.771607 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764565 5918 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:50:39.771607 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764568 5918 flags.go:64] FLAG: --logging-format="text" Oct 30 21:50:39.771607 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764572 5918 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:50:39.771607 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764575 5918 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:50:39.771607 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764578 5918 flags.go:64] FLAG: --manifest-url="" Oct 30 21:50:39.771607 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764581 5918 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:50:39.771607 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764586 5918 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:50:39.772180 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764588 5918 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:50:39.772180 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764592 5918 flags.go:64] FLAG: --max-pods="110" Oct 30 21:50:39.772180 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764595 5918 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:50:39.772180 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764598 5918 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:50:39.772180 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764601 5918 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:50:39.772180 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764605 5918 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:50:39.772180 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764608 5918 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:50:39.772180 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764611 5918 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:50:39.772180 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764614 5918 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:50:39.772180 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764621 5918 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:50:39.772180 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764624 5918 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:50:39.772180 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764627 5918 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:50:39.772180 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764630 5918 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:50:39.772180 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764640 5918 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:50:39.772180 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764643 5918 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:50:39.772180 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764646 5918 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:50:39.772180 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764649 5918 flags.go:64] FLAG: --port="10250" Oct 30 21:50:39.772180 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764652 5918 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:50:39.772180 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764654 5918 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-03fd6ff48ae5bf048" Oct 30 21:50:39.772180 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764658 5918 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:50:39.772180 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764660 5918 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:50:39.772180 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764664 5918 flags.go:64] FLAG: --register-node="true" Oct 30 21:50:39.772180 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764666 5918 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:50:39.772180 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764670 5918 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:50:39.772180 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764673 5918 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:50:39.772180 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764676 5918 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:50:39.772836 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764679 5918 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:50:39.772836 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764685 5918 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:50:39.772836 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764688 5918 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:50:39.772836 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764691 5918 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:50:39.772836 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764693 5918 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:50:39.772836 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764696 5918 flags.go:64] FLAG: --runonce="false" Oct 30 21:50:39.772836 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764699 5918 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:50:39.772836 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764702 5918 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:50:39.772836 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764705 5918 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:50:39.772836 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764708 5918 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:50:39.772836 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764711 5918 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:50:39.772836 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764715 5918 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:50:39.772836 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764718 5918 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:50:39.772836 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764721 5918 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:50:39.772836 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764724 5918 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:50:39.772836 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764726 5918 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:50:39.772836 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764730 5918 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:50:39.772836 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764733 5918 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:50:39.772836 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764736 5918 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:50:39.772836 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764739 5918 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:50:39.772836 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764743 5918 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:50:39.772836 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764748 5918 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:50:39.772836 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764751 5918 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:50:39.772836 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764754 5918 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:50:39.772836 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764759 5918 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:50:39.773568 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764761 5918 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:50:39.773568 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764764 5918 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:50:39.773568 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764767 5918 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:50:39.773568 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764769 5918 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:50:39.773568 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764772 5918 flags.go:64] FLAG: --v="2" Oct 30 21:50:39.773568 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764776 5918 flags.go:64] FLAG: --version="false" Oct 30 21:50:39.773568 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764780 5918 flags.go:64] FLAG: --vmodule="" Oct 30 21:50:39.773568 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764784 5918 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:50:39.773568 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.764789 5918 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:50:39.773568 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764914 5918 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:50:39.773568 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764918 5918 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:50:39.773568 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764921 5918 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:50:39.773568 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764924 5918 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:50:39.773568 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764927 5918 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:50:39.773568 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764930 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:50:39.773568 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764932 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:50:39.773568 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764935 5918 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:50:39.773568 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764937 5918 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:50:39.773568 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764941 5918 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:50:39.773568 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764943 5918 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:50:39.773568 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764946 5918 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:50:39.774068 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764948 5918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:50:39.774068 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764951 5918 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:50:39.774068 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764954 5918 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:50:39.774068 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764956 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:50:39.774068 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764960 5918 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:50:39.774068 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764964 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:50:39.774068 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764967 5918 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:50:39.774068 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764970 5918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:50:39.774068 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764973 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:50:39.774068 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764976 5918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:50:39.774068 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764978 5918 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:50:39.774068 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764981 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:50:39.774068 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764983 5918 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:50:39.774068 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764985 5918 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:50:39.774068 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764988 5918 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:50:39.774068 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764990 5918 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:50:39.774068 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764993 5918 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:50:39.774068 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764995 5918 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:50:39.774068 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.764998 5918 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:50:39.774602 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765001 5918 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:50:39.774602 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765004 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:50:39.774602 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765006 5918 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:50:39.774602 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765009 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:50:39.774602 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765012 5918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:50:39.774602 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765014 5918 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:50:39.774602 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765017 5918 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:50:39.774602 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765019 5918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:50:39.774602 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765021 5918 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:50:39.774602 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765025 5918 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:50:39.774602 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765029 5918 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:50:39.774602 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765032 5918 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:50:39.774602 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765035 5918 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:50:39.774602 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765037 5918 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:50:39.774602 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765040 5918 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:50:39.774602 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765042 5918 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:50:39.774602 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765045 5918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:50:39.774602 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765047 5918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:50:39.774602 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765050 5918 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:50:39.775228 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765053 5918 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:50:39.775228 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765055 5918 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:50:39.775228 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765058 5918 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:50:39.775228 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765061 5918 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:50:39.775228 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765063 5918 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:50:39.775228 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765066 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:50:39.775228 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765068 5918 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:50:39.775228 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765071 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:50:39.775228 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765073 5918 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:50:39.775228 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765075 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:50:39.775228 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765078 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:50:39.775228 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765080 5918 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:50:39.775228 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765083 5918 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:50:39.775228 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765085 5918 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:50:39.775228 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765088 5918 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:50:39.775228 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765090 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:50:39.775228 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765093 5918 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:50:39.775228 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765095 5918 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:50:39.775228 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765097 5918 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:50:39.775228 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765100 5918 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:50:39.775853 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765103 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:50:39.775853 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765105 5918 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:50:39.775853 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765108 5918 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:50:39.775853 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765110 5918 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:50:39.775853 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765113 5918 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:50:39.775853 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765115 5918 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:50:39.775853 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765118 5918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:50:39.775853 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765120 5918 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:50:39.775853 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765123 5918 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:50:39.775853 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765125 5918 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:50:39.775853 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765127 5918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:50:39.775853 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765130 5918 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:50:39.775853 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765132 5918 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:50:39.775853 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765135 5918 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:50:39.775853 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765137 5918 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:50:39.775853 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765140 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:50:39.775853 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765143 5918 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:50:39.775853 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765146 5918 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:50:39.775853 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765148 5918 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:50:39.775853 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765151 5918 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:50:39.776544 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765153 5918 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:50:39.776544 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765156 5918 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:50:39.776544 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765158 5918 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:50:39.776544 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765161 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:50:39.776544 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765163 5918 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:50:39.776544 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765166 5918 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:50:39.776544 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765168 5918 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:50:39.776544 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765170 5918 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:50:39.776544 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765173 5918 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:50:39.776544 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765177 5918 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:50:39.776544 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765180 5918 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:50:39.776544 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765182 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:50:39.776544 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765185 5918 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:50:39.776544 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765187 5918 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:50:39.776544 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765190 5918 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:50:39.776544 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765192 5918 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:50:39.776916 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.765233 5918 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:50:39.776916 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765269 5918 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:50:39.776916 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765274 5918 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:50:39.776916 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765277 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:50:39.776916 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765280 5918 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:50:39.776916 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765283 5918 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:50:39.776916 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765285 5918 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:50:39.776916 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765288 5918 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:50:39.776916 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765290 5918 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:50:39.776916 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765293 5918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:50:39.776916 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765296 5918 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:50:39.776916 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765299 5918 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:50:39.776916 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765302 5918 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:50:39.776916 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765305 5918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:50:39.776916 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765307 5918 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:50:39.776916 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765310 5918 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:50:39.777307 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765313 5918 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:50:39.777307 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765315 5918 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:50:39.777307 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765318 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:50:39.777307 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765321 5918 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:50:39.777307 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765323 5918 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:50:39.777307 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765326 5918 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:50:39.777307 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765328 5918 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:50:39.777307 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765330 5918 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:50:39.777307 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765333 5918 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:50:39.777307 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765335 5918 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:50:39.777307 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765339 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:50:39.777307 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765342 5918 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:50:39.777307 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765345 5918 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:50:39.777307 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765347 5918 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:50:39.777307 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765350 5918 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:50:39.777307 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765352 5918 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:50:39.777307 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765355 5918 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:50:39.777307 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765357 5918 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:50:39.777307 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765360 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:50:39.777307 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765363 5918 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:50:39.777778 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765365 5918 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:50:39.777778 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765368 5918 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:50:39.777778 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765370 5918 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:50:39.777778 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765372 5918 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:50:39.777778 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765375 5918 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:50:39.777778 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765379 5918 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:50:39.777778 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765381 5918 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:50:39.777778 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765384 5918 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:50:39.777778 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765386 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:50:39.777778 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765389 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:50:39.777778 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765392 5918 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:50:39.777778 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765394 5918 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:50:39.777778 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765397 5918 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:50:39.777778 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765399 5918 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:50:39.777778 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765401 5918 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:50:39.777778 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765404 5918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:50:39.777778 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765407 5918 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:50:39.777778 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765409 5918 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:50:39.777778 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765412 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:50:39.778324 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765415 5918 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:50:39.778324 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765419 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:50:39.778324 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765422 5918 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:50:39.778324 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765427 5918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:50:39.778324 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765430 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:50:39.778324 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765433 5918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:50:39.778324 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765435 5918 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:50:39.778324 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765438 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:50:39.778324 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765440 5918 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:50:39.778324 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765443 5918 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:50:39.778324 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765445 5918 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:50:39.778324 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765448 5918 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:50:39.778324 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765450 5918 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:50:39.778324 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765453 5918 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:50:39.778324 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765455 5918 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:50:39.778324 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765458 5918 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:50:39.778324 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765461 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:50:39.778324 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765463 5918 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:50:39.778324 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765467 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:50:39.778840 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765470 5918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:50:39.778840 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765472 5918 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:50:39.778840 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765475 5918 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:50:39.778840 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765477 5918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:50:39.778840 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765480 5918 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:50:39.778840 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765484 5918 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:50:39.778840 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765487 5918 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:50:39.778840 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765490 5918 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:50:39.778840 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765493 5918 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:50:39.778840 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765495 5918 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:50:39.778840 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765498 5918 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:50:39.778840 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765500 5918 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:50:39.778840 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765503 5918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:50:39.778840 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765505 5918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:50:39.778840 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765507 5918 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:50:39.778840 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765510 5918 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:50:39.778840 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765514 5918 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:50:39.778840 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765516 5918 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:50:39.778840 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765519 5918 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:50:39.779312 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765521 5918 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:50:39.779312 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765524 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:50:39.779312 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765527 5918 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:50:39.779312 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765530 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:50:39.779312 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765533 5918 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:50:39.779312 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765535 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:50:39.779312 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765538 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:50:39.779312 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765540 5918 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:50:39.779312 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765543 5918 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:50:39.779312 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765545 5918 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:50:39.779312 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765548 5918 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:50:39.779312 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765550 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:50:39.779312 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765553 5918 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:50:39.779312 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.765556 5918 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:50:39.779312 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.769341 5918 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:50:39.779312 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.769356 5918 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:50:39.779312 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.769373 5918 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:50:39.779312 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.769379 5918 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:50:39.779312 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769435 5918 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:50:39.779822 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769441 5918 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:50:39.779822 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769444 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:50:39.779822 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769448 5918 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:50:39.779822 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769450 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:50:39.779822 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769453 5918 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:50:39.779822 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769456 5918 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:50:39.779822 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769459 5918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:50:39.779822 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769462 5918 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:50:39.779822 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769464 5918 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:50:39.779822 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769467 5918 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:50:39.779822 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769470 5918 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:50:39.779822 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769472 5918 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:50:39.779822 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769475 5918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:50:39.779822 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769478 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:50:39.779822 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769480 5918 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:50:39.779822 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769483 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:50:39.779822 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769486 5918 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:50:39.779822 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769489 5918 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:50:39.779822 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769491 5918 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:50:39.779822 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769494 5918 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:50:39.780402 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769496 5918 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:50:39.780402 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769500 5918 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:50:39.780402 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769502 5918 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:50:39.780402 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769505 5918 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:50:39.780402 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769507 5918 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:50:39.780402 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769510 5918 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:50:39.780402 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769513 5918 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:50:39.780402 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769515 5918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:50:39.780402 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769518 5918 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:50:39.780402 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769520 5918 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:50:39.780402 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769523 5918 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:50:39.780402 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769526 5918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:50:39.780402 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769528 5918 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:50:39.780402 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769531 5918 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:50:39.780402 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769533 5918 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:50:39.780402 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769536 5918 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:50:39.780402 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769539 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:50:39.780402 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769541 5918 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:50:39.780402 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769544 5918 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:50:39.780852 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769546 5918 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:50:39.780852 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769549 5918 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:50:39.780852 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769551 5918 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:50:39.780852 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769553 5918 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:50:39.780852 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769556 5918 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:50:39.780852 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769558 5918 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:50:39.780852 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769561 5918 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:50:39.780852 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769564 5918 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:50:39.780852 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769566 5918 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:50:39.780852 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769570 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:50:39.780852 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769572 5918 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:50:39.780852 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769574 5918 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:50:39.780852 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769577 5918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:50:39.780852 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769581 5918 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:50:39.780852 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769583 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:50:39.780852 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769586 5918 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:50:39.780852 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769588 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:50:39.780852 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769591 5918 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:50:39.780852 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769593 5918 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:50:39.780852 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769595 5918 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:50:39.781338 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769598 5918 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:50:39.781338 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769600 5918 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:50:39.781338 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769603 5918 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:50:39.781338 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769605 5918 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:50:39.781338 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769608 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:50:39.781338 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769611 5918 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:50:39.781338 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769613 5918 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:50:39.781338 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769616 5918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:50:39.781338 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769618 5918 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:50:39.781338 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769621 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:50:39.781338 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769623 5918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:50:39.781338 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769626 5918 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:50:39.781338 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769629 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:50:39.781338 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769631 5918 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:50:39.781338 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769634 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:50:39.781338 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769636 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:50:39.781338 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769638 5918 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:50:39.781338 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769641 5918 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:50:39.781338 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769643 5918 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:50:39.781338 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769646 5918 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:50:39.781996 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769648 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:50:39.781996 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769650 5918 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:50:39.781996 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769653 5918 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:50:39.781996 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769655 5918 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:50:39.781996 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769658 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:50:39.781996 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769661 5918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:50:39.781996 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769663 5918 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:50:39.781996 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769666 5918 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:50:39.781996 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769669 5918 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:50:39.781996 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769673 5918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:50:39.781996 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769675 5918 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:50:39.781996 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769678 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:50:39.781996 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769680 5918 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:50:39.781996 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769683 5918 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:50:39.781996 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769686 5918 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:50:39.781996 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769688 5918 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:50:39.781996 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769690 5918 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:50:39.781996 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769693 5918 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:50:39.781996 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769696 5918 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:50:39.782603 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769698 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:50:39.782603 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769701 5918 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:50:39.782603 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769703 5918 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:50:39.782603 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769705 5918 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:50:39.782603 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769708 5918 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:50:39.782603 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769712 5918 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:50:39.782603 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.769716 5918 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:50:39.782603 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.769743 5918 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:50:39.782603 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770159 5918 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:50:39.782603 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770423 5918 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:50:39.782603 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770428 5918 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:50:39.782603 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770431 5918 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:50:39.782603 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770433 5918 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:50:39.782603 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770437 5918 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:50:39.782603 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770440 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:50:39.782603 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770442 5918 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:50:39.783087 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770449 5918 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:50:39.783087 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770452 5918 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:50:39.783087 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770455 5918 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:50:39.783087 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770458 5918 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:50:39.783087 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770461 5918 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:50:39.783087 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770466 5918 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:50:39.783087 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770470 5918 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:50:39.783087 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770474 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:50:39.783087 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770478 5918 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:50:39.783087 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770482 5918 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:50:39.783087 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770487 5918 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:50:39.783087 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770491 5918 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:50:39.783087 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770500 5918 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:50:39.783087 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770504 5918 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:50:39.783087 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770509 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:50:39.783087 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770514 5918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:50:39.783087 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770519 5918 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:50:39.783087 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770523 5918 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:50:39.783087 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770528 5918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:50:39.783087 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770532 5918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:50:39.783740 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770537 5918 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:50:39.783740 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770542 5918 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:50:39.783740 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770549 5918 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:50:39.783740 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770556 5918 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:50:39.783740 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770566 5918 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:50:39.783740 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770570 5918 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:50:39.783740 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770575 5918 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:50:39.783740 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770579 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:50:39.783740 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770583 5918 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:50:39.783740 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770587 5918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:50:39.783740 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770599 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:50:39.783740 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770604 5918 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:50:39.783740 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770608 5918 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:50:39.783740 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770612 5918 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:50:39.783740 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770617 5918 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:50:39.783740 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770621 5918 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:50:39.783740 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770625 5918 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:50:39.783740 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770634 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:50:39.783740 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770638 5918 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:50:39.784487 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770642 5918 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:50:39.784487 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770646 5918 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:50:39.784487 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770650 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:50:39.784487 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770654 5918 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:50:39.784487 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770659 5918 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:50:39.784487 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770663 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:50:39.784487 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770667 5918 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:50:39.784487 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770671 5918 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:50:39.784487 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770675 5918 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:50:39.784487 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770679 5918 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:50:39.784487 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770684 5918 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:50:39.784487 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770693 5918 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:50:39.784487 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770697 5918 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:50:39.784487 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770701 5918 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:50:39.784487 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770705 5918 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:50:39.784487 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770710 5918 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:50:39.784487 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770714 5918 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:50:39.784487 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770718 5918 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:50:39.784487 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770723 5918 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:50:39.784945 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770727 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:50:39.784945 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770732 5918 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:50:39.784945 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770736 5918 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:50:39.784945 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770748 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:50:39.784945 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770760 5918 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:50:39.784945 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770766 5918 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:50:39.784945 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770770 5918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:50:39.784945 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770806 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:50:39.784945 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770835 5918 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:50:39.784945 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770842 5918 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:50:39.784945 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770847 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:50:39.784945 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770853 5918 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:50:39.784945 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770858 5918 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:50:39.784945 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770863 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:50:39.784945 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770868 5918 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:50:39.784945 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770873 5918 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:50:39.784945 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770878 5918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:50:39.784945 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770883 5918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:50:39.784945 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770889 5918 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:50:39.784945 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770894 5918 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:50:39.785495 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770904 5918 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:50:39.785495 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770908 5918 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:50:39.785495 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770913 5918 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:50:39.785495 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770917 5918 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:50:39.785495 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770921 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:50:39.785495 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770925 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:50:39.785495 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770930 5918 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:50:39.785495 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770935 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:50:39.785495 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770939 5918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:50:39.785495 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770944 5918 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:50:39.785495 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770948 5918 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:50:39.785495 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770952 5918 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:50:39.785495 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770963 5918 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:50:39.785495 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770990 5918 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:50:39.785495 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.770995 5918 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:50:39.785495 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771000 5918 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:50:39.785495 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771005 5918 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:50:39.785495 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771009 5918 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:50:39.785495 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771013 5918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:50:39.785985 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771018 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:50:39.785985 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.771105 5918 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:50:39.785985 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771281 5918 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:50:39.785985 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771286 5918 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:50:39.785985 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771290 5918 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:50:39.785985 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771293 5918 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:50:39.785985 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771296 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:50:39.785985 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771299 5918 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:50:39.785985 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771302 5918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:50:39.785985 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771304 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:50:39.785985 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771307 5918 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:50:39.785985 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771310 5918 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:50:39.785985 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771312 5918 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:50:39.785985 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771316 5918 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:50:39.785985 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771319 5918 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:50:39.785985 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771321 5918 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:50:39.786391 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771324 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:50:39.786391 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771327 5918 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:50:39.786391 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771329 5918 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:50:39.786391 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771332 5918 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:50:39.786391 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771336 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:50:39.786391 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771340 5918 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:50:39.786391 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771343 5918 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:50:39.786391 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771346 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:50:39.786391 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771348 5918 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:50:39.786391 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771351 5918 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:50:39.786391 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771353 5918 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:50:39.786391 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771356 5918 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:50:39.786391 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771358 5918 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:50:39.786391 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771361 5918 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:50:39.786391 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771363 5918 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:50:39.786391 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771366 5918 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:50:39.786391 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771368 5918 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:50:39.786391 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771371 5918 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:50:39.786391 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771373 5918 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:50:39.786844 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771376 5918 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:50:39.786844 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771378 5918 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:50:39.786844 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771381 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:50:39.786844 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771384 5918 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:50:39.786844 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771387 5918 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:50:39.786844 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771389 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:50:39.786844 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771392 5918 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:50:39.786844 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771394 5918 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:50:39.786844 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771397 5918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:50:39.786844 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771399 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:50:39.786844 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771402 5918 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:50:39.786844 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771404 5918 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:50:39.786844 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771407 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:50:39.786844 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771409 5918 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:50:39.786844 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771412 5918 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:50:39.786844 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771414 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:50:39.786844 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771417 5918 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:50:39.786844 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771419 5918 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:50:39.786844 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771422 5918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:50:39.787441 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771424 5918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:50:39.787441 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771427 5918 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:50:39.787441 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771429 5918 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:50:39.787441 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771432 5918 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:50:39.787441 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771434 5918 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:50:39.787441 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771437 5918 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:50:39.787441 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771439 5918 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:50:39.787441 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771441 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:50:39.787441 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771444 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:50:39.787441 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771446 5918 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:50:39.787441 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771449 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:50:39.787441 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771451 5918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:50:39.787441 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771453 5918 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:50:39.787441 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771456 5918 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:50:39.787441 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771458 5918 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:50:39.787441 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771463 5918 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:50:39.787441 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771466 5918 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:50:39.787441 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771469 5918 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:50:39.787441 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771472 5918 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:50:39.787904 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771475 5918 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:50:39.787904 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771477 5918 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:50:39.787904 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771480 5918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:50:39.787904 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771483 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:50:39.787904 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771485 5918 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:50:39.787904 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771488 5918 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:50:39.787904 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771490 5918 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:50:39.787904 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771492 5918 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:50:39.787904 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771495 5918 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:50:39.787904 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771498 5918 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:50:39.787904 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771500 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:50:39.787904 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771503 5918 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:50:39.787904 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771505 5918 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:50:39.787904 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771509 5918 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:50:39.787904 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771512 5918 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:50:39.787904 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771514 5918 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:50:39.787904 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771517 5918 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:50:39.787904 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771520 5918 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:50:39.787904 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771522 5918 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:50:39.787904 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771525 5918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:50:39.788555 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771527 5918 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:50:39.788555 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771530 5918 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:50:39.788555 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771532 5918 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:50:39.788555 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771535 5918 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:50:39.788555 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771537 5918 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:50:39.788555 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771540 5918 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:50:39.788555 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771542 5918 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:50:39.788555 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771544 5918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:50:39.788555 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771547 5918 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:50:39.788555 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771552 5918 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:50:39.788555 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771555 5918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:50:39.788555 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771557 5918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:50:39.788555 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771560 5918 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:50:39.788555 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771562 5918 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:50:39.788555 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:50:39.771567 5918 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:50:39.788555 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.771623 5918 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:50:39.788555 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.772662 5918 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 21:50:39.788555 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.772711 5918 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 21:50:39.788555 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.772885 5918 server.go:1014] "Starting client certificate rotation" Oct 30 21:50:39.789318 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.773031 5918 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:50:39.789318 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.773102 5918 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:44:28 +0000 UTC" deadline="2028-06-20 00:47:31.32944082 +0000 UTC" Oct 30 21:50:39.789318 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.773143 5918 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14354h56m51.556301074s" Oct 30 21:50:39.789318 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.773341 5918 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:50:39.789318 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.774568 5918 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:50:39.789318 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.775852 5918 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:50:39.789318 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.777755 5918 log.go:25] "Validated CRI v1 image API" Oct 30 21:50:39.789318 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.778447 5918 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:50:39.789318 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.780746 5918 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c993a38e-afe9-456e-bb91-d227f9a63332:/dev/nvme0n1p4 efbbe2c6-6ea2-412f-a897-a1d2a68314fa:/dev/nvme0n1p3] Oct 30 21:50:39.789762 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.780765 5918 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/00df8434490defdffffbc4706d440e2f00ab97169b574c117a9fc6d6f1aec695/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/00df8434490defdffffbc4706d440e2f00ab97169b574c117a9fc6d6f1aec695/userdata/shm major:0 minor:472 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/032ce38740007e36580833395e1f6cb5af67f15993db39dec68faf5e16309b8c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/032ce38740007e36580833395e1f6cb5af67f15993db39dec68faf5e16309b8c/userdata/shm major:0 minor:369 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/03c8dd34da7ccc342a64c88427c9faa1d368d5aa84977dfe9abde48b44a5d7ee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/03c8dd34da7ccc342a64c88427c9faa1d368d5aa84977dfe9abde48b44a5d7ee/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/06ccadcde28a65df7d812878f515cd5553b31b449a576308afe8ed61d2926346/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/06ccadcde28a65df7d812878f515cd5553b31b449a576308afe8ed61d2926346/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/239c653cefcebd880c0ec32f20cce418280a8611f3b3a5f0ed35ab1717a315de/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/239c653cefcebd880c0ec32f20cce418280a8611f3b3a5f0ed35ab1717a315de/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3748a0779e5a23d6e1222a0447bf6f1fb23cfc4523a9104e0262381b37c677d0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3748a0779e5a23d6e1222a0447bf6f1fb23cfc4523a9104e0262381b37c677d0/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/414bb35e7badab19b1be107dc7b2cf6d0e1164f956de288edb7d68816b26fed9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/414bb35e7badab19b1be107dc7b2cf6d0e1164f956de288edb7d68816b26fed9/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4a01871d6a7408e50912a0bb509bf6dfe44039eecf78fd92976280b187301f36/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4a01871d6a7408e50912a0bb509bf6dfe44039eecf78fd92976280b187301f36/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/579477397ab60402a9ba83f3aa408531328e8e2901dd2d33996f716274408586/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/579477397ab60402a9ba83f3aa408531328e8e2901dd2d33996f716274408586/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b7492cb0c24700e79719b3ff607e612fb7f8b98a8ac52f0f09fe7715d19c97e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b7492cb0c24700e79719b3ff607e612fb7f8b98a8ac52f0f09fe7715d19c97e/userdata/shm major:0 minor:63 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/67dc0f160b11ce4f01ae7bbcbce36724376f58941b510f9809856eadbbe77b76/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/67dc0f160b11ce4f01ae7bbcbce36724376f58941b510f9809856eadbbe77b76/userdata/shm major:0 minor:247 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7141d4a04393caf7ededd58a5b768f9fdaeec111305133aaf1ee854c4cc89ee3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7141d4a04393caf7ededd58a5b768f9fdaeec111305133aaf1ee854c4cc89ee3/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/75914d1d2d2adc77ca1b238530dda483062212fea9c1fbd92ed4a5e373b89184/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/75914d1d2d2adc77ca1b238530dda483062212fea9c1fbd92ed4a5e373b89184/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8a2e36d351ddb4fe71b03446aa206d5cfb1a975dda8e76eef321bd198f35c791/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8a2e36d351ddb4fe71b03446aa206d5cfb1a975dda8e76eef321bd198f35c791/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/928f9029a8d5765f6d966c38c9db05323be5c91e8183a8819bb902cd6a3e1658/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/928f9029a8d5765f6d966c38c9db05323be5c91e8183a8819bb902cd6a3e1658/userdata/shm major:0 minor:474 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a603d78eeec3bba0ac01e9d232d39fca7877b42828bd2138a4a3b4e06fbbcf23/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a603d78eeec3bba0ac01e9d232d39fca7877b42828bd2138a4a3b4e06fbbcf23/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b54267304a3fa7233171f6499fc3bcdf39e09bfacd448f597e4b0840df316420/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b54267304a3fa7233171f6499fc3bcdf39e09bfacd448f597e4b0840df316420/userdata/shm major:0 minor:254 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bcd76e95362cbb32470158be7f72ebad66a49c3885bb19b9e1c292850d12b328/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bcd76e95362cbb32470158be7f72ebad66a49c3885bb19b9e1c292850d12b328/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c9fd11b3c3ba987ba67e1f854aba8a3c1dc65f84d19575984dbff4e5d0cec644/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c9fd11b3c3ba987ba67e1f854aba8a3c1dc65f84d19575984dbff4e5d0cec644/userdata/shm major:0 minor:470 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e13d803c6a7f26891ce23e9b9f5eb7c1ec83254184e57ffc124fc1f0b4198939/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e13d803c6a7f26891ce23e9b9f5eb7c1ec83254184e57ffc124fc1f0b4198939/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e31f54543dce236fa066ea1688a07b45980761a3231c0c804cc51d8f9e30fd2d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e31f54543dce236fa066ea1688a07b45980761a3231c0c804cc51d8f9e30fd2d/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f5e4b161dde81879deee20f513c5a60b7623438253853280a93b681288a12fb8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f5e4b161dde81879deee20f513c5a60b7623438253853280a93b681288a12fb8/userdata/shm major:0 minor:310 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f9fae51e3797f359f826f7d5c8436433cd1dae5fa3210c0f63881d21e1a27259/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f9fae51e3797f359f826f7d5c8436433cd1dae5fa3210c0f63881d21e1a27259/userdata/shm major:0 minor:447 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fb3e60f0f46a4d1992a0924c8dec57c15e8f16ff740e92fadeaf6f22455e54e6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fb3e60f0f46a4d1992a0924c8dec57c15e8f16ff740e92fadeaf6f22455e54e6/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fde0502c1ae08a6c80f38283d5a6c31b3c9c9835c5fc48a33c0e90a5b21cc559/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fde0502c1ae08a6c80f38283d5a6c31b3c9c9835c5fc48a33c0e90a5b21cc559/userdata/shm major:0 minor:250 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ad6cfef-ff22-4eb5-8910-a761537d737f/volumes/kubernetes.io~projected/kube-api-access-ntp69:{mountpoint:/var/lib/kubelet/pods/0ad6cfef-ff22-4eb5-8910-a761537d737f/volumes/kubernetes.io~projected/kube-api-access-ntp69 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ad6cfef-ff22-4eb5-8910-a761537d737f/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/0ad6cfef-ff22-4eb5-8910-a761537d737f/volumes/kubernetes.io~secret/metrics-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18a7ec6e-13fa-4893-a179-529f13fb2172/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/18a7ec6e-13fa-4893-a179-529f13fb2172/volumes/kubernetes.io~secret/agent-certs major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e76444f-1a5b-460c-b42f-2197b7512574/volumes/kubernetes.io~projected/kube-api-access-mf4tt:{mountpoint:/var/lib/kubelet/pods/1e76444f-1a5b-460c-b42f-2197b7512574/volumes/kubernetes.io~projected/kube-api-access-mf4tt major:0 minor:249 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e76444f-1a5b-460c-b42f-2197b7512574/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/1e76444f-1a5b-460c-b42f-2197b7512574/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e76444f-1a5b-460c-b42f-2197b7512574/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/1e76444f-1a5b-460c-b42f-2197b7512574/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:245 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/290eb94c-6ae5-40e8-8ba3-dbe017243d9d/volumes/kubernetes.io~projected/kube-api-access-58wwh:{mountpoint:/var/lib/kubelet/pods/290eb94c-6ae5-40e8-8ba3-dbe017243d9d/volumes/kubernetes.io~projected/kube-api-access-58wwh major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/290eb94c-6ae5-40e8-8ba3-dbe017243d9d/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/290eb94c-6ae5-40e8-8ba3-dbe017243d9d/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e5352ac-8295-494d-950a-16c09b09272e/volumes/kubernetes.io~projected/kube-api-access-dj8hr:{mountpoint:/var/lib/kubelet/pods/2e5352ac-8295-494d-950a-16c09b09272e/volumes/kubernetes.io~projected/kube-api-access-dj8hr major:0 minor:469 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3c199354-31a1-45b4-8c60-a485fd65f1e2/volumes/kubernetes.io~projected/kube-api-access-j4lcd:{mountpoint:/var/lib/kubelet/pods/3c199354-31a1-45b4-8c60-a485fd65f1e2/volumes/kubernetes.io~projected/kube-api-access-j4lcd major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3c199354-31a1-45b4-8c60-a485fd65f1e2/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/3c199354-31a1-45b4-8c60-a485fd65f1e2/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3c199354-31a1-45b4-8c60-a485fd65f1e2/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/3c199354-31a1-45b4-8c60-a485fd65f1e2/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:242 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4630b412-da87-49f7-bbbb-ba3dc820ddd1/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/4630b412-da87-49f7-bbbb-ba3dc820ddd1/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:468 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c/volumes/kubernetes.io~projected/kube-api-access-n67nc:{mountpoint:/var/lib/kubelet/pods/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c/volumes/kubernetes.io~projected/kube-api-access-n67nc major:0 minor:244 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/579c3ec8-fe12-472f-8ab0-247bfa8ed8b7/volumes/kubernetes.io~projected/kube-api-access-tfsd9:{mountpoint:/var/lib/kubelet/pods/579c3ec8-fe12-472f-8ab0-247bfa8ed8b7/volumes/kubernetes.io~projected/kube-api-access-tfsd9 major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7163574d-9b90-4a4c-83c5-5ca2d95a6275/volumes/kubernetes.io~projected/kube-api-access-zd8ns:{mountpoint:/var/lib/kubelet/pods/7163574d-9b90-4a4c-83c5-5ca2d95a6275/volumes/kubernetes.io~projected/kube-api-access-zd8ns major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7163574d-9b90-4a4c-83c5-5ca2d95a6275/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/7163574d-9b90-4a4c-83c5-5ca2d95a6275/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volume-subpaths/web-config/config-reloader/2 major:0 minor:443 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:443 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volume-subpaths/web-config/prometheus/15 major:0 minor:443 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volumes/kubernetes.io~empty-dir/config-out major:0 minor:441 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volumes/kubernetes.io~projected/kube-api-access-6nd2d:{mountpoint:/var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volumes/kubernetes.io~projected/kube-api-access-6nd2d major:0 minor:446 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volumes/kubernetes.io~projected/tls-assets major:0 minor:445 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volumes/kubernetes.io~secret/config major:0 minor:428 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:440 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:442 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:429 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:439 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:438 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:437 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:444 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volumes/kubernetes.io~secret/web-config major:0 minor:443 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7fd1e2d6-4877-4459-82f9-95b31d098229/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/7fd1e2d6-4877-4459-82f9-95b31d098229/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7fd1e2d6-4877-4459-82f9-95b31d098229/volumes/kubernetes.io~projected/kube-api-access-lp4k2:{mountpoint:/var/lib/kubelet/pods/7fd1e2d6-4877-4459-82f9-95b31d098229/volumes/kubernetes.io~projected/kube-api-access-lp4k2 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7fd1e2d6-4877-4459-82f9-95b31d098229/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/7fd1e2d6-4877-4459-82f9-95b31d098229/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83a9baf0-064d-4544-94b4-a0dd090dbd04/volumes/kubernetes.io~projected/kube-api-access-qp9j4:{mountpoint:/var/lib/kubelet/pods/83a9baf0-064d-4544-94b4-a0dd090dbd04/volumes/kubernetes.io~projected/kube-api-access-qp9j4 major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83a9baf0-064d-4544-94b4-a0dd090dbd04/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/83a9baf0-064d-4544-94b4-a0dd090dbd04/volumes/kubernetes.io~secret/metrics-certs major:0 minor:467 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a4d3550-acb2-4cf1-b192-adf694522695/volumes/kubernetes.io~projected/kube-api-access-5k48m:{mountpoint:/var/lib/kubelet/pods/8a4d3550-acb2-4cf1-b192-adf694522695/volumes/kubernetes.io~projected/kube-api-access-5k48m major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a4d3550-acb2-4cf1-b192-adf694522695/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/8a4d3550-acb2-4cf1-b192-adf694522695/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a4d3550-acb2-4cf1-b192-adf694522695/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/8a4d3550-acb2-4cf1-b192-adf694522695/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a4d3550-acb2-4cf1-b192-adf694522695/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/8a4d3550-acb2-4cf1-b192-adf694522695/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a4d3550-acb2-4cf1-b192-adf694522695/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/8a4d3550-acb2-4cf1-b192-adf694522695/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8/volumes/kubernetes.io~projected/kube-api-access-q6q6v:{mountpoint:/var/lib/kubelet/pods/9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8/volumes/kubernetes.io~projected/kube-api-access-q6q6v major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volume-subpaths/web-config/alertmanager/9 major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volume-subpaths/web-config/config-reloader/6 major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volumes/kubernetes.io~empty-dir/config-out major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volumes/kubernetes.io~projected/kube-api-access-g25nc:{mountpoint:/var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volumes/kubernetes.io~projected/kube-api-access-g25nc major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volumes/kubernetes.io~projected/tls-assets major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volumes/kubernetes.io~secret/config-volume major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volumes/kubernetes.io~secret/web-config major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4105fd0-16f9-492e-ae10-1f7458524eeb/volumes/kubernetes.io~projected/kube-api-access-t4xbt:{mountpoint:/var/lib/kubelet/pods/b4105fd0-16f9-492e-ae10-1f7458524eeb/volumes/kubernetes.io~projected/kube-api-access-t4xbt major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2226976-1041-4798-b567-6fb6590c0f6f/volumes/kubernetes.io~projected/kube-api-access-pp9qb:{mountpoint:/var/lib/kubelet/pods/d2226976-1041-4798-b567-6fb6590c0f6f/volumes/kubernetes.io~projected/kube-api-access-pp9qb major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d962c24e-5855-453e-97d5-78bad4a19fbc/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/d962c24e-5855-453e-97d5-78bad4a19fbc/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d962c24e-5855-453e-97d5-78bad4a19fbc/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/d962c24e-5855-453e-97d5-78bad4a19fbc/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d962c24e-5855-453e-97d5-78bad4a19fbc/volumes/kubernetes.io~projected/kube-api-access-h6nx7:{mountpoint:/var/lib/kubelet/pods/d962c24e-5855-453e-97d5-78bad4a19fbc/volumes/kubernetes.io~projected/kube-api-access-h6nx7 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1f2183b-a823-43ae-b7c8-c4b24dd3b403/volumes/kubernetes.io~projected/kube-api-access-ctmq5:{mountpoint:/var/lib/kubelet/pods/e1f2183b-a823-43ae-b7c8-c4b24dd3b403/volumes/kubernetes.io~projected/kube-api-access-ctmq5 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1f2183b-a823-43ae-b7c8-c4b24dd3b403/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/e1f2183b-a823-43ae-b7c8-c4b24dd3b403/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e321c1b1-e9e2-46fc-9a35-14a3d6bed233/volumes/kubernetes.io~projected/kube-api-access-q7nrk:{mountpoint:/var/lib/kubelet/pods/e321c1b1-e9e2-46fc-9a35-14a3d6bed233/volumes/kubernetes.io~projected/kube-api-access-q7nrk major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e321c1b1-e9e2-46fc-9a35-14a3d6bed233/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/e321c1b1-e9e2-46fc-9a35-14a3d6bed233/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e321c1b1-e9e2-46fc-9a35-14a3d6bed233/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/e321c1b1-e9e2-46fc-9a35-14a3d6bed233/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e321c1b1-e9e2-46fc-9a35-14a3d6bed233/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/e321c1b1-e9e2-46fc-9a35-14a3d6bed233/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7b50ca6-0f9c-40f8-a09d-004b055105b8/volumes/kubernetes.io~projected/kube-api-access-svfrj:{mountpoint:/var/lib/kubelet/pods/e7b50ca6-0f9c-40f8-a09d-004b055105b8/volumes/kubernetes.io~projected/kube-api-access-svfrj major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4925cf0-04b5-4ca4-b284-c0355e63b4ca/volumes/kubernetes.io~projected/kube-api-access-sgqbt:{mountpoint:/var/lib/kubelet/pods/f4925cf0-04b5-4ca4-b284-c0355e63b4ca/volumes/kubernetes.io~projected/kube-api-access-sgqbt major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4925cf0-04b5-4ca4-b284-c0355e63b4ca/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/f4925cf0-04b5-4ca4-b284-c0355e63b4ca/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:326 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4925cf0-04b5-4ca4-b284-c0355e63b4ca/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/f4925cf0-04b5-4ca4-b284-c0355e63b4ca/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4925cf0-04b5-4ca4-b284-c0355e63b4ca/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/f4925cf0-04b5-4ca4-b284-c0355e63b4ca/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4925cf0-04b5-4ca4-b284-c0355e63b4ca/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/f4925cf0-04b5-4ca4-b284-c0355e63b4ca/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4925cf0-04b5-4ca4-b284-c0355e63b4ca/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/f4925cf0-04b5-4ca4-b284-c0355e63b4ca/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4925cf0-04b5-4ca4-b284-c0355e63b4ca/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/f4925cf0-04b5-4ca4-b284-c0355e63b4ca/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f5d7f625-40b0-46b6-8489-fdd1daa0068e/volumes/kubernetes.io~projected/kube-api-access-kkk9k:{mountpoint:/var/lib/kubelet/pods/f5d7f625-40b0-46b6-8489-fdd1daa0068e/volumes/kubernetes.io~projected/kube-api-access-kkk9k major:0 minor:56 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/1c04fc26860e5e341c879d23bbb2238d439dc149f8582e4946e2292061322503/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/c6979c7b6e1effa0ec6cf5199ba6c053f63d6125befe04dea436a327384cf280/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/5fb78a1a4a7758d23a65591997a6a24ba70761fabbb451387023216dc1e6ab03/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/f49503d02a94c3b55b2c2eb4298650ebbc0dd543dde3831973593dea925ff283/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/119115415ace655380291be3ea2b7e60e0f20afb3959e7b4f57dd2e2a9e8a5d8/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/762475ed3871a747d1c744878913e9a40f6a3848bdd8ce8e0f54e66f44cbe0c1/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/eaf8f7aab8215a64ce487dd76b1218271600abb00ef333f789ac950817c36f81/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/8283dc06738515739ec461b190b42196aea0ca62310202afe80814490920012b/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/b4b73c51b36cc107c7b3dcbc69dbb77b005bb7d7196315498e3eaf6dfdcefe82/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/6d41e8d0de6701d59bae0e05837dae062431b4b319a234371efb31e016177ef7/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/1a0d59aeda50667e1daec77c79cfcf070052763944cdeca6a34509515a28b3e5/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/a23cfed7fc995c13b10b2c030ca0aedc31073d6b266746ff9369fd234ed1bd77/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/e3cf0fe09eb8f17ccb7889bddd3649a3e5d3767e48a14a6bd3ff73efba2b8ede/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/d768469bc2b8a53dc7820f257004a06fe387a0e745ad62bca7846cd39f58cecb/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-158:{mountpoint:/var/lib/containers/storage/overlay/d2bae0b98a28d2c8225d4d18552c5c446749f76b17a13a223e06099fa51a6b47/merged major:0 minor:158 fsType:overlay blockSize:0} overlay_0-160:{mountpoint:/var/lib/containers/storage/overlay/4e093f34a95e4f8b039f0be1bb9e61a34f279c1ba7142fb4ed1fd6713ea06eed/merged major:0 minor:160 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/9b9b08483be711f0afdabc6d3a0862f5bfa8ab176302c4b6447c84b1661e7e13/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/e8d80c9063345b9aa6fed20cd586fd57e0fa114511a0eabf679305b13c77402f/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/ce133cd1bb907e81f2b20dea28a55bcf1774f43606970f26c73765a2f8e7a042/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-184:{mountpoint:/var/lib/containers/storage/overlay/6c7bb8a247e130d7fb77ac4e294e25e40eec3707e9db546cfa8f4d9cbb8e00cc/merged major:0 minor:184 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/2f47f68cfacd3b6d9a60d36f9ceae44757346d2c2cfd64e67a2b521a485d4b72/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/7ef12ee0aa4ea4f048ce6483d3d9feb25653c31093508e7ec5717374388ac873/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/a425ddb4549f9043fb6233087cd4905ee894efa72cf4b37c1acff1a6a842f036/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/d4c48c2dd5349d9bc84daed23ed3c6f166b9a0cd931a8cc5e8bfc0ac1a6202e7/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/86a98d4624f5ef5c63d16f14c1160393399ebed071712b69af5d886875c80559/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/9a2efe8875efc60a20a57773bab31ec7cd9fd10c6651ef44a8b3a96bcb9ef065/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/17c27e1d9d50768a614eece22b4be30e7d8d126c362ef0b826a519ec150db3fe/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/2373aa28ca0ac5c30bb746f7751597664731d91a5d3597433871aa0ac1a5b58e/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/71fa0d73308370cb1f527657f2e514874e7ce36a7f0e99f9a6061747c1466f4c/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/8724ca9f344b1e82fca22b86250d7cd43feabec36aba60a4382c848069b19617/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/69ba9f5d16de4b3ae5568873a9d67b7e375376f664dfd69e7760bf77eb5415b1/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/c1b56b16110107dd3fdd0d9aba74c2b9b595d4d32c36a4b22db89f2889324f38/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/b5b6fa478b5dbebb3227a4998e4f4ce8da7ec80e55d2dfe24fca44808808b6c3/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/82bd175a65c0ea37e2d7a988dd17789a55a1e7916c421531656ed5b967462dcf/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/2874cda34d0bab2109f41ab817a22fc584a54a95d36c7e175d0bbb45b4e450d6/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/c726ee5b62ca48d274bc8ab7bdaf36867770bca143dc3e0aec349eb86e1b5af4/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/8e542101f5d83fb89afaa35e16051ab5d3533ed7807f4f4ef1b3ed0ecdeb2e84/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/b2366f9efde7f9f3fa1370c7c6fd539ef85c0b1b53d66af10e0f4d570f5a77a2/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/12e17efbc2269fccec7d243ee6b65c893bd2dbd0a90f1bf86b05aa0dc05a341b/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/05dce0f17ca950a48806f91a7d0f2be1858a036224ce69892378fa4aa5175a34/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/8547faab6ec839ecbea4f59a3618148d03c67aef2d9637cd5d1a2eb1781a8efd/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/4415cf687c2dd6ddf66675f5f06e4f9d5928aa6efb58bbba4874420f5443e453/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/b918ee8d3c1e131346984ffa4df2af6e348c22a174cc1e4b4c9d685b613e17d1/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/835c78955945192cbf2a5797bae089e526ae8b476e62f1b4aa5ebe6746877b13/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/dd9334bf5a279ecfaca6edb93a3498ad8bb6ae5c83e7921804952371392fbaae/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/4bbda24027282b6f25e2d746caae55e591f6e36df7837f67629e8c326e763365/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/19ebc6bf261e85c8703797449cc2f81c77f0c3eaf229a965d2cfd91912920a7e/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-365:{mountpoint:/var/lib/containers/storage/overlay/259cd31ee8c7f2518fcb9bff23c8175100ee7cc9d9828d31792220f48b29e659/merged major:0 minor:365 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/e39b52c99601dcbcbd4cd48595065fd02c5e63681cfa006d58a2dfe7b1af39ab/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/4ba16a01f3eda2ce2baaa5138c8b2e8e2c6758babb124fa2954ca3b76d2806c7/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-379:{mountpoint:/var/lib/containers/storage/overlay/e197cb9ef9838adb490c673737044c97290337719d85fa93327aa03ef609fae5/merged major:0 minor:379 fsType:overlay blockSize:0} overlay_0-381:{mountpoint:/var/lib/containers/storage/overlay/a355d4db6bff5295594f70d3e5d8cda9baa98fb4ed70c320160c20fa4e0da0eb/merged major:0 minor:381 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/2c392c038ff8a52d0c13c126a3d80b647736893e679a50ec5336ff21685c669a/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-395:{mountpoint:/var/lib/containers/storage/overlay/ed522e6b49e505b4d6c793382f0532b826abab9952024209ba8fb11aea126626/merged major:0 minor:395 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/063cd1c2b6c58f884b2b0b7111b5bb1a3e6b2154ed7d8b82f8cbd8a7a8774520/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/2e6d11587a38aedbed779e3f3749a5fd36c9b945e04d3799f704298778e08a07/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-411:{mountpoint:/var/lib/containers/storage/overlay/d43f8916daac34b3c1aab7881b0ee8c7ab9f03f97e8543d30e51e6921f733adf/merged major:0 minor:411 fsType:overlay blockSize:0} overlay_0-416:{mountpoint:/var/lib/containers/storage/overlay/fb09caf35e2adc7ccb82900d5b3183a8bcb996d9f206425e701e8378b9d60f38/merged major:0 minor:416 fsType:overlay blockSize:0} overlay_0-421:{mountpoint:/var/lib/containers/storage/overlay/8c76fd7dc2ea8c1abebe75f9ee0dd7669f1c15a43d9346c583059579d9c45fb7/merged major:0 minor:421 fsType:overlay blockSize:0} overlay_0-426:{mountpoint:/var/lib/containers/storage/overlay/578bb55151636daaf12ffed2e950c49811c2cd9c0cad86b257b55696fb385b48/merged major:0 minor:426 fsType:overlay blockSize:0} overlay_0-449:{mountpoint:/var/lib/containers/storage/overlay/39aa3636fe1d40a6b92b62611b7df24129f69b71f522e0cad67f205e8116e30d/merged major:0 minor:449 fsType:overlay blockSize:0} overlay_0-451:{mountpoint:/var/lib/containers/storage/overlay/22bca342b54623fddc6a6ccf857fc4c0d9f6ebacf67b41d3106cd389fed58450/merged major:0 minor:451 fsType:overlay blockSize:0} overlay_0-457:{mountpoint:/var/lib/containers/storage/overlay/6d05b2069295f264ecd498b16408cce91a083e1acd8c9ac3cc82617684f64a56/merged major:0 minor:457 fsType:overlay blockSize:0} overlay_0-462:{mountpoint:/var/lib/containers/storage/overlay/ff47eb37f5d89b6891d5a6adeeb1c06bdd59dbe5ab8235dc184bc3d36a05524e/merged major:0 minor:462 fsType:overlay blockSize:0} overlay_0-476:{mountpoint:/var/lib/containers/storage/overlay/aacfc5203903cc67a7104e8c29d322fd8d3b5039efc472244f6967ab408b3a58/merged major:0 minor:476 fsType:overlay blockSize:0} overlay_0-478:{mountpoint:/var/lib/containers/storage/overlay/bea137f9660e89a2258ceebc15f0c25996113e5d3eb49994c001585c1748c89c/merged major:0 minor:478 fsType:overlay blockSize:0} overlay_0-480:{mountpoint:/var/lib/containers/storage/overlay/904255edd71df02ef6022822a09ca52efaf22e7192793fd9f7233eb1b1792bfd/merged major:0 minor:480 fsType:overlay blockSize:0} overlay_0-482:{mountpoint:/var/lib/containers/storage/overlay/53689c0d1b2003b655cfab12c3cd9dbf7d32b63802a6ef74b786ee6d370d3704/merged major:0 minor:482 fsType:overlay blockSize:0} overlay_0-484:{mountpoint:/var/lib/containers/storage/overlay/f4ab0c39d3c0a2fe3e3a74b4d22b86c10759d4ea0d916089956a28d8f9c88a83/merged major:0 minor:484 fsType:overlay blockSize:0} overlay_0-490:{mountpoint:/var/lib/containers/storage/overlay/386286a6d14f8b9ac403601474d4ed80a0c8d9366350776ad2ce9d4759d92743/merged major:0 minor:490 fsType:overlay blockSize:0} overlay_0-495:{mountpoint:/var/lib/containers/storage/overlay/61d602bfc4106fd1e8bfcb16fb9b3dd2b7ff690515c9aa293c818d04856345d8/merged major:0 minor:495 fsType:overlay blockSize:0} overlay_0-500:{mountpoint:/var/lib/containers/storage/overlay/1bdb4fc87e1562a87cb46fca58babf3d6eb7938cc218a9e90e3a3e4a0f0cace2/merged major:0 minor:500 fsType:overlay blockSize:0} overlay_0-505:{mountpoint:/var/lib/containers/storage/overlay/353a6ca81166850df42168862ce86a072cfe89bccc401fd06a8bb57ec7729fed/merged major:0 minor:505 fsType:overlay blockSize:0} overlay_0-510:{mountpoint:/var/lib/containers/storage/overlay/db0ef7f7338b4db44d15fe8d4205fb2615fb4cf3222909c350d07feaaec92a48/merged major:0 minor:510 fsType:overlay blockSize:0} overlay_0-515:{mountpoint:/var/lib/containers/storage/overlay/1b30e24ea8c5204197808451e58814d800c1328fdda98daabbec80c8af1229d0/merged major:0 minor:515 fsType:overlay blockSize:0} overlay_0-521:{mountpoint:/var/lib/containers/storage/overlay/f8450f99920bb2a924126fa4216313fe41e38c4f614a3df5bf8d397ad1f991a1/merged major:0 minor:521 fsType:overlay blockSize:0} overlay_0-526:{mountpoint:/var/lib/containers/storage/overlay/67174f8c5d6c6b950c2f271929f24b62d034ddc14f71aacc9c27194764300a1f/merged major:0 minor:526 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/9e2255dd1fd02f147f13fb3f08d482ccdc8c8ffaef2201e7d82eab6454b680d8/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/1a53a73cbcfc4ae232214cbf865a21a6c2602690a2f022f48f2255991e346b69/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/50cb5409b0788863c5db9ae9626fa86c036ac2c45b9201496d582a02cd03a283/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/64ef107b972d3805d96dd2dad508b96eb7b219abd23ed06f962d4ff2c0062840/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/6c816dc67956a20b58a44858f9deb3acb52fbe516d06b93bb8ff665a7b60ff97/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/e3dd052e78d6ba3ac2671e5b2c49cc910c1e48f5636a7b4b22bfbbb83b34debb/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/dee0e1c42f0ca44c7b935626c983c98634b2d292024780bbc035461923ee4b4e/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/635f88b995ba1b07fa6267a5df12a9952919a578d5bcf0b85647752ab3d54945/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/b83ededfbc7e9a1b498179d3c589839bd7dbf9ba2336dc4f57b8c5f034514cc7/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/6073a39eed44eee61f6eede32c812f4d124421058dbb2fabe7035a4a91aa46a0/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 21:50:39.790444 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.789485 5918 manager.go:217] Machine: {Timestamp:2026-10-30 21:50:39.788615336 +0000 UTC m=+0.059313417 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100030 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec298e9147d176c257d2d4c9379a8bba SystemUUID:ec298e91-47d1-76c2-57d2-d4c9379a8bba BootID:43e37fc7-77f8-4761-ae62-e22182a891f6 Filesystems:[{Device:/var/lib/kubelet/pods/f4925cf0-04b5-4ca4-b284-c0355e63b4ca/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:322 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:440 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-482 DeviceMajor:0 DeviceMinor:482 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/239c653cefcebd880c0ec32f20cce418280a8611f3b3a5f0ed35ab1717a315de/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8a4d3550-acb2-4cf1-b192-adf694522695/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:353 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:241 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c/volumes/kubernetes.io~projected/kube-api-access-n67nc DeviceMajor:0 DeviceMinor:244 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:286 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e321c1b1-e9e2-46fc-9a35-14a3d6bed233/volumes/kubernetes.io~projected/kube-api-access-q7nrk DeviceMajor:0 DeviceMinor:345 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/7fd1e2d6-4877-4459-82f9-95b31d098229/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:445 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/83a9baf0-064d-4544-94b4-a0dd090dbd04/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:467 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d962c24e-5855-453e-97d5-78bad4a19fbc/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8/volumes/kubernetes.io~projected/kube-api-access-q6q6v DeviceMajor:0 DeviceMinor:51 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7141d4a04393caf7ededd58a5b768f9fdaeec111305133aaf1ee854c4cc89ee3/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4a01871d6a7408e50912a0bb509bf6dfe44039eecf78fd92976280b187301f36/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3c199354-31a1-45b4-8c60-a485fd65f1e2/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:242 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e7b50ca6-0f9c-40f8-a09d-004b055105b8/volumes/kubernetes.io~projected/kube-api-access-svfrj DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/75914d1d2d2adc77ca1b238530dda483062212fea9c1fbd92ed4a5e373b89184/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fde0502c1ae08a6c80f38283d5a6c31b3c9c9835c5fc48a33c0e90a5b21cc559/userdata/shm DeviceMajor:0 DeviceMinor:250 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e321c1b1-e9e2-46fc-9a35-14a3d6bed233/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:344 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e13d803c6a7f26891ce23e9b9f5eb7c1ec83254184e57ffc124fc1f0b4198939/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1e76444f-1a5b-460c-b42f-2197b7512574/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:245 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b54267304a3fa7233171f6499fc3bcdf39e09bfacd448f597e4b0840df316420/userdata/shm DeviceMajor:0 DeviceMinor:254 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e31f54543dce236fa066ea1688a07b45980761a3231c0c804cc51d8f9e30fd2d/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8a4d3550-acb2-4cf1-b192-adf694522695/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:354 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7fd1e2d6-4877-4459-82f9-95b31d098229/volumes/kubernetes.io~projected/kube-api-access-lp4k2 DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0ad6cfef-ff22-4eb5-8910-a761537d737f/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/06ccadcde28a65df7d812878f515cd5553b31b449a576308afe8ed61d2926346/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-416 DeviceMajor:0 DeviceMinor:416 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:437 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:444 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7163574d-9b90-4a4c-83c5-5ca2d95a6275/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:299 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-411 DeviceMajor:0 DeviceMinor:411 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:428 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:438 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/928f9029a8d5765f6d966c38c9db05323be5c91e8183a8819bb902cd6a3e1658/userdata/shm DeviceMajor:0 DeviceMinor:474 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7fd1e2d6-4877-4459-82f9-95b31d098229/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/67dc0f160b11ce4f01ae7bbcbce36724376f58941b510f9809856eadbbe77b76/userdata/shm DeviceMajor:0 DeviceMinor:247 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:443 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fb3e60f0f46a4d1992a0924c8dec57c15e8f16ff740e92fadeaf6f22455e54e6/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1e76444f-1a5b-460c-b42f-2197b7512574/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:246 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f4925cf0-04b5-4ca4-b284-c0355e63b4ca/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:329 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:441 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c9fd11b3c3ba987ba67e1f854aba8a3c1dc65f84d19575984dbff4e5d0cec644/userdata/shm DeviceMajor:0 DeviceMinor:470 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/00df8434490defdffffbc4706d440e2f00ab97169b574c117a9fc6d6f1aec695/userdata/shm DeviceMajor:0 DeviceMinor:472 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-478 DeviceMajor:0 DeviceMinor:478 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b4105fd0-16f9-492e-ae10-1f7458524eeb/volumes/kubernetes.io~projected/kube-api-access-t4xbt DeviceMajor:0 DeviceMinor:53 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e1f2183b-a823-43ae-b7c8-c4b24dd3b403/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:287 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-510 DeviceMajor:0 DeviceMinor:510 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-526 DeviceMajor:0 DeviceMinor:526 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d962c24e-5855-453e-97d5-78bad4a19fbc/volumes/kubernetes.io~projected/kube-api-access-h6nx7 DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f5d7f625-40b0-46b6-8489-fdd1daa0068e/volumes/kubernetes.io~projected/kube-api-access-kkk9k DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:300 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-476 DeviceMajor:0 DeviceMinor:476 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-500 DeviceMajor:0 DeviceMinor:500 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-521 DeviceMajor:0 DeviceMinor:521 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d2226976-1041-4798-b567-6fb6590c0f6f/volumes/kubernetes.io~projected/kube-api-access-pp9qb DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:236 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/032ce38740007e36580833395e1f6cb5af67f15993db39dec68faf5e16309b8c/userdata/shm DeviceMajor:0 DeviceMinor:369 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volumes/kubernetes.io~projected/kube-api-access-6nd2d DeviceMajor:0 DeviceMinor:446 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-515 DeviceMajor:0 DeviceMinor:515 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/579477397ab60402a9ba83f3aa408531328e8e2901dd2d33996f716274408586/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e321c1b1-e9e2-46fc-9a35-14a3d6bed233/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:342 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:302 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e1f2183b-a823-43ae-b7c8-c4b24dd3b403/volumes/kubernetes.io~projected/kube-api-access-ctmq5 DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bcd76e95362cbb32470158be7f72ebad66a49c3885bb19b9e1c292850d12b328/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3c199354-31a1-45b4-8c60-a485fd65f1e2/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:240 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b7492cb0c24700e79719b3ff607e612fb7f8b98a8ac52f0f09fe7715d19c97e/userdata/shm DeviceMajor:0 DeviceMinor:63 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:303 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f4925cf0-04b5-4ca4-b284-c0355e63b4ca/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:330 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8a4d3550-acb2-4cf1-b192-adf694522695/volumes/kubernetes.io~projected/kube-api-access-5k48m DeviceMajor:0 DeviceMinor:356 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-381 DeviceMajor:0 DeviceMinor:381 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-421 DeviceMajor:0 DeviceMinor:421 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-495 DeviceMajor:0 DeviceMinor:495 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f5e4b161dde81879deee20f513c5a60b7623438253853280a93b681288a12fb8/userdata/shm DeviceMajor:0 DeviceMinor:310 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e321c1b1-e9e2-46fc-9a35-14a3d6bed233/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:343 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-379 DeviceMajor:0 DeviceMinor:379 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-449 DeviceMajor:0 DeviceMinor:449 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4630b412-da87-49f7-bbbb-ba3dc820ddd1/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:468 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:443 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3748a0779e5a23d6e1222a0447bf6f1fb23cfc4523a9104e0262381b37c677d0/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f4925cf0-04b5-4ca4-b284-c0355e63b4ca/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:328 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-426 DeviceMajor:0 DeviceMinor:426 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:439 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/579c3ec8-fe12-472f-8ab0-247bfa8ed8b7/volumes/kubernetes.io~projected/kube-api-access-tfsd9 DeviceMajor:0 DeviceMinor:52 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1e76444f-1a5b-460c-b42f-2197b7512574/volumes/kubernetes.io~projected/kube-api-access-mf4tt DeviceMajor:0 DeviceMinor:249 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:297 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:429 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:443 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d962c24e-5855-453e-97d5-78bad4a19fbc/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:47 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volumes/kubernetes.io~projected/kube-api-access-g25nc DeviceMajor:0 DeviceMinor:309 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/290eb94c-6ae5-40e8-8ba3-dbe017243d9d/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:338 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-395 DeviceMajor:0 DeviceMinor:395 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-484 DeviceMajor:0 DeviceMinor:484 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-505 DeviceMajor:0 DeviceMinor:505 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0ad6cfef-ff22-4eb5-8910-a761537d737f/volumes/kubernetes.io~projected/kube-api-access-ntp69 DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:301 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f4925cf0-04b5-4ca4-b284-c0355e63b4ca/volumes/kubernetes.io~projected/kube-api-access-sgqbt DeviceMajor:0 DeviceMinor:331 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:300 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:443 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/03c8dd34da7ccc342a64c88427c9faa1d368d5aa84977dfe9abde48b44a5d7ee/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a603d78eeec3bba0ac01e9d232d39fca7877b42828bd2138a4a3b4e06fbbcf23/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:300 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-451 DeviceMajor:0 DeviceMinor:451 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3c199354-31a1-45b4-8c60-a485fd65f1e2/volumes/kubernetes.io~projected/kube-api-access-j4lcd DeviceMajor:0 DeviceMinor:243 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-480 DeviceMajor:0 DeviceMinor:480 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7163574d-9b90-4a4c-83c5-5ca2d95a6275/volumes/kubernetes.io~projected/kube-api-access-zd8ns DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8a4d3550-acb2-4cf1-b192-adf694522695/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:352 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:442 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f4925cf0-04b5-4ca4-b284-c0355e63b4ca/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:327 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f4925cf0-04b5-4ca4-b284-c0355e63b4ca/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:326 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/290eb94c-6ae5-40e8-8ba3-dbe017243d9d/volumes/kubernetes.io~projected/kube-api-access-58wwh DeviceMajor:0 DeviceMinor:339 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-365 DeviceMajor:0 DeviceMinor:365 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f9fae51e3797f359f826f7d5c8436433cd1dae5fa3210c0f63881d21e1a27259/userdata/shm DeviceMajor:0 DeviceMinor:447 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-160 DeviceMajor:0 DeviceMinor:160 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-462 DeviceMajor:0 DeviceMinor:462 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/18a7ec6e-13fa-4893-a179-529f13fb2172/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:49 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/83a9baf0-064d-4544-94b4-a0dd090dbd04/volumes/kubernetes.io~projected/kube-api-access-qp9j4 DeviceMajor:0 DeviceMinor:54 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:300 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2e5352ac-8295-494d-950a-16c09b09272e/volumes/kubernetes.io~projected/kube-api-access-dj8hr DeviceMajor:0 DeviceMinor:469 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/414bb35e7badab19b1be107dc7b2cf6d0e1164f956de288edb7d68816b26fed9/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-457 DeviceMajor:0 DeviceMinor:457 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8a2e36d351ddb4fe71b03446aa206d5cfb1a975dda8e76eef321bd198f35c791/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-184 DeviceMajor:0 DeviceMinor:184 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8a4d3550-acb2-4cf1-b192-adf694522695/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:355 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:298 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-490 DeviceMajor:0 DeviceMinor:490 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-158 DeviceMajor:0 DeviceMinor:158 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:298 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:00df8434490defd MacAddress:aa:9d:b6:78:16:dc Speed:10000 Mtu:8901} {Name:032ce38740007e3 MacAddress:5a:be:0a:0b:1f:4b Speed:10000 Mtu:8901} {Name:03c8dd34da7ccc3 MacAddress:9a:58:99:9f:30:1b Speed:10000 Mtu:8901} {Name:06ccadcde28a65d MacAddress:a2:2b:b6:8d:90:11 Speed:10000 Mtu:8901} {Name:67dc0f160b11ce4 MacAddress:52:da:b2:6f:68:0f Speed:10000 Mtu:8901} {Name:928f9029a8d5765 MacAddress:7e:9d:d9:f9:e2:5d Speed:10000 Mtu:8901} {Name:a603d78eeec3bba MacAddress:ba:7c:55:c7:f5:69 Speed:10000 Mtu:8901} {Name:b54267304a3fa72 MacAddress:42:b0:2e:9f:e2:6b Speed:10000 Mtu:8901} {Name:bcd76e95362cbb3 MacAddress:1e:05:ae:df:b5:c6 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:51:9a:1c:aa:a7 Speed:0 Mtu:9001} {Name:br-int MacAddress:1e:dd:fe:af:72:81 Speed:0 Mtu:8901} {Name:c9fd11b3c3ba987 MacAddress:5e:05:e0:05:9f:b9 Speed:10000 Mtu:8901} {Name:e31f54543dce236 MacAddress:82:fa:fb:0a:d9:fa Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:51:9a:1c:aa:a7 Speed:0 Mtu:9001} {Name:f5e4b161dde8187 MacAddress:9e:61:6d:98:e0:26 Speed:10000 Mtu:8901} {Name:f9fae51e3797f35 MacAddress:ba:b4:11:25:c1:0f Speed:10000 Mtu:8901} {Name:fb3e60f0f46a4d1 MacAddress:aa:95:54:6f:a0:cb Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:ce:25:a6:80:6a:e4 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:a6:3a:57:37:91:10 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:50:39.790693 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.790451 5918 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:50:39.790693 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.790507 5918 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:50:39.790803 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.790778 5918 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:50:39.790926 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.790804 5918 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-81.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:50:39.790970 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.790928 5918 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:50:39.790970 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.790936 5918 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:50:39.790970 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.790950 5918 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:50:39.790970 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.790963 5918 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:50:39.791108 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.791041 5918 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:50:39.791195 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.791186 5918 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:50:39.791327 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.791318 5918 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:50:39.791364 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.791356 5918 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:50:39.791393 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.791368 5918 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:50:39.791393 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.791379 5918 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:50:39.791393 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.791389 5918 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:50:39.791508 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.791398 5918 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:50:39.792194 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.792179 5918 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:50:39.792194 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.792193 5918 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:50:39.792688 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.792669 5918 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:50:39.795347 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.795331 5918 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:50:39.795347 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.795345 5918 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:50:39.795468 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.795458 5918 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 21:50:39.795660 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.795649 5918 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:50:39.795696 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.795679 5918 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:50:39.795891 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.795881 5918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:50:39.795933 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.795899 5918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:50:39.795933 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.795905 5918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:50:39.795933 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.795914 5918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:50:39.795933 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.795924 5918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:50:39.795933 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.795932 5918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:50:39.796064 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.795937 5918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:50:39.796064 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.795943 5918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:50:39.796064 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.795950 5918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:50:39.796064 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.795958 5918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:50:39.796064 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.795978 5918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:50:39.796064 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.795988 5918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:50:39.796064 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.796019 5918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:50:39.796064 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.796027 5918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:50:39.799502 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.799484 5918 server.go:1263] "Started kubelet" Oct 30 21:50:39.799645 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.799572 5918 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:50:39.799645 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.799585 5918 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:50:39.799750 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.799692 5918 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:50:39.799998 ip-10-0-137-81 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:50:39.800127 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.800107 5918 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:50:39.801612 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.801593 5918 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:50:39.807155 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.807135 5918 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:50:39.807270 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.807166 5918 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:50:39.807955 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.807332 5918 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:50:39.807955 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.807347 5918 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:50:39.807955 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.807364 5918 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:50:39.808401 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.808372 5918 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:44:29 +0000 UTC" deadline="2028-07-04 03:45:49.394418485 +0000 UTC" Oct 30 21:50:39.808494 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.808401 5918 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14693h55m9.586020883s" Oct 30 21:50:39.808494 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:50:39.808423 5918 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-81.us-west-1.compute.internal\" not found" Oct 30 21:50:39.809767 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.808831 5918 factory.go:153] Registering CRI-O factory Oct 30 21:50:39.809767 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.808849 5918 factory.go:223] Registration of the crio container factory successfully Oct 30 21:50:39.809767 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.808904 5918 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:50:39.809767 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.808914 5918 factory.go:55] Registering systemd factory Oct 30 21:50:39.809767 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.808922 5918 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:50:39.809767 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.808938 5918 factory.go:103] Registering Raw factory Oct 30 21:50:39.809767 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.808953 5918 manager.go:1199] Started watching for new ooms in manager Oct 30 21:50:39.809767 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.809318 5918 manager.go:319] Starting recovery of all containers Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.811379 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="736951e5-1596-4606-94d9-80d8b50d35c3" volumeName="kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.811420 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" volumeName="kubernetes.io/empty-dir/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-config-out" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.811435 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4925cf0-04b5-4ca4-b284-c0355e63b4ca" volumeName="kubernetes.io/projected/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-kube-api-access-sgqbt" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.811461 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ad6cfef-ff22-4eb5-8910-a761537d737f" volumeName="kubernetes.io/configmap/0ad6cfef-ff22-4eb5-8910-a761537d737f-config-volume" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.811474 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7163574d-9b90-4a4c-83c5-5ca2d95a6275" volumeName="kubernetes.io/projected/7163574d-9b90-4a4c-83c5-5ca2d95a6275-kube-api-access-zd8ns" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.811486 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" volumeName="kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.811498 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c199354-31a1-45b4-8c60-a485fd65f1e2" volumeName="kubernetes.io/configmap/3c199354-31a1-45b4-8c60-a485fd65f1e2-metrics-client-ca" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.811510 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c199354-31a1-45b4-8c60-a485fd65f1e2" volumeName="kubernetes.io/secret/3c199354-31a1-45b4-8c60-a485fd65f1e2-openshift-state-metrics-tls" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.811524 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2226976-1041-4798-b567-6fb6590c0f6f" volumeName="kubernetes.io/configmap/d2226976-1041-4798-b567-6fb6590c0f6f-cni-binary-copy" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.811549 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4925cf0-04b5-4ca4-b284-c0355e63b4ca" volumeName="kubernetes.io/secret/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-secret-grpc-tls" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.811573 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18a7ec6e-13fa-4893-a179-529f13fb2172" volumeName="kubernetes.io/configmap/18a7ec6e-13fa-4893-a179-529f13fb2172-konnectivity-ca" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.811585 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e5352ac-8295-494d-950a-16c09b09272e" volumeName="kubernetes.io/projected/2e5352ac-8295-494d-950a-16c09b09272e-kube-api-access-dj8hr" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.811595 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e321c1b1-e9e2-46fc-9a35-14a3d6bed233" volumeName="kubernetes.io/projected/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-kube-api-access-q7nrk" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.811615 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="736951e5-1596-4606-94d9-80d8b50d35c3" volumeName="kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.811628 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d962c24e-5855-453e-97d5-78bad4a19fbc" volumeName="kubernetes.io/empty-dir/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-tuned" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.811639 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18a7ec6e-13fa-4893-a179-529f13fb2172" volumeName="kubernetes.io/secret/18a7ec6e-13fa-4893-a179-529f13fb2172-agent-certs" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.811650 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a7edc22-1eb1-4458-8f8c-6c46a2bd583c" volumeName="kubernetes.io/configmap/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.811669 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="736951e5-1596-4606-94d9-80d8b50d35c3" volumeName="kubernetes.io/empty-dir/736951e5-1596-4606-94d9-80d8b50d35c3-config-out" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.811684 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7fd1e2d6-4877-4459-82f9-95b31d098229" volumeName="kubernetes.io/configmap/7fd1e2d6-4877-4459-82f9-95b31d098229-ovnkube-script-lib" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.811694 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="736951e5-1596-4606-94d9-80d8b50d35c3" volumeName="kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.811712 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="736951e5-1596-4606-94d9-80d8b50d35c3" volumeName="kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-web-config" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.811724 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a4d3550-acb2-4cf1-b192-adf694522695" volumeName="kubernetes.io/secret/8a4d3550-acb2-4cf1-b192-adf694522695-secret-telemeter-client" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.811745 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a4d3550-acb2-4cf1-b192-adf694522695" volumeName="kubernetes.io/secret/8a4d3550-acb2-4cf1-b192-adf694522695-telemeter-client-tls" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.811756 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4105fd0-16f9-492e-ae10-1f7458524eeb" volumeName="kubernetes.io/configmap/b4105fd0-16f9-492e-ae10-1f7458524eeb-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.811769 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2226976-1041-4798-b567-6fb6590c0f6f" volumeName="kubernetes.io/configmap/d2226976-1041-4798-b567-6fb6590c0f6f-multus-daemon-config" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.811804 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" volumeName="kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.811816 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4925cf0-04b5-4ca4-b284-c0355e63b4ca" volumeName="kubernetes.io/secret/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.811826 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4925cf0-04b5-4ca4-b284-c0355e63b4ca" volumeName="kubernetes.io/secret/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.811844 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a7edc22-1eb1-4458-8f8c-6c46a2bd583c" volumeName="kubernetes.io/projected/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-kube-api-access-n67nc" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.811856 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c199354-31a1-45b4-8c60-a485fd65f1e2" volumeName="kubernetes.io/projected/3c199354-31a1-45b4-8c60-a485fd65f1e2-kube-api-access-j4lcd" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.811872 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83a9baf0-064d-4544-94b4-a0dd090dbd04" volumeName="kubernetes.io/secret/83a9baf0-064d-4544-94b4-a0dd090dbd04-metrics-certs" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.811921 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" volumeName="kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.811933 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4105fd0-16f9-492e-ae10-1f7458524eeb" volumeName="kubernetes.io/projected/b4105fd0-16f9-492e-ae10-1f7458524eeb-kube-api-access-t4xbt" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.811946 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1f2183b-a823-43ae-b7c8-c4b24dd3b403" volumeName="kubernetes.io/configmap/e1f2183b-a823-43ae-b7c8-c4b24dd3b403-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.811967 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="290eb94c-6ae5-40e8-8ba3-dbe017243d9d" volumeName="kubernetes.io/secret/290eb94c-6ae5-40e8-8ba3-dbe017243d9d-monitoring-plugin-cert" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.811980 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7b50ca6-0f9c-40f8-a09d-004b055105b8" volumeName="kubernetes.io/projected/e7b50ca6-0f9c-40f8-a09d-004b055105b8-kube-api-access-svfrj" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.811996 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7163574d-9b90-4a4c-83c5-5ca2d95a6275" volumeName="kubernetes.io/secret/7163574d-9b90-4a4c-83c5-5ca2d95a6275-cert" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812011 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" volumeName="kubernetes.io/projected/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-kube-api-access-g25nc" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812026 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d962c24e-5855-453e-97d5-78bad4a19fbc" volumeName="kubernetes.io/empty-dir/d962c24e-5855-453e-97d5-78bad4a19fbc-tmp" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812041 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a7edc22-1eb1-4458-8f8c-6c46a2bd583c" volumeName="kubernetes.io/empty-dir/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-node-exporter-textfile" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812056 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="736951e5-1596-4606-94d9-80d8b50d35c3" volumeName="kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-config" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812067 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d962c24e-5855-453e-97d5-78bad4a19fbc" volumeName="kubernetes.io/projected/d962c24e-5855-453e-97d5-78bad4a19fbc-kube-api-access-h6nx7" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812078 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="290eb94c-6ae5-40e8-8ba3-dbe017243d9d" volumeName="kubernetes.io/projected/290eb94c-6ae5-40e8-8ba3-dbe017243d9d-kube-api-access-58wwh" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812090 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="736951e5-1596-4606-94d9-80d8b50d35c3" volumeName="kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-configmap-metrics-client-ca" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812105 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="736951e5-1596-4606-94d9-80d8b50d35c3" volumeName="kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812117 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e321c1b1-e9e2-46fc-9a35-14a3d6bed233" volumeName="kubernetes.io/empty-dir/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-audit-log" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812141 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ad6cfef-ff22-4eb5-8910-a761537d737f" volumeName="kubernetes.io/empty-dir/0ad6cfef-ff22-4eb5-8910-a761537d737f-tmp-dir" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812155 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1f2183b-a823-43ae-b7c8-c4b24dd3b403" volumeName="kubernetes.io/projected/e1f2183b-a823-43ae-b7c8-c4b24dd3b403-kube-api-access-ctmq5" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812169 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4630b412-da87-49f7-bbbb-ba3dc820ddd1" volumeName="kubernetes.io/secret/4630b412-da87-49f7-bbbb-ba3dc820ddd1-original-pull-secret" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812243 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7fd1e2d6-4877-4459-82f9-95b31d098229" volumeName="kubernetes.io/configmap/7fd1e2d6-4877-4459-82f9-95b31d098229-ovnkube-config" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812259 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e321c1b1-e9e2-46fc-9a35-14a3d6bed233" volumeName="kubernetes.io/secret/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-secret-metrics-server-tls" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812272 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a7edc22-1eb1-4458-8f8c-6c46a2bd583c" volumeName="kubernetes.io/secret/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-node-exporter-tls" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812305 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="736951e5-1596-4606-94d9-80d8b50d35c3" volumeName="kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812321 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a7edc22-1eb1-4458-8f8c-6c46a2bd583c" volumeName="kubernetes.io/secret/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812364 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a7edc22-1eb1-4458-8f8c-6c46a2bd583c" volumeName="kubernetes.io/configmap/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-metrics-client-ca" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812405 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e321c1b1-e9e2-46fc-9a35-14a3d6bed233" volumeName="kubernetes.io/secret/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812481 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7b50ca6-0f9c-40f8-a09d-004b055105b8" volumeName="kubernetes.io/configmap/e7b50ca6-0f9c-40f8-a09d-004b055105b8-iptables-alerter-script" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812511 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e76444f-1a5b-460c-b42f-2197b7512574" volumeName="kubernetes.io/secret/1e76444f-1a5b-460c-b42f-2197b7512574-kube-state-metrics-tls" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812524 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c199354-31a1-45b4-8c60-a485fd65f1e2" volumeName="kubernetes.io/secret/3c199354-31a1-45b4-8c60-a485fd65f1e2-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812535 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a4d3550-acb2-4cf1-b192-adf694522695" volumeName="kubernetes.io/configmap/8a4d3550-acb2-4cf1-b192-adf694522695-metrics-client-ca" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812572 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a4d3550-acb2-4cf1-b192-adf694522695" volumeName="kubernetes.io/configmap/8a4d3550-acb2-4cf1-b192-adf694522695-telemeter-trusted-ca-bundle" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812583 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" volumeName="kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-web-config" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812620 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4105fd0-16f9-492e-ae10-1f7458524eeb" volumeName="kubernetes.io/configmap/b4105fd0-16f9-492e-ae10-1f7458524eeb-cni-binary-copy" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812631 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7fd1e2d6-4877-4459-82f9-95b31d098229" volumeName="kubernetes.io/secret/7fd1e2d6-4877-4459-82f9-95b31d098229-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812643 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="736951e5-1596-4606-94d9-80d8b50d35c3" volumeName="kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812666 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="736951e5-1596-4606-94d9-80d8b50d35c3" volumeName="kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-grpc-tls" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812697 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" volumeName="kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-config-volume" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812717 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="736951e5-1596-4606-94d9-80d8b50d35c3" volumeName="kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812729 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="736951e5-1596-4606-94d9-80d8b50d35c3" volumeName="kubernetes.io/projected/736951e5-1596-4606-94d9-80d8b50d35c3-kube-api-access-6nd2d" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812739 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="736951e5-1596-4606-94d9-80d8b50d35c3" volumeName="kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812758 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" volumeName="kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812770 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2226976-1041-4798-b567-6fb6590c0f6f" volumeName="kubernetes.io/projected/d2226976-1041-4798-b567-6fb6590c0f6f-kube-api-access-pp9qb" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812781 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5d7f625-40b0-46b6-8489-fdd1daa0068e" volumeName="kubernetes.io/projected/f5d7f625-40b0-46b6-8489-fdd1daa0068e-kube-api-access-kkk9k" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812821 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" volumeName="kubernetes.io/projected/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-tls-assets" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812831 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e321c1b1-e9e2-46fc-9a35-14a3d6bed233" volumeName="kubernetes.io/configmap/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812843 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4925cf0-04b5-4ca4-b284-c0355e63b4ca" volumeName="kubernetes.io/secret/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-secret-thanos-querier-tls" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812859 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="736951e5-1596-4606-94d9-80d8b50d35c3" volumeName="kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812872 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83a9baf0-064d-4544-94b4-a0dd090dbd04" volumeName="kubernetes.io/projected/83a9baf0-064d-4544-94b4-a0dd090dbd04-kube-api-access-qp9j4" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812881 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4105fd0-16f9-492e-ae10-1f7458524eeb" volumeName="kubernetes.io/configmap/b4105fd0-16f9-492e-ae10-1f7458524eeb-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812891 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e76444f-1a5b-460c-b42f-2197b7512574" volumeName="kubernetes.io/secret/1e76444f-1a5b-460c-b42f-2197b7512574-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812902 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="736951e5-1596-4606-94d9-80d8b50d35c3" volumeName="kubernetes.io/empty-dir/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-db" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812914 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8" volumeName="kubernetes.io/configmap/9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8-serviceca" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812926 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" volumeName="kubernetes.io/configmap/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-metrics-client-ca" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812938 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" volumeName="kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-cluster-tls-config" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812950 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e76444f-1a5b-460c-b42f-2197b7512574" volumeName="kubernetes.io/empty-dir/1e76444f-1a5b-460c-b42f-2197b7512574-volume-directive-shadow" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812960 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="736951e5-1596-4606-94d9-80d8b50d35c3" volumeName="kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812970 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="579c3ec8-fe12-472f-8ab0-247bfa8ed8b7" volumeName="kubernetes.io/empty-dir/579c3ec8-fe12-472f-8ab0-247bfa8ed8b7-tmp-dir" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812983 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e76444f-1a5b-460c-b42f-2197b7512574" volumeName="kubernetes.io/configmap/1e76444f-1a5b-460c-b42f-2197b7512574-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.812994 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="736951e5-1596-4606-94d9-80d8b50d35c3" volumeName="kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-metrics-client-certs" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.813005 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4925cf0-04b5-4ca4-b284-c0355e63b4ca" volumeName="kubernetes.io/empty-dir/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-tmp" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.813016 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7fd1e2d6-4877-4459-82f9-95b31d098229" volumeName="kubernetes.io/configmap/7fd1e2d6-4877-4459-82f9-95b31d098229-env-overrides" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.813026 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a4d3550-acb2-4cf1-b192-adf694522695" volumeName="kubernetes.io/configmap/8a4d3550-acb2-4cf1-b192-adf694522695-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.813036 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" volumeName="kubernetes.io/configmap/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.813049 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4925cf0-04b5-4ca4-b284-c0355e63b4ca" volumeName="kubernetes.io/secret/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.813061 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8" volumeName="kubernetes.io/projected/9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8-kube-api-access-q6q6v" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.813074 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1f2183b-a823-43ae-b7c8-c4b24dd3b403" volumeName="kubernetes.io/empty-dir/e1f2183b-a823-43ae-b7c8-c4b24dd3b403-data-volume" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.813084 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4925cf0-04b5-4ca4-b284-c0355e63b4ca" volumeName="kubernetes.io/configmap/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-metrics-client-ca" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.813107 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="579c3ec8-fe12-472f-8ab0-247bfa8ed8b7" volumeName="kubernetes.io/projected/579c3ec8-fe12-472f-8ab0-247bfa8ed8b7-kube-api-access-tfsd9" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.813130 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e76444f-1a5b-460c-b42f-2197b7512574" volumeName="kubernetes.io/configmap/1e76444f-1a5b-460c-b42f-2197b7512574-metrics-client-ca" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.813141 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" volumeName="kubernetes.io/empty-dir/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-alertmanager-main-db" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.813154 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1f2183b-a823-43ae-b7c8-c4b24dd3b403" volumeName="kubernetes.io/secret/e1f2183b-a823-43ae-b7c8-c4b24dd3b403-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.813165 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e321c1b1-e9e2-46fc-9a35-14a3d6bed233" volumeName="kubernetes.io/configmap/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-metrics-server-audit-profiles" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.813177 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e321c1b1-e9e2-46fc-9a35-14a3d6bed233" volumeName="kubernetes.io/secret/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-client-ca-bundle" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.813193 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7fd1e2d6-4877-4459-82f9-95b31d098229" volumeName="kubernetes.io/projected/7fd1e2d6-4877-4459-82f9-95b31d098229-kube-api-access-lp4k2" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.813206 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ad6cfef-ff22-4eb5-8910-a761537d737f" volumeName="kubernetes.io/projected/0ad6cfef-ff22-4eb5-8910-a761537d737f-kube-api-access-ntp69" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.813240 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="736951e5-1596-4606-94d9-80d8b50d35c3" volumeName="kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.813255 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a4d3550-acb2-4cf1-b192-adf694522695" volumeName="kubernetes.io/projected/8a4d3550-acb2-4cf1-b192-adf694522695-kube-api-access-5k48m" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.813266 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a4d3550-acb2-4cf1-b192-adf694522695" volumeName="kubernetes.io/secret/8a4d3550-acb2-4cf1-b192-adf694522695-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.813278 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a4d3550-acb2-4cf1-b192-adf694522695" volumeName="kubernetes.io/secret/8a4d3550-acb2-4cf1-b192-adf694522695-federate-client-tls" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.813289 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ad6cfef-ff22-4eb5-8910-a761537d737f" volumeName="kubernetes.io/secret/0ad6cfef-ff22-4eb5-8910-a761537d737f-metrics-tls" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.813299 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e76444f-1a5b-460c-b42f-2197b7512574" volumeName="kubernetes.io/projected/1e76444f-1a5b-460c-b42f-2197b7512574-kube-api-access-mf4tt" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.813309 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="736951e5-1596-4606-94d9-80d8b50d35c3" volumeName="kubernetes.io/projected/736951e5-1596-4606-94d9-80d8b50d35c3-tls-assets" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.813338 5918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4925cf0-04b5-4ca4-b284-c0355e63b4ca" volumeName="kubernetes.io/secret/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.813360 5918 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:50:39.814242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.813367 5918 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:50:39.835320 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.835277 5918 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:50:39.843683 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.843661 5918 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:50:39.843683 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.843683 5918 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:50:39.843836 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.843704 5918 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:50:39.843836 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:50:39.843743 5918 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 21:50:39.860906 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.860870 5918 generic.go:356] "Generic (PLEG): container finished" podID="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" containerID="b7029e8c3647fc8cf124a1d4e669b174d5c966d763a6a2cbc18befc88e3d1ffc" exitCode=0 Oct 30 21:50:39.895904 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.895866 5918 generic.go:356] "Generic (PLEG): container finished" podID="4a7edc22-1eb1-4458-8f8c-6c46a2bd583c" containerID="8b8e2ef3d207dd9acbf3a214fec4b521041f5ebd3986fe535ca44813a98e918c" exitCode=0 Oct 30 21:50:39.906256 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.906203 5918 generic.go:356] "Generic (PLEG): container finished" podID="b4105fd0-16f9-492e-ae10-1f7458524eeb" containerID="88213f3653a0b8bd50d75209b8503b1d8b3af276ad7312bee50dd1558d5ade5f" exitCode=0 Oct 30 21:50:39.906256 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.906254 5918 generic.go:356] "Generic (PLEG): container finished" podID="b4105fd0-16f9-492e-ae10-1f7458524eeb" containerID="df3ab91aec64f7a669bf65cae46a3c31abfb020db6e17d622ed8ca3ce8771b47" exitCode=0 Oct 30 21:50:39.906398 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.906266 5918 generic.go:356] "Generic (PLEG): container finished" podID="b4105fd0-16f9-492e-ae10-1f7458524eeb" containerID="e1a04b5ad21206e5b7e9319e80537b4f5e37fea736731c8a1548397883cd83d8" exitCode=0 Oct 30 21:50:39.906398 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.906275 5918 generic.go:356] "Generic (PLEG): container finished" podID="b4105fd0-16f9-492e-ae10-1f7458524eeb" containerID="eac26567ad4fb6633d1a3b2cca0d87f6444761e1778a963c5a3cea7c3ac12024" exitCode=0 Oct 30 21:50:39.906398 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.906284 5918 generic.go:356] "Generic (PLEG): container finished" podID="b4105fd0-16f9-492e-ae10-1f7458524eeb" containerID="74ba2e46e24ff5a83c78a6b30e6f62b704c65774c5811a80debd3cc86b37f5e6" exitCode=0 Oct 30 21:50:39.906398 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.906292 5918 generic.go:356] "Generic (PLEG): container finished" podID="b4105fd0-16f9-492e-ae10-1f7458524eeb" containerID="171726a64cfc861e06ebb78fab3019699d041756721df0a0d0b517a71041c046" exitCode=0 Oct 30 21:50:39.909552 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:50:39.909526 5918 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-81.us-west-1.compute.internal\" not found" Oct 30 21:50:39.913299 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.913264 5918 generic.go:356] "Generic (PLEG): container finished" podID="736951e5-1596-4606-94d9-80d8b50d35c3" containerID="6ecd79a7c6d356f8e64253560432de55058bb4c2da2408ddf9e73b11af211256" exitCode=0 Oct 30 21:50:39.917153 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.917127 5918 generic.go:356] "Generic (PLEG): container finished" podID="8bdaae95a41fc764b37958d5699dbc6f" containerID="e1ec7bd724076009f96580dd30dd373e2022054d34721e211874664b85440e9b" exitCode=0 Oct 30 21:50:39.933634 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.933612 5918 manager.go:324] Recovery completed Oct 30 21:50:39.943823 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:50:39.943800 5918 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 21:50:39.959843 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.959824 5918 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:50:39.961619 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.961600 5918 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:50:39.961710 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.961632 5918 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:50:39.961710 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.961647 5918 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:50:39.964727 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.964707 5918 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:50:39.964727 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.964722 5918 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:50:39.964871 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.964730 5918 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:50:39.964871 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.964738 5918 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:50:39.964956 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.964872 5918 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 21:50:39.964956 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.964887 5918 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 21:50:39.964956 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.964908 5918 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 21:50:39.964956 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.964916 5918 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 21:50:39.964956 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.964923 5918 policy_none.go:50] "Start" Oct 30 21:50:39.964956 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.964932 5918 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:50:39.964956 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.964941 5918 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:50:39.965144 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.965077 5918 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 21:50:39.965144 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.965085 5918 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 21:50:39.965144 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.965091 5918 policy_none.go:44] "Start" Oct 30 21:50:39.965710 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.965692 5918 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:50:39.965789 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.965727 5918 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:50:39.965789 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.965742 5918 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:50:39.970178 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.970154 5918 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:50:39.972774 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.972758 5918 manager.go:341] "Starting Device Plugin manager" Oct 30 21:50:39.972852 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:50:39.972780 5918 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:50:39.972852 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.972789 5918 server.go:87] "Starting device plugin registration server" Oct 30 21:50:39.973038 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.973025 5918 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:50:39.973103 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.973037 5918 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:50:39.973160 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.973139 5918 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:50:39.973309 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.973269 5918 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:50:39.973417 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.973315 5918 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:50:39.973417 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.973323 5918 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:50:39.973532 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.973381 5918 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:50:39.973304258Z","UUID":"c83ec3a4-0b49-4c09-83a8-f0451d9a8712","Handler":null,"Name":"","Endpoint":""} Oct 30 21:50:39.974743 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.974724 5918 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:50:39.974836 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:39.974749 5918 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:50:39.974936 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:50:39.974916 5918 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:50:39.975017 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:50:39.974959 5918 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-81.us-west-1.compute.internal\" not found" Oct 30 21:50:40.007605 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.007575 5918 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:50:40.007755 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.007682 5918 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 21:50:40.009307 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.009290 5918 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:50:40.009681 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.009664 5918 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:50:40.073416 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.073390 5918 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:50:40.074496 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.074478 5918 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:50:40.074607 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.074511 5918 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:50:40.074607 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.074525 5918 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-81.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:50:40.074607 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.074602 5918 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:50:40.107902 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.107874 5918 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:50:40.144638 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.144583 5918 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-west-1.compute.internal"] Oct 30 21:50:40.145247 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.145188 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-west-1.compute.internal" event={"ID":"ff2bf62cbfb1edc876c4b4c7d0808622","Type":"ContainerStarted","Data":"858cbf4cc2fc8d7295ab25c152c6d8eba107b132ae05f8c7f554d680f37e646b"} Oct 30 21:50:40.145247 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.145251 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-west-1.compute.internal" event={"ID":"ff2bf62cbfb1edc876c4b4c7d0808622","Type":"ContainerStarted","Data":"e13d803c6a7f26891ce23e9b9f5eb7c1ec83254184e57ffc124fc1f0b4198939"} Oct 30 21:50:40.145414 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.145348 5918 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:50:40.145414 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.145361 5918 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:50:40.145528 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.145351 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal" event={"ID":"8bdaae95a41fc764b37958d5699dbc6f","Type":"ContainerStarted","Data":"42b975907e04084f274c85de36330639352f755c619906e4f37740af4b3ca3fb"} Oct 30 21:50:40.145528 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.145443 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal" event={"ID":"8bdaae95a41fc764b37958d5699dbc6f","Type":"ContainerDied","Data":"e1ec7bd724076009f96580dd30dd373e2022054d34721e211874664b85440e9b"} Oct 30 21:50:40.145528 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.145455 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal" event={"ID":"8bdaae95a41fc764b37958d5699dbc6f","Type":"ContainerStarted","Data":"7141d4a04393caf7ededd58a5b768f9fdaeec111305133aaf1ee854c4cc89ee3"} Oct 30 21:50:40.224440 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.224349 5918 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:50:40.224440 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:50:40.224398 5918 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-81.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:50:40.224644 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.224607 5918 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:50:40.224644 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:50:40.224640 5918 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:50:40.226277 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.226257 5918 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:50:40.226365 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.226328 5918 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:50:40.308616 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.308348 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ff2bf62cbfb1edc876c4b4c7d0808622-config\") pod \"kube-apiserver-proxy-ip-10-0-137-81.us-west-1.compute.internal\" (UID: \"ff2bf62cbfb1edc876c4b4c7d0808622\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:50:40.308616 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.308391 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8bdaae95a41fc764b37958d5699dbc6f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal\" (UID: \"8bdaae95a41fc764b37958d5699dbc6f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:50:40.308616 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.308417 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bdaae95a41fc764b37958d5699dbc6f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal\" (UID: \"8bdaae95a41fc764b37958d5699dbc6f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:50:40.409094 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.409065 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8bdaae95a41fc764b37958d5699dbc6f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal\" (UID: \"8bdaae95a41fc764b37958d5699dbc6f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:50:40.409094 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.409097 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bdaae95a41fc764b37958d5699dbc6f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal\" (UID: \"8bdaae95a41fc764b37958d5699dbc6f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:50:40.409349 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.409113 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ff2bf62cbfb1edc876c4b4c7d0808622-config\") pod \"kube-apiserver-proxy-ip-10-0-137-81.us-west-1.compute.internal\" (UID: \"ff2bf62cbfb1edc876c4b4c7d0808622\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:50:40.409349 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.409154 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ff2bf62cbfb1edc876c4b4c7d0808622-config\") pod \"kube-apiserver-proxy-ip-10-0-137-81.us-west-1.compute.internal\" (UID: \"ff2bf62cbfb1edc876c4b4c7d0808622\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:50:40.409349 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.409187 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bdaae95a41fc764b37958d5699dbc6f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal\" (UID: \"8bdaae95a41fc764b37958d5699dbc6f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:50:40.409349 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.409186 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8bdaae95a41fc764b37958d5699dbc6f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal\" (UID: \"8bdaae95a41fc764b37958d5699dbc6f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:50:40.792463 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.792422 5918 apiserver.go:52] "Watching apiserver" Oct 30 21:50:40.907321 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.907284 5918 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:50:40.907968 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.907938 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b","openshift-image-registry/node-ca-4dqsc","openshift-insights/insights-runtime-extractor-v6wts","openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4","openshift-multus/multus-b7k9p","openshift-monitoring/node-exporter-q5c65","openshift-multus/multus-additional-cni-plugins-d4fs7","openshift-network-operator/iptables-alerter-q8gsv","kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-west-1.compute.internal","openshift-multus/network-metrics-daemon-h4v24","openshift-network-diagnostics/network-check-target-mg6qm","openshift-dns/dns-default-zf4n4","openshift-monitoring/telemeter-client-89884678b-sxtjs","openshift-monitoring/thanos-querier-796d66f799-qtc4t","openshift-cluster-node-tuning-operator/tuned-zxdbg","openshift-monitoring/alertmanager-main-0","kube-system/global-pull-secret-syncer-xbwzg","openshift-monitoring/monitoring-plugin-77f4f76594-dd5t2","openshift-monitoring/metrics-server-d6fb658f8-zvqw4","openshift-monitoring/prometheus-k8s-0","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal","openshift-dns/node-resolver-8qxjv","openshift-ingress-canary/ingress-canary-7khqw","openshift-monitoring/openshift-state-metrics-7bd7489957-x796z","openshift-ovn-kubernetes/ovnkube-node-6th89","kube-system/konnectivity-agent-hmx87"] Oct 30 21:50:40.923450 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.923421 5918 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:50:40.923592 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.923578 5918 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:50:40.962297 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.962276 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:50:40.962297 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.962302 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pqcb5\"" Oct 30 21:50:40.962488 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.962373 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-gmxqg\"" Oct 30 21:50:40.963192 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.963172 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:50:40.964522 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.964504 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-w9pjm\"" Oct 30 21:50:40.964620 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.964535 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rrcm9\"" Oct 30 21:50:40.964620 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.964562 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wvpps\"" Oct 30 21:50:40.964782 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.964767 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:50:40.964844 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.964815 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:50:40.964844 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.964833 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qjhgs\"" Oct 30 21:50:40.964955 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.964853 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:50:40.964955 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.964821 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:50:40.964955 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.964873 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:50:40.964955 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.964856 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:50:40.964955 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.964888 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:50:40.964955 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.964901 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:50:40.964955 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.964919 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:50:40.964955 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.964835 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:50:40.964955 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.964941 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:50:40.964955 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.964930 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-k97q2\"" Oct 30 21:50:40.965354 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.964976 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:50:40.965354 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.965062 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:50:40.965354 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.965142 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r6nvz\"" Oct 30 21:50:40.965354 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.965170 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:50:40.965517 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.965370 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:50:40.965517 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.965382 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:50:40.965517 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.965392 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:50:40.965517 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.965445 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:50:40.965517 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.965476 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:50:40.965517 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.965487 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:50:40.965713 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.965536 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:50:40.965713 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.965587 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9mdmg\"" Oct 30 21:50:40.965713 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.965613 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fs82w\"" Oct 30 21:50:40.965790 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.965753 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:50:40.965931 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.965919 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-f5s29\"" Oct 30 21:50:40.965960 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.965943 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:50:40.966475 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.966461 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:50:40.966530 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.966466 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:50:40.966530 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.966515 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:50:40.966586 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.966517 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:50:40.970253 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.970241 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:50:40.970349 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.970333 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:50:40.970401 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.970348 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-c6mrn\"" Oct 30 21:50:40.970401 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.970395 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:50:40.970470 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.970407 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:50:40.970470 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.970444 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:50:40.970532 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:40.970466 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jjx9q\"" Oct 30 21:50:41.003549 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.003522 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:50:41.003640 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.003591 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:50:41.003640 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.003626 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 21:50:41.003729 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.003667 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:50:41.003729 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.003693 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:50:41.003941 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.003926 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lwnsl\"" Oct 30 21:50:41.004033 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.004018 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:50:41.004083 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.004061 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:50:41.004247 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.004231 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-vjtf6\"" Oct 30 21:50:41.004291 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.004237 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:50:41.009119 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.009099 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 21:50:41.009206 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.009122 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 21:50:41.009301 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.009288 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 21:50:41.010028 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.010009 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 21:50:41.010832 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.010815 5918 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:50:41.012077 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.012029 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.012077 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.012071 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f5d7f625-40b0-46b6-8489-fdd1daa0068e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:50:41.012668 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.012096 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-multus-cni-dir\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.012668 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.012159 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-systemd-units\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.012668 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.012191 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 21:50:41.012668 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.012207 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-metrics-server-audit-profiles\") pod \"metrics-server-d6fb658f8-zvqw4\" (UID: \"e321c1b1-e9e2-46fc-9a35-14a3d6bed233\") " pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" Oct 30 21:50:41.012668 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.012275 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e76444f-1a5b-460c-b42f-2197b7512574-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-lbjd4\" (UID: \"1e76444f-1a5b-460c-b42f-2197b7512574\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4" Oct 30 21:50:41.012668 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.012314 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/290eb94c-6ae5-40e8-8ba3-dbe017243d9d-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-dd5t2\" (UID: \"290eb94c-6ae5-40e8-8ba3-dbe017243d9d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dd5t2" Oct 30 21:50:41.012668 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.012353 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58wwh\" (UniqueName: \"kubernetes.io/projected/290eb94c-6ae5-40e8-8ba3-dbe017243d9d-kube-api-access-58wwh\") pod \"monitoring-plugin-77f4f76594-dd5t2\" (UID: \"290eb94c-6ae5-40e8-8ba3-dbe017243d9d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dd5t2" Oct 30 21:50:41.012668 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.012378 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6nx7\" (UniqueName: \"kubernetes.io/projected/d962c24e-5855-453e-97d5-78bad4a19fbc-kube-api-access-h6nx7\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.012668 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.012403 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 21:50:41.012668 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.012405 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4xbt\" (UniqueName: \"kubernetes.io/projected/b4105fd0-16f9-492e-ae10-1f7458524eeb-kube-api-access-t4xbt\") pod \"multus-additional-cni-plugins-d4fs7\" (UID: \"b4105fd0-16f9-492e-ae10-1f7458524eeb\") " pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:50:41.012668 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.012382 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 21:50:41.012668 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.012483 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:50:41.013297 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.013027 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-metrics-server-audit-profiles\") pod \"metrics-server-d6fb658f8-zvqw4\" (UID: \"e321c1b1-e9e2-46fc-9a35-14a3d6bed233\") " pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" Oct 30 21:50:41.013297 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.013039 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-host-run-k8s-cni-cncf-io\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.013297 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.013090 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-host-var-lib-cni-multus\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.014365 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.014336 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ad6cfef-ff22-4eb5-8910-a761537d737f-tmp-dir\") pod \"dns-default-zf4n4\" (UID: \"0ad6cfef-ff22-4eb5-8910-a761537d737f\") " pod="openshift-dns/dns-default-zf4n4" Oct 30 21:50:41.014485 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.014394 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-sysconfig\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.014485 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.014428 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f5d7f625-40b0-46b6-8489-fdd1daa0068e-etc-selinux\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:50:41.014485 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.014464 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:41.014485 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.014463 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ad6cfef-ff22-4eb5-8910-a761537d737f-tmp-dir\") pod \"dns-default-zf4n4\" (UID: \"0ad6cfef-ff22-4eb5-8910-a761537d737f\") " pod="openshift-dns/dns-default-zf4n4" Oct 30 21:50:41.014790 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.014525 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/579c3ec8-fe12-472f-8ab0-247bfa8ed8b7-tmp-dir\") pod \"node-resolver-8qxjv\" (UID: \"579c3ec8-fe12-472f-8ab0-247bfa8ed8b7\") " pod="openshift-dns/node-resolver-8qxjv" Oct 30 21:50:41.014790 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.014569 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-cnibin\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.014790 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.014706 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/579c3ec8-fe12-472f-8ab0-247bfa8ed8b7-tmp-dir\") pod \"node-resolver-8qxjv\" (UID: \"579c3ec8-fe12-472f-8ab0-247bfa8ed8b7\") " pod="openshift-dns/node-resolver-8qxjv" Oct 30 21:50:41.014790 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.014710 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-run-openvswitch\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.014790 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.014749 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-config-out\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:41.014790 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.014790 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:41.015500 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.014817 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-secret-grpc-tls\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:41.015500 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.014850 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-config\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.015500 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.014888 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.015500 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.014920 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.015500 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.014954 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-kubernetes\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.015500 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.014971 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-config-out\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:41.015500 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.014987 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-sys\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.015500 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.015043 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:41.015500 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.015285 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.015500 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.015354 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf4tt\" (UniqueName: \"kubernetes.io/projected/1e76444f-1a5b-460c-b42f-2197b7512574-kube-api-access-mf4tt\") pod \"kube-state-metrics-5dc8d7d89b-lbjd4\" (UID: \"1e76444f-1a5b-460c-b42f-2197b7512574\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4" Oct 30 21:50:41.015500 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.015385 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d2226976-1041-4798-b567-6fb6590c0f6f-multus-daemon-config\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.015500 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.015425 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-run-ovn-kubernetes\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.016577 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.015641 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d2226976-1041-4798-b567-6fb6590c0f6f-multus-daemon-config\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.016577 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.015830 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.016577 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.015887 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/18a7ec6e-13fa-4893-a179-529f13fb2172-agent-certs\") pod \"konnectivity-agent-hmx87\" (UID: \"18a7ec6e-13fa-4893-a179-529f13fb2172\") " pod="kube-system/konnectivity-agent-hmx87" Oct 30 21:50:41.016577 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.015947 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b4105fd0-16f9-492e-ae10-1f7458524eeb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d4fs7\" (UID: \"b4105fd0-16f9-492e-ae10-1f7458524eeb\") " pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:50:41.016577 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.016156 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e76444f-1a5b-460c-b42f-2197b7512574-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-lbjd4\" (UID: \"1e76444f-1a5b-460c-b42f-2197b7512574\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4" Oct 30 21:50:41.016577 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.016190 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/18a7ec6e-13fa-4893-a179-529f13fb2172-agent-certs\") pod \"konnectivity-agent-hmx87\" (UID: \"18a7ec6e-13fa-4893-a179-529f13fb2172\") " pod="kube-system/konnectivity-agent-hmx87" Oct 30 21:50:41.016577 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.016451 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 21:50:41.017352 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.017244 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 21:50:41.017352 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.017323 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 21:50:41.018533 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.018491 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.018665 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.018620 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-system-cni-dir\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.018726 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.018692 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b4105fd0-16f9-492e-ae10-1f7458524eeb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d4fs7\" (UID: \"b4105fd0-16f9-492e-ae10-1f7458524eeb\") " pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:50:41.018837 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.018793 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-multus-socket-dir-parent\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.018910 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.018894 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:41.019010 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.018991 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.019063 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.019028 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.019112 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.019065 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-sysctl-d\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.019112 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.019095 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-lib-modules\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.019281 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.019132 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-host\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.019334 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.019315 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d2226976-1041-4798-b567-6fb6590c0f6f-cni-binary-copy\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.019379 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.019359 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:41.019434 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.019400 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-node-exporter-accelerators-collector-config\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:41.019486 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.019413 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.019531 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.019484 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-multus-conf-dir\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.019531 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.019518 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7fd1e2d6-4877-4459-82f9-95b31d098229-ovnkube-config\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.019855 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.019807 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e1f2183b-a823-43ae-b7c8-c4b24dd3b403-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v6wts\" (UID: \"e1f2183b-a823-43ae-b7c8-c4b24dd3b403\") " pod="openshift-insights/insights-runtime-extractor-v6wts" Oct 30 21:50:41.019915 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.019877 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-node-exporter-accelerators-collector-config\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:41.019915 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.019891 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:41.019915 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.019878 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7fd1e2d6-4877-4459-82f9-95b31d098229-ovnkube-config\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.019915 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.019905 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d2226976-1041-4798-b567-6fb6590c0f6f-cni-binary-copy\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.020106 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.019930 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a4d3550-acb2-4cf1-b192-adf694522695-serving-certs-ca-bundle\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:41.020106 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.020003 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8a4d3550-acb2-4cf1-b192-adf694522695-federate-client-tls\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:41.020106 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.020038 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 21:50:41.020483 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.020453 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:41.020483 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.020471 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj8hr\" (UniqueName: \"kubernetes.io/projected/2e5352ac-8295-494d-950a-16c09b09272e-kube-api-access-dj8hr\") pod \"network-check-target-mg6qm\" (UID: \"2e5352ac-8295-494d-950a-16c09b09272e\") " pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:50:41.020591 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.020524 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b4105fd0-16f9-492e-ae10-1f7458524eeb-cni-binary-copy\") pod \"multus-additional-cni-plugins-d4fs7\" (UID: \"b4105fd0-16f9-492e-ae10-1f7458524eeb\") " pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:50:41.020591 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.020555 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:41.020591 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.020561 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a4d3550-acb2-4cf1-b192-adf694522695-serving-certs-ca-bundle\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:41.020591 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.020565 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e1f2183b-a823-43ae-b7c8-c4b24dd3b403-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v6wts\" (UID: \"e1f2183b-a823-43ae-b7c8-c4b24dd3b403\") " pod="openshift-insights/insights-runtime-extractor-v6wts" Oct 30 21:50:41.020591 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.020582 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp9j4\" (UniqueName: \"kubernetes.io/projected/83a9baf0-064d-4544-94b4-a0dd090dbd04-kube-api-access-qp9j4\") pod \"network-metrics-daemon-h4v24\" (UID: \"83a9baf0-064d-4544-94b4-a0dd090dbd04\") " pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:50:41.020776 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.020688 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctmq5\" (UniqueName: \"kubernetes.io/projected/e1f2183b-a823-43ae-b7c8-c4b24dd3b403-kube-api-access-ctmq5\") pod \"insights-runtime-extractor-v6wts\" (UID: \"e1f2183b-a823-43ae-b7c8-c4b24dd3b403\") " pod="openshift-insights/insights-runtime-extractor-v6wts" Oct 30 21:50:41.020776 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.020732 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a4d3550-acb2-4cf1-b192-adf694522695-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:41.020776 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.020758 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ad6cfef-ff22-4eb5-8910-a761537d737f-metrics-tls\") pod \"dns-default-zf4n4\" (UID: \"0ad6cfef-ff22-4eb5-8910-a761537d737f\") " pod="openshift-dns/dns-default-zf4n4" Oct 30 21:50:41.020900 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.020791 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f5d7f625-40b0-46b6-8489-fdd1daa0068e-device-dir\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:50:41.020900 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.020817 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:41.020900 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.020860 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-metrics-client-ca\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:41.020900 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.020876 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b4105fd0-16f9-492e-ae10-1f7458524eeb-cni-binary-copy\") pod \"multus-additional-cni-plugins-d4fs7\" (UID: \"b4105fd0-16f9-492e-ae10-1f7458524eeb\") " pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:50:41.021078 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.020945 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4630b412-da87-49f7-bbbb-ba3dc820ddd1-dbus\") pod \"global-pull-secret-syncer-xbwzg\" (UID: \"4630b412-da87-49f7-bbbb-ba3dc820ddd1\") " pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:50:41.021078 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.021000 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntp69\" (UniqueName: \"kubernetes.io/projected/0ad6cfef-ff22-4eb5-8910-a761537d737f-kube-api-access-ntp69\") pod \"dns-default-zf4n4\" (UID: \"0ad6cfef-ff22-4eb5-8910-a761537d737f\") " pod="openshift-dns/dns-default-zf4n4" Oct 30 21:50:41.021078 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.021003 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 21:50:41.021078 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.021052 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-hostroot\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.021249 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.021101 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-host-run-multus-certs\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.021249 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.021063 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-metrics-client-ca\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:41.021249 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.021127 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-run-ovn\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.021249 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.021167 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.021249 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.021243 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.021392 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.021273 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7b50ca6-0f9c-40f8-a09d-004b055105b8-host-slash\") pod \"iptables-alerter-q8gsv\" (UID: \"e7b50ca6-0f9c-40f8-a09d-004b055105b8\") " pod="openshift-network-operator/iptables-alerter-q8gsv" Oct 30 21:50:41.021392 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.021278 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ad6cfef-ff22-4eb5-8910-a761537d737f-metrics-tls\") pod \"dns-default-zf4n4\" (UID: \"0ad6cfef-ff22-4eb5-8910-a761537d737f\") " pod="openshift-dns/dns-default-zf4n4" Oct 30 21:50:41.021392 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.021296 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-host-var-lib-cni-bin\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.021392 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.021321 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp4k2\" (UniqueName: \"kubernetes.io/projected/7fd1e2d6-4877-4459-82f9-95b31d098229-kube-api-access-lp4k2\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.021504 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.021452 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c199354-31a1-45b4-8c60-a485fd65f1e2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-x796z\" (UID: \"3c199354-31a1-45b4-8c60-a485fd65f1e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x796z" Oct 30 21:50:41.021553 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.021515 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-cni-bin\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.021596 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.021579 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1e76444f-1a5b-460c-b42f-2197b7512574-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-lbjd4\" (UID: \"1e76444f-1a5b-460c-b42f-2197b7512574\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4" Oct 30 21:50:41.021639 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.021623 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.021689 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.021669 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c199354-31a1-45b4-8c60-a485fd65f1e2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-x796z\" (UID: \"3c199354-31a1-45b4-8c60-a485fd65f1e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x796z" Oct 30 21:50:41.021729 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.021709 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-sysctl-conf\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.021729 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.021717 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.021806 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.021717 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1e76444f-1a5b-460c-b42f-2197b7512574-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-lbjd4\" (UID: \"1e76444f-1a5b-460c-b42f-2197b7512574\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4" Oct 30 21:50:41.021863 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.021845 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-host-run-netns\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.021912 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.021895 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7163574d-9b90-4a4c-83c5-5ca2d95a6275-cert\") pod \"ingress-canary-7khqw\" (UID: \"7163574d-9b90-4a4c-83c5-5ca2d95a6275\") " pod="openshift-ingress-canary/ingress-canary-7khqw" Oct 30 21:50:41.021955 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.021940 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-config-volume\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:41.021987 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.021973 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/736951e5-1596-4606-94d9-80d8b50d35c3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.022034 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.022005 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.022081 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.022035 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nd2d\" (UniqueName: \"kubernetes.io/projected/736951e5-1596-4606-94d9-80d8b50d35c3-kube-api-access-6nd2d\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.022119 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.022066 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-modprobe-d\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.022119 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.022097 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7163574d-9b90-4a4c-83c5-5ca2d95a6275-cert\") pod \"ingress-canary-7khqw\" (UID: \"7163574d-9b90-4a4c-83c5-5ca2d95a6275\") " pod="openshift-ingress-canary/ingress-canary-7khqw" Oct 30 21:50:41.022119 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.022110 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/579c3ec8-fe12-472f-8ab0-247bfa8ed8b7-hosts-file\") pod \"node-resolver-8qxjv\" (UID: \"579c3ec8-fe12-472f-8ab0-247bfa8ed8b7\") " pod="openshift-dns/node-resolver-8qxjv" Oct 30 21:50:41.022294 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.022143 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6q6v\" (UniqueName: \"kubernetes.io/projected/9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8-kube-api-access-q6q6v\") pod \"node-ca-4dqsc\" (UID: \"9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8\") " pod="openshift-image-registry/node-ca-4dqsc" Oct 30 21:50:41.022294 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.022205 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-slash\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.022294 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.022268 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b4105fd0-16f9-492e-ae10-1f7458524eeb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d4fs7\" (UID: \"b4105fd0-16f9-492e-ae10-1f7458524eeb\") " pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:50:41.022406 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.022336 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-log-socket\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.022406 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.022372 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-web-config\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:41.022465 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.022404 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.022465 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.022420 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:41.022525 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.022459 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgqbt\" (UniqueName: \"kubernetes.io/projected/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-kube-api-access-sgqbt\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:41.022525 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.022486 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-var-lib-kubelet\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.022525 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.022496 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b4105fd0-16f9-492e-ae10-1f7458524eeb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d4fs7\" (UID: \"b4105fd0-16f9-492e-ae10-1f7458524eeb\") " pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:50:41.022525 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.022505 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8-host\") pod \"node-ca-4dqsc\" (UID: \"9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8\") " pod="openshift-image-registry/node-ca-4dqsc" Oct 30 21:50:41.022654 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.022606 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8-serviceca\") pod \"node-ca-4dqsc\" (UID: \"9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8\") " pod="openshift-image-registry/node-ca-4dqsc" Oct 30 21:50:41.022654 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.022623 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:41.022654 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.022626 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4lcd\" (UniqueName: \"kubernetes.io/projected/3c199354-31a1-45b4-8c60-a485fd65f1e2-kube-api-access-j4lcd\") pod \"openshift-state-metrics-7bd7489957-x796z\" (UID: \"3c199354-31a1-45b4-8c60-a485fd65f1e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x796z" Oct 30 21:50:41.022749 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.022668 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4630b412-da87-49f7-bbbb-ba3dc820ddd1-kubelet-config\") pod \"global-pull-secret-syncer-xbwzg\" (UID: \"4630b412-da87-49f7-bbbb-ba3dc820ddd1\") " pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:50:41.022749 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.022694 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-systemd\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.022809 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.022753 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfsd9\" (UniqueName: \"kubernetes.io/projected/579c3ec8-fe12-472f-8ab0-247bfa8ed8b7-kube-api-access-tfsd9\") pod \"node-resolver-8qxjv\" (UID: \"579c3ec8-fe12-472f-8ab0-247bfa8ed8b7\") " pod="openshift-dns/node-resolver-8qxjv" Oct 30 21:50:41.022842 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.022801 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkk9k\" (UniqueName: \"kubernetes.io/projected/f5d7f625-40b0-46b6-8489-fdd1daa0068e-kube-api-access-kkk9k\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:50:41.022882 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.022843 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-etc-openvswitch\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.022914 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.022883 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/18a7ec6e-13fa-4893-a179-529f13fb2172-konnectivity-ca\") pod \"konnectivity-agent-hmx87\" (UID: \"18a7ec6e-13fa-4893-a179-529f13fb2172\") " pod="kube-system/konnectivity-agent-hmx87" Oct 30 21:50:41.022950 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.022932 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-node-exporter-tls\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:41.022981 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.022959 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-node-exporter-wtmp\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:41.023030 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.022987 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c199354-31a1-45b4-8c60-a485fd65f1e2-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-x796z\" (UID: \"3c199354-31a1-45b4-8c60-a485fd65f1e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x796z" Oct 30 21:50:41.023079 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.023022 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-host-var-lib-kubelet\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.023128 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.023080 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-run-systemd\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.023176 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.023135 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:41.023176 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.023138 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/18a7ec6e-13fa-4893-a179-529f13fb2172-konnectivity-ca\") pod \"konnectivity-agent-hmx87\" (UID: \"18a7ec6e-13fa-4893-a179-529f13fb2172\") " pod="kube-system/konnectivity-agent-hmx87" Oct 30 21:50:41.023289 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.023175 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7nrk\" (UniqueName: \"kubernetes.io/projected/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-kube-api-access-q7nrk\") pod \"metrics-server-d6fb658f8-zvqw4\" (UID: \"e321c1b1-e9e2-46fc-9a35-14a3d6bed233\") " pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" Oct 30 21:50:41.023289 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.023266 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8-serviceca\") pod \"node-ca-4dqsc\" (UID: \"9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8\") " pod="openshift-image-registry/node-ca-4dqsc" Oct 30 21:50:41.023289 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.023271 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e76444f-1a5b-460c-b42f-2197b7512574-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-lbjd4\" (UID: \"1e76444f-1a5b-460c-b42f-2197b7512574\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4" Oct 30 21:50:41.023424 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.023312 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n67nc\" (UniqueName: \"kubernetes.io/projected/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-kube-api-access-n67nc\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:41.023424 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.023360 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.023424 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.023397 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:41.023424 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.023414 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c199354-31a1-45b4-8c60-a485fd65f1e2-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-x796z\" (UID: \"3c199354-31a1-45b4-8c60-a485fd65f1e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x796z" Oct 30 21:50:41.023608 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.023466 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4105fd0-16f9-492e-ae10-1f7458524eeb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d4fs7\" (UID: \"b4105fd0-16f9-492e-ae10-1f7458524eeb\") " pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:50:41.023608 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.023485 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-etc-kubernetes\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.023608 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.023516 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-client-ca-bundle\") pod \"metrics-server-d6fb658f8-zvqw4\" (UID: \"e321c1b1-e9e2-46fc-9a35-14a3d6bed233\") " pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" Oct 30 21:50:41.023608 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.023566 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e1f2183b-a823-43ae-b7c8-c4b24dd3b403-crio-socket\") pod \"insights-runtime-extractor-v6wts\" (UID: \"e1f2183b-a823-43ae-b7c8-c4b24dd3b403\") " pod="openshift-insights/insights-runtime-extractor-v6wts" Oct 30 21:50:41.023779 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.023617 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-tmp\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:41.023779 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.023645 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8a4d3550-acb2-4cf1-b192-adf694522695-secret-telemeter-client\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:41.023779 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.023673 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad6cfef-ff22-4eb5-8910-a761537d737f-config-volume\") pod \"dns-default-zf4n4\" (UID: \"0ad6cfef-ff22-4eb5-8910-a761537d737f\") " pod="openshift-dns/dns-default-zf4n4" Oct 30 21:50:41.023779 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.023689 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-run-netns\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.023779 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.023708 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7fd1e2d6-4877-4459-82f9-95b31d098229-env-overrides\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.023779 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.023721 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-tmp\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:41.023779 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.023747 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d962c24e-5855-453e-97d5-78bad4a19fbc-tmp\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.024064 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.023854 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7fd1e2d6-4877-4459-82f9-95b31d098229-env-overrides\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.024064 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.023861 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad6cfef-ff22-4eb5-8910-a761537d737f-config-volume\") pod \"dns-default-zf4n4\" (UID: \"0ad6cfef-ff22-4eb5-8910-a761537d737f\") " pod="openshift-dns/dns-default-zf4n4" Oct 30 21:50:41.024064 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.023867 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-web-config\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.024064 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.023892 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-run\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.024064 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.023880 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d962c24e-5855-453e-97d5-78bad4a19fbc-tmp\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.024064 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.023919 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-tuned\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.024064 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.023951 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4105fd0-16f9-492e-ae10-1f7458524eeb-system-cni-dir\") pod \"multus-additional-cni-plugins-d4fs7\" (UID: \"b4105fd0-16f9-492e-ae10-1f7458524eeb\") " pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:50:41.024064 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.023971 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-tuned\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.024064 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.023972 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-var-lib-openvswitch\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.024064 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.024004 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-secret-metrics-server-client-certs\") pod \"metrics-server-d6fb658f8-zvqw4\" (UID: \"e321c1b1-e9e2-46fc-9a35-14a3d6bed233\") " pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" Oct 30 21:50:41.024064 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.024024 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d6fb658f8-zvqw4\" (UID: \"e321c1b1-e9e2-46fc-9a35-14a3d6bed233\") " pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" Oct 30 21:50:41.024064 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.024041 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1f2183b-a823-43ae-b7c8-c4b24dd3b403-data-volume\") pod \"insights-runtime-extractor-v6wts\" (UID: \"e1f2183b-a823-43ae-b7c8-c4b24dd3b403\") " pod="openshift-insights/insights-runtime-extractor-v6wts" Oct 30 21:50:41.024064 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.024064 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/736951e5-1596-4606-94d9-80d8b50d35c3-config-out\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.024613 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.024079 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.024613 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.024097 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f5d7f625-40b0-46b6-8489-fdd1daa0068e-socket-dir\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:50:41.024613 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.024140 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1f2183b-a823-43ae-b7c8-c4b24dd3b403-data-volume\") pod \"insights-runtime-extractor-v6wts\" (UID: \"e1f2183b-a823-43ae-b7c8-c4b24dd3b403\") " pod="openshift-insights/insights-runtime-extractor-v6wts" Oct 30 21:50:41.024613 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.024165 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/736951e5-1596-4606-94d9-80d8b50d35c3-config-out\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.024613 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.024186 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:41.024613 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.024208 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1e76444f-1a5b-460c-b42f-2197b7512574-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-lbjd4\" (UID: \"1e76444f-1a5b-460c-b42f-2197b7512574\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4" Oct 30 21:50:41.024613 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.024246 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d6fb658f8-zvqw4\" (UID: \"e321c1b1-e9e2-46fc-9a35-14a3d6bed233\") " pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" Oct 30 21:50:41.024613 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.024251 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4630b412-da87-49f7-bbbb-ba3dc820ddd1-original-pull-secret\") pod \"global-pull-secret-syncer-xbwzg\" (UID: \"4630b412-da87-49f7-bbbb-ba3dc820ddd1\") " pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:50:41.024613 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.024295 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:41.024613 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.024298 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.024613 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.024439 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.024613 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.024470 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1e76444f-1a5b-460c-b42f-2197b7512574-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-lbjd4\" (UID: \"1e76444f-1a5b-460c-b42f-2197b7512574\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4" Oct 30 21:50:41.024613 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.024474 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-node-log\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.024613 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.024525 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7fd1e2d6-4877-4459-82f9-95b31d098229-ovn-node-metrics-cert\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.024613 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.024585 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp9qb\" (UniqueName: \"kubernetes.io/projected/d2226976-1041-4798-b567-6fb6590c0f6f-kube-api-access-pp9qb\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.024613 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.024588 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4630b412-da87-49f7-bbbb-ba3dc820ddd1-original-pull-secret\") pod \"global-pull-secret-syncer-xbwzg\" (UID: \"4630b412-da87-49f7-bbbb-ba3dc820ddd1\") " pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:50:41.025396 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.024625 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-cni-netd\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.025396 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.024661 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-audit-log\") pod \"metrics-server-d6fb658f8-zvqw4\" (UID: \"e321c1b1-e9e2-46fc-9a35-14a3d6bed233\") " pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" Oct 30 21:50:41.025396 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.024689 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-node-exporter-textfile\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:41.025396 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.024728 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3c199354-31a1-45b4-8c60-a485fd65f1e2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-x796z\" (UID: \"3c199354-31a1-45b4-8c60-a485fd65f1e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x796z" Oct 30 21:50:41.025396 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.024755 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:41.025396 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.024780 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k48m\" (UniqueName: \"kubernetes.io/projected/8a4d3550-acb2-4cf1-b192-adf694522695-kube-api-access-5k48m\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:41.025396 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.024793 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-node-exporter-textfile\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:41.025396 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.024807 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-sys\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:41.025396 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.024855 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-audit-log\") pod \"metrics-server-d6fb658f8-zvqw4\" (UID: \"e321c1b1-e9e2-46fc-9a35-14a3d6bed233\") " pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" Oct 30 21:50:41.025396 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.024856 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7fd1e2d6-4877-4459-82f9-95b31d098229-ovn-node-metrics-cert\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.025396 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.024873 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-metrics-client-ca\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:41.025396 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.024905 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e76444f-1a5b-460c-b42f-2197b7512574-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-lbjd4\" (UID: \"1e76444f-1a5b-460c-b42f-2197b7512574\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4" Oct 30 21:50:41.025396 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.024956 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f5d7f625-40b0-46b6-8489-fdd1daa0068e-sys-fs\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:50:41.025396 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.025060 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4105fd0-16f9-492e-ae10-1f7458524eeb-cnibin\") pod \"multus-additional-cni-plugins-d4fs7\" (UID: \"b4105fd0-16f9-492e-ae10-1f7458524eeb\") " pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:50:41.025396 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.025086 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e7b50ca6-0f9c-40f8-a09d-004b055105b8-iptables-alerter-script\") pod \"iptables-alerter-q8gsv\" (UID: \"e7b50ca6-0f9c-40f8-a09d-004b055105b8\") " pod="openshift-network-operator/iptables-alerter-q8gsv" Oct 30 21:50:41.025396 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.025093 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-metrics-client-ca\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:41.025396 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.025134 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.025396 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.025156 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.025396 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.025170 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-root\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:41.025396 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.025186 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g25nc\" (UniqueName: \"kubernetes.io/projected/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-kube-api-access-g25nc\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:41.025396 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.025202 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8a4d3550-acb2-4cf1-b192-adf694522695-telemeter-client-tls\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:41.025396 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.025269 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svfrj\" (UniqueName: \"kubernetes.io/projected/e7b50ca6-0f9c-40f8-a09d-004b055105b8-kube-api-access-svfrj\") pod \"iptables-alerter-q8gsv\" (UID: \"e7b50ca6-0f9c-40f8-a09d-004b055105b8\") " pod="openshift-network-operator/iptables-alerter-q8gsv" Oct 30 21:50:41.025396 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.025305 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-kubelet\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.025396 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.025304 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e7b50ca6-0f9c-40f8-a09d-004b055105b8-iptables-alerter-script\") pod \"iptables-alerter-q8gsv\" (UID: \"e7b50ca6-0f9c-40f8-a09d-004b055105b8\") " pod="openshift-network-operator/iptables-alerter-q8gsv" Oct 30 21:50:41.025396 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.025360 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7fd1e2d6-4877-4459-82f9-95b31d098229-ovnkube-script-lib\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.025396 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.025388 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd8ns\" (UniqueName: \"kubernetes.io/projected/7163574d-9b90-4a4c-83c5-5ca2d95a6275-kube-api-access-zd8ns\") pod \"ingress-canary-7khqw\" (UID: \"7163574d-9b90-4a4c-83c5-5ca2d95a6275\") " pod="openshift-ingress-canary/ingress-canary-7khqw" Oct 30 21:50:41.026261 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.025424 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a9baf0-064d-4544-94b4-a0dd090dbd04-metrics-certs\") pod \"network-metrics-daemon-h4v24\" (UID: \"83a9baf0-064d-4544-94b4-a0dd090dbd04\") " pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:50:41.026261 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.025481 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-secret-thanos-querier-tls\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:41.026261 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.025517 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:41.026261 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.025519 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:41.026261 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.025573 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a4d3550-acb2-4cf1-b192-adf694522695-metrics-client-ca\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:41.026261 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.025596 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7fd1e2d6-4877-4459-82f9-95b31d098229-ovnkube-script-lib\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.026261 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.025611 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f5d7f625-40b0-46b6-8489-fdd1daa0068e-registration-dir\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:50:41.026261 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.025658 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-secret-metrics-server-tls\") pod \"metrics-server-d6fb658f8-zvqw4\" (UID: \"e321c1b1-e9e2-46fc-9a35-14a3d6bed233\") " pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" Oct 30 21:50:41.026261 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.025684 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e1f2183b-a823-43ae-b7c8-c4b24dd3b403-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v6wts\" (UID: \"e1f2183b-a823-43ae-b7c8-c4b24dd3b403\") " pod="openshift-insights/insights-runtime-extractor-v6wts" Oct 30 21:50:41.026261 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.025686 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a9baf0-064d-4544-94b4-a0dd090dbd04-metrics-certs\") pod \"network-metrics-daemon-h4v24\" (UID: \"83a9baf0-064d-4544-94b4-a0dd090dbd04\") " pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:50:41.026261 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.025751 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:41.026261 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.025788 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a4d3550-acb2-4cf1-b192-adf694522695-metrics-client-ca\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:41.026261 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.025784 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a4d3550-acb2-4cf1-b192-adf694522695-telemeter-trusted-ca-bundle\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:41.026261 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.025867 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e1f2183b-a823-43ae-b7c8-c4b24dd3b403-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v6wts\" (UID: \"e1f2183b-a823-43ae-b7c8-c4b24dd3b403\") " pod="openshift-insights/insights-runtime-extractor-v6wts" Oct 30 21:50:41.028905 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.028886 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.059867 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.059806 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 21:50:41.062252 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.062234 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 21:50:41.062321 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.062298 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 21:50:41.062413 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.062386 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 21:50:41.062467 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.062450 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 21:50:41.062554 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.062540 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 21:50:41.062707 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.062691 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 21:50:41.062766 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.062738 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 21:50:41.062766 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.062738 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-dhvch\"" Oct 30 21:50:41.062856 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.062769 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 21:50:41.062856 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.062805 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-9vsls\"" Oct 30 21:50:41.062856 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.062842 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 21:50:41.062977 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.062863 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-esign78lrs486\"" Oct 30 21:50:41.062977 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.062884 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 21:50:41.062977 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.062891 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-mcdr9\"" Oct 30 21:50:41.062977 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.062906 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 21:50:41.062977 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.062894 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-bmzgm\"" Oct 30 21:50:41.063322 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.063038 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9jiopb23p179d\"" Oct 30 21:50:41.063322 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.063105 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 21:50:41.063322 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.063109 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 21:50:41.063322 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.063114 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-3a3p8gb9pmm2\"" Oct 30 21:50:41.063322 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.063108 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 21:50:41.063322 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.063192 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-78zg5\"" Oct 30 21:50:41.063322 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.063202 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-t5jsh\"" Oct 30 21:50:41.063322 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.063202 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 21:50:41.063322 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.063244 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 21:50:41.063322 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.063267 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 21:50:41.063322 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.063278 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 21:50:41.063322 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.063287 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 21:50:41.063322 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.063317 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 21:50:41.063322 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.063330 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 21:50:41.063893 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.063358 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 21:50:41.063893 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.063383 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 21:50:41.063893 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.063402 5918 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:50:41.063893 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.063330 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 21:50:41.063893 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.063424 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-45gdr\"" Oct 30 21:50:41.063893 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.063430 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 21:50:41.063893 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.063451 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 21:50:41.063893 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.063453 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 21:50:41.063893 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.063466 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 21:50:41.063893 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.063483 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 21:50:41.063893 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.063498 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 21:50:41.063893 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.063516 5918 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:50:41.063893 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.063562 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-v5k6j\"" Oct 30 21:50:41.063893 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:50:41.063612 5918 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:50:41.063893 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.063719 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 21:50:41.063893 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.063732 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 21:50:41.063893 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.063772 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.063893 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.063814 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-client-ca-bundle\") pod \"metrics-server-d6fb658f8-zvqw4\" (UID: \"e321c1b1-e9e2-46fc-9a35-14a3d6bed233\") " pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" Oct 30 21:50:41.063893 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.063843 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 21:50:41.063893 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:50:41.063858 5918 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-81.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-81.us-west-1.compute.internal" Oct 30 21:50:41.064852 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.063922 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 21:50:41.064852 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.064052 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8a4d3550-acb2-4cf1-b192-adf694522695-secret-telemeter-client\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:41.064852 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.064063 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-web-config\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.064852 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.064178 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-secret-metrics-server-client-certs\") pod \"metrics-server-d6fb658f8-zvqw4\" (UID: \"e321c1b1-e9e2-46fc-9a35-14a3d6bed233\") " pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" Oct 30 21:50:41.064852 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.064251 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-4k7qj\"" Oct 30 21:50:41.064852 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.064392 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.064852 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.064642 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.064852 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.064747 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.064852 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.064830 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:41.065168 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.064970 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:41.065168 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.064996 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3c199354-31a1-45b4-8c60-a485fd65f1e2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-x796z\" (UID: \"3c199354-31a1-45b4-8c60-a485fd65f1e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x796z" Oct 30 21:50:41.066125 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.065948 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:41.066125 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.065974 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-secret-metrics-server-tls\") pod \"metrics-server-d6fb658f8-zvqw4\" (UID: \"e321c1b1-e9e2-46fc-9a35-14a3d6bed233\") " pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" Oct 30 21:50:41.066125 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.065982 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:41.066125 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.066004 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e76444f-1a5b-460c-b42f-2197b7512574-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-lbjd4\" (UID: \"1e76444f-1a5b-460c-b42f-2197b7512574\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4" Oct 30 21:50:41.066125 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.066031 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-secret-thanos-querier-tls\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:41.066125 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.066040 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.066125 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.066045 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.066125 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.066068 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8a4d3550-acb2-4cf1-b192-adf694522695-telemeter-client-tls\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:41.066125 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.066094 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:41.066457 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.066165 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-secret-grpc-tls\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:41.066457 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.066349 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-config\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.066697 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.066680 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a4d3550-acb2-4cf1-b192-adf694522695-telemeter-trusted-ca-bundle\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:41.069460 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.069442 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.069518 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.069445 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:41.070526 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.070509 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:41.071660 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.071642 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:41.071722 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.071697 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8a4d3550-acb2-4cf1-b192-adf694522695-federate-client-tls\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:41.071771 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.071758 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:41.071814 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.071779 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a4d3550-acb2-4cf1-b192-adf694522695-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:41.072936 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.072915 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.073014 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.072931 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/736951e5-1596-4606-94d9-80d8b50d35c3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.073014 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.072924 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-web-config\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:41.073014 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.072933 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-config-volume\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:41.073014 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.072970 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/290eb94c-6ae5-40e8-8ba3-dbe017243d9d-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-dd5t2\" (UID: \"290eb94c-6ae5-40e8-8ba3-dbe017243d9d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dd5t2" Oct 30 21:50:41.074900 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.074264 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-node-exporter-tls\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:41.074900 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.074285 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e76444f-1a5b-460c-b42f-2197b7512574-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-lbjd4\" (UID: \"1e76444f-1a5b-460c-b42f-2197b7512574\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4" Oct 30 21:50:41.075010 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.074911 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4xbt\" (UniqueName: \"kubernetes.io/projected/b4105fd0-16f9-492e-ae10-1f7458524eeb-kube-api-access-t4xbt\") pod \"multus-additional-cni-plugins-d4fs7\" (UID: \"b4105fd0-16f9-492e-ae10-1f7458524eeb\") " pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:50:41.075010 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.074940 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6nx7\" (UniqueName: \"kubernetes.io/projected/d962c24e-5855-453e-97d5-78bad4a19fbc-kube-api-access-h6nx7\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.075075 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.075037 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58wwh\" (UniqueName: \"kubernetes.io/projected/290eb94c-6ae5-40e8-8ba3-dbe017243d9d-kube-api-access-58wwh\") pod \"monitoring-plugin-77f4f76594-dd5t2\" (UID: \"290eb94c-6ae5-40e8-8ba3-dbe017243d9d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dd5t2" Oct 30 21:50:41.075526 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.075511 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.076720 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.076703 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf4tt\" (UniqueName: \"kubernetes.io/projected/1e76444f-1a5b-460c-b42f-2197b7512574-kube-api-access-mf4tt\") pod \"kube-state-metrics-5dc8d7d89b-lbjd4\" (UID: \"1e76444f-1a5b-460c-b42f-2197b7512574\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbjd4" Oct 30 21:50:41.081756 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.081735 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp9j4\" (UniqueName: \"kubernetes.io/projected/83a9baf0-064d-4544-94b4-a0dd090dbd04-kube-api-access-qp9j4\") pod \"network-metrics-daemon-h4v24\" (UID: \"83a9baf0-064d-4544-94b4-a0dd090dbd04\") " pod="openshift-multus/network-metrics-daemon-h4v24" Oct 30 21:50:41.082241 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.082202 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntp69\" (UniqueName: \"kubernetes.io/projected/0ad6cfef-ff22-4eb5-8910-a761537d737f-kube-api-access-ntp69\") pod \"dns-default-zf4n4\" (UID: \"0ad6cfef-ff22-4eb5-8910-a761537d737f\") " pod="openshift-dns/dns-default-zf4n4" Oct 30 21:50:41.082345 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.082329 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctmq5\" (UniqueName: \"kubernetes.io/projected/e1f2183b-a823-43ae-b7c8-c4b24dd3b403-kube-api-access-ctmq5\") pod \"insights-runtime-extractor-v6wts\" (UID: \"e1f2183b-a823-43ae-b7c8-c4b24dd3b403\") " pod="openshift-insights/insights-runtime-extractor-v6wts" Oct 30 21:50:41.082383 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.082331 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp4k2\" (UniqueName: \"kubernetes.io/projected/7fd1e2d6-4877-4459-82f9-95b31d098229-kube-api-access-lp4k2\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.082519 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.082502 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj8hr\" (UniqueName: \"kubernetes.io/projected/2e5352ac-8295-494d-950a-16c09b09272e-kube-api-access-dj8hr\") pod \"network-check-target-mg6qm\" (UID: \"2e5352ac-8295-494d-950a-16c09b09272e\") " pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:50:41.082958 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.082942 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nd2d\" (UniqueName: \"kubernetes.io/projected/736951e5-1596-4606-94d9-80d8b50d35c3-kube-api-access-6nd2d\") pod \"prometheus-k8s-0\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:41.083656 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.083628 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkk9k\" (UniqueName: \"kubernetes.io/projected/f5d7f625-40b0-46b6-8489-fdd1daa0068e-kube-api-access-kkk9k\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:50:41.087265 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.087240 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n67nc\" (UniqueName: \"kubernetes.io/projected/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-kube-api-access-n67nc\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:41.087448 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.087428 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6q6v\" (UniqueName: \"kubernetes.io/projected/9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8-kube-api-access-q6q6v\") pod \"node-ca-4dqsc\" (UID: \"9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8\") " pod="openshift-image-registry/node-ca-4dqsc" Oct 30 21:50:41.087496 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.087452 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfsd9\" (UniqueName: \"kubernetes.io/projected/579c3ec8-fe12-472f-8ab0-247bfa8ed8b7-kube-api-access-tfsd9\") pod \"node-resolver-8qxjv\" (UID: \"579c3ec8-fe12-472f-8ab0-247bfa8ed8b7\") " pod="openshift-dns/node-resolver-8qxjv" Oct 30 21:50:41.087528 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.087489 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4lcd\" (UniqueName: \"kubernetes.io/projected/3c199354-31a1-45b4-8c60-a485fd65f1e2-kube-api-access-j4lcd\") pod \"openshift-state-metrics-7bd7489957-x796z\" (UID: \"3c199354-31a1-45b4-8c60-a485fd65f1e2\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x796z" Oct 30 21:50:41.087528 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.087497 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgqbt\" (UniqueName: \"kubernetes.io/projected/f4925cf0-04b5-4ca4-b284-c0355e63b4ca-kube-api-access-sgqbt\") pod \"thanos-querier-796d66f799-qtc4t\" (UID: \"f4925cf0-04b5-4ca4-b284-c0355e63b4ca\") " pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:41.087705 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.087690 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7nrk\" (UniqueName: \"kubernetes.io/projected/e321c1b1-e9e2-46fc-9a35-14a3d6bed233-kube-api-access-q7nrk\") pod \"metrics-server-d6fb658f8-zvqw4\" (UID: \"e321c1b1-e9e2-46fc-9a35-14a3d6bed233\") " pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" Oct 30 21:50:41.089404 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.089387 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd8ns\" (UniqueName: \"kubernetes.io/projected/7163574d-9b90-4a4c-83c5-5ca2d95a6275-kube-api-access-zd8ns\") pod \"ingress-canary-7khqw\" (UID: \"7163574d-9b90-4a4c-83c5-5ca2d95a6275\") " pod="openshift-ingress-canary/ingress-canary-7khqw" Oct 30 21:50:41.089715 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.089696 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp9qb\" (UniqueName: \"kubernetes.io/projected/d2226976-1041-4798-b567-6fb6590c0f6f-kube-api-access-pp9qb\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.090028 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.090012 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k48m\" (UniqueName: \"kubernetes.io/projected/8a4d3550-acb2-4cf1-b192-adf694522695-kube-api-access-5k48m\") pod \"telemeter-client-89884678b-sxtjs\" (UID: \"8a4d3550-acb2-4cf1-b192-adf694522695\") " pod="openshift-monitoring/telemeter-client-89884678b-sxtjs" Oct 30 21:50:41.090141 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.090127 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svfrj\" (UniqueName: \"kubernetes.io/projected/e7b50ca6-0f9c-40f8-a09d-004b055105b8-kube-api-access-svfrj\") pod \"iptables-alerter-q8gsv\" (UID: \"e7b50ca6-0f9c-40f8-a09d-004b055105b8\") " pod="openshift-network-operator/iptables-alerter-q8gsv" Oct 30 21:50:41.090309 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.090293 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g25nc\" (UniqueName: \"kubernetes.io/projected/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-kube-api-access-g25nc\") pod \"alertmanager-main-0\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:50:41.126929 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.126903 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-cnibin\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.127008 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.126935 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-run-openvswitch\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.127008 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.126954 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-kubernetes\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.127008 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.126967 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-sys\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.127008 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.126984 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-run-ovn-kubernetes\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.127008 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127000 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-system-cni-dir\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.127195 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127014 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-multus-socket-dir-parent\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.127195 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127026 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-cnibin\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.127195 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127048 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-sys\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.127195 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127028 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-run-openvswitch\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.127195 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127043 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-kubernetes\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.127195 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127074 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-multus-socket-dir-parent\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.127195 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127085 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-system-cni-dir\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.127195 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127034 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-sysctl-d\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.127195 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127123 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-lib-modules\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.127195 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127145 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-host\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.127195 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127150 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-sysctl-d\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.127195 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127070 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-run-ovn-kubernetes\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.127195 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127174 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-host\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.127195 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127174 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-multus-conf-dir\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.127195 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127194 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-lib-modules\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.127626 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127197 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-multus-conf-dir\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.127626 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127207 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f5d7f625-40b0-46b6-8489-fdd1daa0068e-device-dir\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:50:41.127626 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127257 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4630b412-da87-49f7-bbbb-ba3dc820ddd1-dbus\") pod \"global-pull-secret-syncer-xbwzg\" (UID: \"4630b412-da87-49f7-bbbb-ba3dc820ddd1\") " pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:50:41.127626 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127273 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-hostroot\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.127626 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127256 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f5d7f625-40b0-46b6-8489-fdd1daa0068e-device-dir\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:50:41.127626 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127293 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-host-run-multus-certs\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.127626 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127330 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-host-run-multus-certs\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.127626 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127335 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4630b412-da87-49f7-bbbb-ba3dc820ddd1-dbus\") pod \"global-pull-secret-syncer-xbwzg\" (UID: \"4630b412-da87-49f7-bbbb-ba3dc820ddd1\") " pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:50:41.127626 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127340 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-hostroot\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.127626 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127335 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-run-ovn\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.127626 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127364 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-run-ovn\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.127626 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127387 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.127626 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127409 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7b50ca6-0f9c-40f8-a09d-004b055105b8-host-slash\") pod \"iptables-alerter-q8gsv\" (UID: \"e7b50ca6-0f9c-40f8-a09d-004b055105b8\") " pod="openshift-network-operator/iptables-alerter-q8gsv" Oct 30 21:50:41.127626 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127423 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-host-var-lib-cni-bin\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.127626 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127426 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.127626 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127446 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-cni-bin\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.127626 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127450 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-host-var-lib-cni-bin\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.127626 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127455 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7b50ca6-0f9c-40f8-a09d-004b055105b8-host-slash\") pod \"iptables-alerter-q8gsv\" (UID: \"e7b50ca6-0f9c-40f8-a09d-004b055105b8\") " pod="openshift-network-operator/iptables-alerter-q8gsv" Oct 30 21:50:41.127626 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127468 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-sysctl-conf\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.127626 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127482 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-host-run-netns\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.127626 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127500 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-modprobe-d\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.127626 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127512 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-cni-bin\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.127626 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127516 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/579c3ec8-fe12-472f-8ab0-247bfa8ed8b7-hosts-file\") pod \"node-resolver-8qxjv\" (UID: \"579c3ec8-fe12-472f-8ab0-247bfa8ed8b7\") " pod="openshift-dns/node-resolver-8qxjv" Oct 30 21:50:41.127626 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127533 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-slash\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.127626 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127552 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-log-socket\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.127626 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127565 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-host-run-netns\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.127626 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127567 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/579c3ec8-fe12-472f-8ab0-247bfa8ed8b7-hosts-file\") pod \"node-resolver-8qxjv\" (UID: \"579c3ec8-fe12-472f-8ab0-247bfa8ed8b7\") " pod="openshift-dns/node-resolver-8qxjv" Oct 30 21:50:41.127626 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127577 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-var-lib-kubelet\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.127626 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127583 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-slash\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.127626 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127585 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-modprobe-d\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.127626 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127598 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-log-socket\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.127626 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127599 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8-host\") pod \"node-ca-4dqsc\" (UID: \"9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8\") " pod="openshift-image-registry/node-ca-4dqsc" Oct 30 21:50:41.127626 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127625 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-sysctl-conf\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.127626 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127641 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4630b412-da87-49f7-bbbb-ba3dc820ddd1-kubelet-config\") pod \"global-pull-secret-syncer-xbwzg\" (UID: \"4630b412-da87-49f7-bbbb-ba3dc820ddd1\") " pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127646 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-var-lib-kubelet\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127643 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8-host\") pod \"node-ca-4dqsc\" (UID: \"9478ab3b-4826-4ac9-9b7d-0f907dd4c7f8\") " pod="openshift-image-registry/node-ca-4dqsc" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127675 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-systemd\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127685 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4630b412-da87-49f7-bbbb-ba3dc820ddd1-kubelet-config\") pod \"global-pull-secret-syncer-xbwzg\" (UID: \"4630b412-da87-49f7-bbbb-ba3dc820ddd1\") " pod="kube-system/global-pull-secret-syncer-xbwzg" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127695 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-etc-openvswitch\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127711 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-node-exporter-wtmp\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127724 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-systemd\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127729 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-host-var-lib-kubelet\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127754 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-node-exporter-wtmp\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127752 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-run-systemd\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127764 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-host-var-lib-kubelet\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127725 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-etc-openvswitch\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127779 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4105fd0-16f9-492e-ae10-1f7458524eeb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d4fs7\" (UID: \"b4105fd0-16f9-492e-ae10-1f7458524eeb\") " pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127784 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-run-systemd\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127807 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4105fd0-16f9-492e-ae10-1f7458524eeb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d4fs7\" (UID: \"b4105fd0-16f9-492e-ae10-1f7458524eeb\") " pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127813 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-etc-kubernetes\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127834 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e1f2183b-a823-43ae-b7c8-c4b24dd3b403-crio-socket\") pod \"insights-runtime-extractor-v6wts\" (UID: \"e1f2183b-a823-43ae-b7c8-c4b24dd3b403\") " pod="openshift-insights/insights-runtime-extractor-v6wts" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127850 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-run-netns\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127867 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-run\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127894 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4105fd0-16f9-492e-ae10-1f7458524eeb-system-cni-dir\") pod \"multus-additional-cni-plugins-d4fs7\" (UID: \"b4105fd0-16f9-492e-ae10-1f7458524eeb\") " pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127897 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-run-netns\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127901 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-etc-kubernetes\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127918 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-var-lib-openvswitch\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127932 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e1f2183b-a823-43ae-b7c8-c4b24dd3b403-crio-socket\") pod \"insights-runtime-extractor-v6wts\" (UID: \"e1f2183b-a823-43ae-b7c8-c4b24dd3b403\") " pod="openshift-insights/insights-runtime-extractor-v6wts" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127945 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4105fd0-16f9-492e-ae10-1f7458524eeb-system-cni-dir\") pod \"multus-additional-cni-plugins-d4fs7\" (UID: \"b4105fd0-16f9-492e-ae10-1f7458524eeb\") " pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127947 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f5d7f625-40b0-46b6-8489-fdd1daa0068e-socket-dir\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127977 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-node-log\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127978 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-run\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127975 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-var-lib-openvswitch\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.127983 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f5d7f625-40b0-46b6-8489-fdd1daa0068e-socket-dir\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.128000 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-node-log\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.128001 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-cni-netd\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.128018 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-cni-netd\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.128026 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-sys\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.128042 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f5d7f625-40b0-46b6-8489-fdd1daa0068e-sys-fs\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.128056 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4105fd0-16f9-492e-ae10-1f7458524eeb-cnibin\") pod \"multus-additional-cni-plugins-d4fs7\" (UID: \"b4105fd0-16f9-492e-ae10-1f7458524eeb\") " pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.128060 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-sys\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.128073 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-root\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.128088 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-kubelet\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.128093 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4105fd0-16f9-492e-ae10-1f7458524eeb-cnibin\") pod \"multus-additional-cni-plugins-d4fs7\" (UID: \"b4105fd0-16f9-492e-ae10-1f7458524eeb\") " pod="openshift-multus/multus-additional-cni-plugins-d4fs7" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.128089 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f5d7f625-40b0-46b6-8489-fdd1daa0068e-sys-fs\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.128107 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f5d7f625-40b0-46b6-8489-fdd1daa0068e-registration-dir\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.128114 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-host-kubelet\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.128123 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4a7edc22-1eb1-4458-8f8c-6c46a2bd583c-root\") pod \"node-exporter-q5c65\" (UID: \"4a7edc22-1eb1-4458-8f8c-6c46a2bd583c\") " pod="openshift-monitoring/node-exporter-q5c65" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.128131 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f5d7f625-40b0-46b6-8489-fdd1daa0068e-registration-dir\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.128141 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f5d7f625-40b0-46b6-8489-fdd1daa0068e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.128160 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-multus-cni-dir\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.128175 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-systemd-units\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.128188 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f5d7f625-40b0-46b6-8489-fdd1daa0068e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.128194 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-host-run-k8s-cni-cncf-io\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.128241 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-host-var-lib-cni-multus\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.128249 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-host-run-k8s-cni-cncf-io\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.128254 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-multus-cni-dir\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.128253 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7fd1e2d6-4877-4459-82f9-95b31d098229-systemd-units\") pod \"ovnkube-node-6th89\" (UID: \"7fd1e2d6-4877-4459-82f9-95b31d098229\") " pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.128265 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-sysconfig\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.128283 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d2226976-1041-4798-b567-6fb6590c0f6f-host-var-lib-cni-multus\") pod \"multus-b7k9p\" (UID: \"d2226976-1041-4798-b567-6fb6590c0f6f\") " pod="openshift-multus/multus-b7k9p" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.128288 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f5d7f625-40b0-46b6-8489-fdd1daa0068e-etc-selinux\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.128295 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d962c24e-5855-453e-97d5-78bad4a19fbc-etc-sysconfig\") pod \"tuned-zxdbg\" (UID: \"d962c24e-5855-453e-97d5-78bad4a19fbc\") " pod="openshift-cluster-node-tuning-operator/tuned-zxdbg" Oct 30 21:50:41.128558 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:41.128352 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f5d7f625-40b0-46b6-8489-fdd1daa0068e-etc-selinux\") pod \"aws-ebs-csi-driver-node-2vv8b\" (UID: \"f5d7f625-40b0-46b6-8489-fdd1daa0068e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2vv8b" Oct 30 21:50:42.874917 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:42.874885 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dd5t2" Oct 30 21:50:43.743784 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:43.743752 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zf4n4" Oct 30 21:50:44.130467 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:44.130370 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:44.135748 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:44.135724 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hmx87" Oct 30 21:50:44.149009 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:44.148985 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-796d66f799-qtc4t" Oct 30 21:50:44.150103 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:44.150082 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:50:47.508453 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:47.508424 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:50.262151 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:50.262072 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:50:50.904642 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:50.904595 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:50.904894 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:50.904813 5918 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:50:51.273484 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:50:51.273447 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 21:50:55.902622 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:55.902586 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6th89" Oct 30 21:50:58.451421 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:50:58.451390 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:06.944067 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:06.944037 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-d6fb658f8-zvqw4" Oct 30 21:51:10.265500 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:10.265471 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mg6qm" Oct 30 21:51:16.274725 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.274693 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 21:51:16.275246 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.275022 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" containerName="alertmanager" containerID="cri-o://ce58dbf55668090f8fd32f2003f91b73cf879f6b4b2c2073b1f40f9c401d6978" gracePeriod=120 Oct 30 21:51:16.275246 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.275085 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" containerName="kube-rbac-proxy-metric" containerID="cri-o://6834d44b6c689c55ee085cf407a92ddb84232029fb7ed209e68358efa2240328" gracePeriod=120 Oct 30 21:51:16.275246 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.275122 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" containerName="prom-label-proxy" containerID="cri-o://37b4de03de908b697a0b88f5f516f1218143ae6f9cf5322c8245d4d3ab69948a" gracePeriod=120 Oct 30 21:51:16.276062 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.276012 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" containerName="kube-rbac-proxy" containerID="cri-o://5105322e8d9ba68c630906594410279a6ffa306bdf34adbcbf9d72aa63c2d5a2" gracePeriod=120 Oct 30 21:51:16.276823 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.276767 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" containerName="config-reloader" containerID="cri-o://080d93f93f7820f56af52a80a4e3d83c84dac6d82b71772824d68d0457d5bef7" gracePeriod=120 Oct 30 21:51:16.280753 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.280729 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" containerName="kube-rbac-proxy-web" containerID="cri-o://235067a08860012a63a021cead0fdc6bc25c7fe1b7fe749b46b504ae96a7e6cd" gracePeriod=120 Oct 30 21:51:16.532701 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.532649 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:16.645139 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.645106 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-kube-rbac-proxy-web\") pod \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " Oct 30 21:51:16.645341 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.645155 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-config-out\" (UniqueName: \"kubernetes.io/empty-dir/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-config-out\") pod \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " Oct 30 21:51:16.645341 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.645185 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-kube-rbac-proxy-metric\") pod \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " Oct 30 21:51:16.645341 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.645240 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-cluster-tls-config\") pod \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " Oct 30 21:51:16.645341 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.645282 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-tls-assets\" (UniqueName: \"kubernetes.io/projected/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-tls-assets\") pod \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " Oct 30 21:51:16.645341 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.645325 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-alertmanager-main-db\") pod \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " Oct 30 21:51:16.645652 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.645385 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-kube-rbac-proxy\") pod \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " Oct 30 21:51:16.645652 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.645424 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-alertmanager-trusted-ca-bundle\") pod \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " Oct 30 21:51:16.645652 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.645465 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-config-volume\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-config-volume\") pod \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " Oct 30 21:51:16.645652 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.645494 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-main-tls\") pod \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " Oct 30 21:51:16.645652 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.645524 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-metrics-client-ca\") pod \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " Oct 30 21:51:16.645652 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.645567 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-kube-api-access-g25nc\" (UniqueName: \"kubernetes.io/projected/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-kube-api-access-g25nc\") pod \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " Oct 30 21:51:16.645652 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.645596 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-web-config\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-web-config\") pod \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\" (UID: \"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735\") " Oct 30 21:51:16.646036 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.646000 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-alertmanager-main-db" pod "9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" (UID: "9e62c7d9-71b5-4f9b-a587-aaf06c7b0735"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:16.646517 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.646486 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-metrics-client-ca" pod "9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" (UID: "9e62c7d9-71b5-4f9b-a587-aaf06c7b0735"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:16.648514 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.648447 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-alertmanager-trusted-ca-bundle" pod "9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" (UID: "9e62c7d9-71b5-4f9b-a587-aaf06c7b0735"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:16.648648 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.648596 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-kube-rbac-proxy-web" pod "9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" (UID: "9e62c7d9-71b5-4f9b-a587-aaf06c7b0735"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:16.648768 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.648732 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-tls-assets" pod "9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" (UID: "9e62c7d9-71b5-4f9b-a587-aaf06c7b0735"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:16.649993 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.649967 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-kube-rbac-proxy-metric" pod "9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" (UID: "9e62c7d9-71b5-4f9b-a587-aaf06c7b0735"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:16.650198 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.650154 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-config-out" pod "9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" (UID: "9e62c7d9-71b5-4f9b-a587-aaf06c7b0735"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:16.650337 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.650314 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-config-volume" pod "9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" (UID: "9e62c7d9-71b5-4f9b-a587-aaf06c7b0735"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:16.650534 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.650516 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-kube-api-access-g25nc" pod "9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" (UID: "9e62c7d9-71b5-4f9b-a587-aaf06c7b0735"). InnerVolumeSpecName "kube-api-access-g25nc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:16.650620 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.650602 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-main-tls" pod "9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" (UID: "9e62c7d9-71b5-4f9b-a587-aaf06c7b0735"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:16.650738 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.650698 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-kube-rbac-proxy" pod "9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" (UID: "9e62c7d9-71b5-4f9b-a587-aaf06c7b0735"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:16.653643 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.653539 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-cluster-tls-config" pod "9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" (UID: "9e62c7d9-71b5-4f9b-a587-aaf06c7b0735"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:16.659796 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.659776 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-web-config" pod "9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" (UID: "9e62c7d9-71b5-4f9b-a587-aaf06c7b0735"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:16.746774 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.746739 5918 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:16.746774 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.746767 5918 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-config-out\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:16.746774 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.746777 5918 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:16.746995 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.746788 5918 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-cluster-tls-config\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:16.746995 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.746797 5918 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-tls-assets\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:16.746995 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.746806 5918 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-alertmanager-main-db\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:16.746995 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.746816 5918 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:16.746995 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.746824 5918 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:16.746995 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.746833 5918 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-config-volume\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:16.746995 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.746841 5918 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-secret-alertmanager-main-tls\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:16.746995 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.746850 5918 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-metrics-client-ca\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:16.746995 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.746859 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g25nc\" (UniqueName: \"kubernetes.io/projected/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-kube-api-access-g25nc\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:16.746995 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:16.746869 5918 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735-web-config\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:17.061818 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.061780 5918 generic.go:356] "Generic (PLEG): container finished" podID="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" containerID="37b4de03de908b697a0b88f5f516f1218143ae6f9cf5322c8245d4d3ab69948a" exitCode=0 Oct 30 21:51:17.061818 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.061814 5918 generic.go:356] "Generic (PLEG): container finished" podID="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" containerID="6834d44b6c689c55ee085cf407a92ddb84232029fb7ed209e68358efa2240328" exitCode=0 Oct 30 21:51:17.061818 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.061824 5918 generic.go:356] "Generic (PLEG): container finished" podID="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" containerID="5105322e8d9ba68c630906594410279a6ffa306bdf34adbcbf9d72aa63c2d5a2" exitCode=0 Oct 30 21:51:17.062060 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.061832 5918 generic.go:356] "Generic (PLEG): container finished" podID="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" containerID="235067a08860012a63a021cead0fdc6bc25c7fe1b7fe749b46b504ae96a7e6cd" exitCode=0 Oct 30 21:51:17.062060 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.061841 5918 generic.go:356] "Generic (PLEG): container finished" podID="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" containerID="080d93f93f7820f56af52a80a4e3d83c84dac6d82b71772824d68d0457d5bef7" exitCode=0 Oct 30 21:51:17.062060 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.061852 5918 generic.go:356] "Generic (PLEG): container finished" podID="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" containerID="ce58dbf55668090f8fd32f2003f91b73cf879f6b4b2c2073b1f40f9c401d6978" exitCode=0 Oct 30 21:51:17.062060 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.061862 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735","Type":"ContainerDied","Data":"37b4de03de908b697a0b88f5f516f1218143ae6f9cf5322c8245d4d3ab69948a"} Oct 30 21:51:17.062060 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.061882 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.062060 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.061938 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735","Type":"ContainerDied","Data":"6834d44b6c689c55ee085cf407a92ddb84232029fb7ed209e68358efa2240328"} Oct 30 21:51:17.062060 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.061966 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735","Type":"ContainerDied","Data":"5105322e8d9ba68c630906594410279a6ffa306bdf34adbcbf9d72aa63c2d5a2"} Oct 30 21:51:17.062060 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.061992 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735","Type":"ContainerDied","Data":"235067a08860012a63a021cead0fdc6bc25c7fe1b7fe749b46b504ae96a7e6cd"} Oct 30 21:51:17.062060 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.062010 5918 scope.go:122] "RemoveContainer" containerID="37b4de03de908b697a0b88f5f516f1218143ae6f9cf5322c8245d4d3ab69948a" Oct 30 21:51:17.062060 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.062018 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735","Type":"ContainerDied","Data":"080d93f93f7820f56af52a80a4e3d83c84dac6d82b71772824d68d0457d5bef7"} Oct 30 21:51:17.062060 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.062043 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735","Type":"ContainerDied","Data":"ce58dbf55668090f8fd32f2003f91b73cf879f6b4b2c2073b1f40f9c401d6978"} Oct 30 21:51:17.062513 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.062072 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9e62c7d9-71b5-4f9b-a587-aaf06c7b0735","Type":"ContainerDied","Data":"f5e4b161dde81879deee20f513c5a60b7623438253853280a93b681288a12fb8"} Oct 30 21:51:17.070942 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.070926 5918 scope.go:122] "RemoveContainer" containerID="6834d44b6c689c55ee085cf407a92ddb84232029fb7ed209e68358efa2240328" Oct 30 21:51:17.077728 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.077711 5918 scope.go:122] "RemoveContainer" containerID="5105322e8d9ba68c630906594410279a6ffa306bdf34adbcbf9d72aa63c2d5a2" Oct 30 21:51:17.084270 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.084251 5918 scope.go:122] "RemoveContainer" containerID="235067a08860012a63a021cead0fdc6bc25c7fe1b7fe749b46b504ae96a7e6cd" Oct 30 21:51:17.091176 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.091154 5918 scope.go:122] "RemoveContainer" containerID="080d93f93f7820f56af52a80a4e3d83c84dac6d82b71772824d68d0457d5bef7" Oct 30 21:51:17.097854 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.097835 5918 scope.go:122] "RemoveContainer" containerID="ce58dbf55668090f8fd32f2003f91b73cf879f6b4b2c2073b1f40f9c401d6978" Oct 30 21:51:17.106190 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.106173 5918 scope.go:122] "RemoveContainer" containerID="b7029e8c3647fc8cf124a1d4e669b174d5c966d763a6a2cbc18befc88e3d1ffc" Oct 30 21:51:17.113258 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.113242 5918 scope.go:122] "RemoveContainer" containerID="37b4de03de908b697a0b88f5f516f1218143ae6f9cf5322c8245d4d3ab69948a" Oct 30 21:51:17.113560 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:51:17.113541 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37b4de03de908b697a0b88f5f516f1218143ae6f9cf5322c8245d4d3ab69948a\": container with ID starting with 37b4de03de908b697a0b88f5f516f1218143ae6f9cf5322c8245d4d3ab69948a not found: ID does not exist" containerID="37b4de03de908b697a0b88f5f516f1218143ae6f9cf5322c8245d4d3ab69948a" Oct 30 21:51:17.113610 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.113570 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37b4de03de908b697a0b88f5f516f1218143ae6f9cf5322c8245d4d3ab69948a"} err="failed to get container status \"37b4de03de908b697a0b88f5f516f1218143ae6f9cf5322c8245d4d3ab69948a\": rpc error: code = NotFound desc = could not find container \"37b4de03de908b697a0b88f5f516f1218143ae6f9cf5322c8245d4d3ab69948a\": container with ID starting with 37b4de03de908b697a0b88f5f516f1218143ae6f9cf5322c8245d4d3ab69948a not found: ID does not exist" Oct 30 21:51:17.113691 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.113613 5918 scope.go:122] "RemoveContainer" containerID="6834d44b6c689c55ee085cf407a92ddb84232029fb7ed209e68358efa2240328" Oct 30 21:51:17.113863 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:51:17.113845 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6834d44b6c689c55ee085cf407a92ddb84232029fb7ed209e68358efa2240328\": container with ID starting with 6834d44b6c689c55ee085cf407a92ddb84232029fb7ed209e68358efa2240328 not found: ID does not exist" containerID="6834d44b6c689c55ee085cf407a92ddb84232029fb7ed209e68358efa2240328" Oct 30 21:51:17.113921 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.113874 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6834d44b6c689c55ee085cf407a92ddb84232029fb7ed209e68358efa2240328"} err="failed to get container status \"6834d44b6c689c55ee085cf407a92ddb84232029fb7ed209e68358efa2240328\": rpc error: code = NotFound desc = could not find container \"6834d44b6c689c55ee085cf407a92ddb84232029fb7ed209e68358efa2240328\": container with ID starting with 6834d44b6c689c55ee085cf407a92ddb84232029fb7ed209e68358efa2240328 not found: ID does not exist" Oct 30 21:51:17.113921 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.113897 5918 scope.go:122] "RemoveContainer" containerID="5105322e8d9ba68c630906594410279a6ffa306bdf34adbcbf9d72aa63c2d5a2" Oct 30 21:51:17.114133 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:51:17.114116 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5105322e8d9ba68c630906594410279a6ffa306bdf34adbcbf9d72aa63c2d5a2\": container with ID starting with 5105322e8d9ba68c630906594410279a6ffa306bdf34adbcbf9d72aa63c2d5a2 not found: ID does not exist" containerID="5105322e8d9ba68c630906594410279a6ffa306bdf34adbcbf9d72aa63c2d5a2" Oct 30 21:51:17.114168 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.114141 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5105322e8d9ba68c630906594410279a6ffa306bdf34adbcbf9d72aa63c2d5a2"} err="failed to get container status \"5105322e8d9ba68c630906594410279a6ffa306bdf34adbcbf9d72aa63c2d5a2\": rpc error: code = NotFound desc = could not find container \"5105322e8d9ba68c630906594410279a6ffa306bdf34adbcbf9d72aa63c2d5a2\": container with ID starting with 5105322e8d9ba68c630906594410279a6ffa306bdf34adbcbf9d72aa63c2d5a2 not found: ID does not exist" Oct 30 21:51:17.114168 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.114156 5918 scope.go:122] "RemoveContainer" containerID="235067a08860012a63a021cead0fdc6bc25c7fe1b7fe749b46b504ae96a7e6cd" Oct 30 21:51:17.114421 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:51:17.114404 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"235067a08860012a63a021cead0fdc6bc25c7fe1b7fe749b46b504ae96a7e6cd\": container with ID starting with 235067a08860012a63a021cead0fdc6bc25c7fe1b7fe749b46b504ae96a7e6cd not found: ID does not exist" containerID="235067a08860012a63a021cead0fdc6bc25c7fe1b7fe749b46b504ae96a7e6cd" Oct 30 21:51:17.114478 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.114431 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"235067a08860012a63a021cead0fdc6bc25c7fe1b7fe749b46b504ae96a7e6cd"} err="failed to get container status \"235067a08860012a63a021cead0fdc6bc25c7fe1b7fe749b46b504ae96a7e6cd\": rpc error: code = NotFound desc = could not find container \"235067a08860012a63a021cead0fdc6bc25c7fe1b7fe749b46b504ae96a7e6cd\": container with ID starting with 235067a08860012a63a021cead0fdc6bc25c7fe1b7fe749b46b504ae96a7e6cd not found: ID does not exist" Oct 30 21:51:17.114478 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.114453 5918 scope.go:122] "RemoveContainer" containerID="080d93f93f7820f56af52a80a4e3d83c84dac6d82b71772824d68d0457d5bef7" Oct 30 21:51:17.114696 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:51:17.114677 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"080d93f93f7820f56af52a80a4e3d83c84dac6d82b71772824d68d0457d5bef7\": container with ID starting with 080d93f93f7820f56af52a80a4e3d83c84dac6d82b71772824d68d0457d5bef7 not found: ID does not exist" containerID="080d93f93f7820f56af52a80a4e3d83c84dac6d82b71772824d68d0457d5bef7" Oct 30 21:51:17.114742 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.114702 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"080d93f93f7820f56af52a80a4e3d83c84dac6d82b71772824d68d0457d5bef7"} err="failed to get container status \"080d93f93f7820f56af52a80a4e3d83c84dac6d82b71772824d68d0457d5bef7\": rpc error: code = NotFound desc = could not find container \"080d93f93f7820f56af52a80a4e3d83c84dac6d82b71772824d68d0457d5bef7\": container with ID starting with 080d93f93f7820f56af52a80a4e3d83c84dac6d82b71772824d68d0457d5bef7 not found: ID does not exist" Oct 30 21:51:17.114742 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.114715 5918 scope.go:122] "RemoveContainer" containerID="ce58dbf55668090f8fd32f2003f91b73cf879f6b4b2c2073b1f40f9c401d6978" Oct 30 21:51:17.114974 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:51:17.114958 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce58dbf55668090f8fd32f2003f91b73cf879f6b4b2c2073b1f40f9c401d6978\": container with ID starting with ce58dbf55668090f8fd32f2003f91b73cf879f6b4b2c2073b1f40f9c401d6978 not found: ID does not exist" containerID="ce58dbf55668090f8fd32f2003f91b73cf879f6b4b2c2073b1f40f9c401d6978" Oct 30 21:51:17.115015 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.114978 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce58dbf55668090f8fd32f2003f91b73cf879f6b4b2c2073b1f40f9c401d6978"} err="failed to get container status \"ce58dbf55668090f8fd32f2003f91b73cf879f6b4b2c2073b1f40f9c401d6978\": rpc error: code = NotFound desc = could not find container \"ce58dbf55668090f8fd32f2003f91b73cf879f6b4b2c2073b1f40f9c401d6978\": container with ID starting with ce58dbf55668090f8fd32f2003f91b73cf879f6b4b2c2073b1f40f9c401d6978 not found: ID does not exist" Oct 30 21:51:17.115015 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.114991 5918 scope.go:122] "RemoveContainer" containerID="b7029e8c3647fc8cf124a1d4e669b174d5c966d763a6a2cbc18befc88e3d1ffc" Oct 30 21:51:17.115202 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:51:17.115184 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7029e8c3647fc8cf124a1d4e669b174d5c966d763a6a2cbc18befc88e3d1ffc\": container with ID starting with b7029e8c3647fc8cf124a1d4e669b174d5c966d763a6a2cbc18befc88e3d1ffc not found: ID does not exist" containerID="b7029e8c3647fc8cf124a1d4e669b174d5c966d763a6a2cbc18befc88e3d1ffc" Oct 30 21:51:17.115298 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.115209 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7029e8c3647fc8cf124a1d4e669b174d5c966d763a6a2cbc18befc88e3d1ffc"} err="failed to get container status \"b7029e8c3647fc8cf124a1d4e669b174d5c966d763a6a2cbc18befc88e3d1ffc\": rpc error: code = NotFound desc = could not find container \"b7029e8c3647fc8cf124a1d4e669b174d5c966d763a6a2cbc18befc88e3d1ffc\": container with ID starting with b7029e8c3647fc8cf124a1d4e669b174d5c966d763a6a2cbc18befc88e3d1ffc not found: ID does not exist" Oct 30 21:51:17.115298 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.115244 5918 scope.go:122] "RemoveContainer" containerID="37b4de03de908b697a0b88f5f516f1218143ae6f9cf5322c8245d4d3ab69948a" Oct 30 21:51:17.115475 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.115458 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37b4de03de908b697a0b88f5f516f1218143ae6f9cf5322c8245d4d3ab69948a"} err="failed to get container status \"37b4de03de908b697a0b88f5f516f1218143ae6f9cf5322c8245d4d3ab69948a\": rpc error: code = NotFound desc = could not find container \"37b4de03de908b697a0b88f5f516f1218143ae6f9cf5322c8245d4d3ab69948a\": container with ID starting with 37b4de03de908b697a0b88f5f516f1218143ae6f9cf5322c8245d4d3ab69948a not found: ID does not exist" Oct 30 21:51:17.115520 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.115475 5918 scope.go:122] "RemoveContainer" containerID="6834d44b6c689c55ee085cf407a92ddb84232029fb7ed209e68358efa2240328" Oct 30 21:51:17.115788 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.115688 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6834d44b6c689c55ee085cf407a92ddb84232029fb7ed209e68358efa2240328"} err="failed to get container status \"6834d44b6c689c55ee085cf407a92ddb84232029fb7ed209e68358efa2240328\": rpc error: code = NotFound desc = could not find container \"6834d44b6c689c55ee085cf407a92ddb84232029fb7ed209e68358efa2240328\": container with ID starting with 6834d44b6c689c55ee085cf407a92ddb84232029fb7ed209e68358efa2240328 not found: ID does not exist" Oct 30 21:51:17.115788 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.115713 5918 scope.go:122] "RemoveContainer" containerID="5105322e8d9ba68c630906594410279a6ffa306bdf34adbcbf9d72aa63c2d5a2" Oct 30 21:51:17.115995 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.115927 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5105322e8d9ba68c630906594410279a6ffa306bdf34adbcbf9d72aa63c2d5a2"} err="failed to get container status \"5105322e8d9ba68c630906594410279a6ffa306bdf34adbcbf9d72aa63c2d5a2\": rpc error: code = NotFound desc = could not find container \"5105322e8d9ba68c630906594410279a6ffa306bdf34adbcbf9d72aa63c2d5a2\": container with ID starting with 5105322e8d9ba68c630906594410279a6ffa306bdf34adbcbf9d72aa63c2d5a2 not found: ID does not exist" Oct 30 21:51:17.115995 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.115946 5918 scope.go:122] "RemoveContainer" containerID="235067a08860012a63a021cead0fdc6bc25c7fe1b7fe749b46b504ae96a7e6cd" Oct 30 21:51:17.116243 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.116204 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"235067a08860012a63a021cead0fdc6bc25c7fe1b7fe749b46b504ae96a7e6cd"} err="failed to get container status \"235067a08860012a63a021cead0fdc6bc25c7fe1b7fe749b46b504ae96a7e6cd\": rpc error: code = NotFound desc = could not find container \"235067a08860012a63a021cead0fdc6bc25c7fe1b7fe749b46b504ae96a7e6cd\": container with ID starting with 235067a08860012a63a021cead0fdc6bc25c7fe1b7fe749b46b504ae96a7e6cd not found: ID does not exist" Oct 30 21:51:17.116243 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.116243 5918 scope.go:122] "RemoveContainer" containerID="080d93f93f7820f56af52a80a4e3d83c84dac6d82b71772824d68d0457d5bef7" Oct 30 21:51:17.116466 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.116448 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"080d93f93f7820f56af52a80a4e3d83c84dac6d82b71772824d68d0457d5bef7"} err="failed to get container status \"080d93f93f7820f56af52a80a4e3d83c84dac6d82b71772824d68d0457d5bef7\": rpc error: code = NotFound desc = could not find container \"080d93f93f7820f56af52a80a4e3d83c84dac6d82b71772824d68d0457d5bef7\": container with ID starting with 080d93f93f7820f56af52a80a4e3d83c84dac6d82b71772824d68d0457d5bef7 not found: ID does not exist" Oct 30 21:51:17.116466 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.116465 5918 scope.go:122] "RemoveContainer" containerID="ce58dbf55668090f8fd32f2003f91b73cf879f6b4b2c2073b1f40f9c401d6978" Oct 30 21:51:17.116677 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.116656 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce58dbf55668090f8fd32f2003f91b73cf879f6b4b2c2073b1f40f9c401d6978"} err="failed to get container status \"ce58dbf55668090f8fd32f2003f91b73cf879f6b4b2c2073b1f40f9c401d6978\": rpc error: code = NotFound desc = could not find container \"ce58dbf55668090f8fd32f2003f91b73cf879f6b4b2c2073b1f40f9c401d6978\": container with ID starting with ce58dbf55668090f8fd32f2003f91b73cf879f6b4b2c2073b1f40f9c401d6978 not found: ID does not exist" Oct 30 21:51:17.116752 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.116678 5918 scope.go:122] "RemoveContainer" containerID="b7029e8c3647fc8cf124a1d4e669b174d5c966d763a6a2cbc18befc88e3d1ffc" Oct 30 21:51:17.116917 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.116902 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7029e8c3647fc8cf124a1d4e669b174d5c966d763a6a2cbc18befc88e3d1ffc"} err="failed to get container status \"b7029e8c3647fc8cf124a1d4e669b174d5c966d763a6a2cbc18befc88e3d1ffc\": rpc error: code = NotFound desc = could not find container \"b7029e8c3647fc8cf124a1d4e669b174d5c966d763a6a2cbc18befc88e3d1ffc\": container with ID starting with b7029e8c3647fc8cf124a1d4e669b174d5c966d763a6a2cbc18befc88e3d1ffc not found: ID does not exist" Oct 30 21:51:17.116964 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.116917 5918 scope.go:122] "RemoveContainer" containerID="37b4de03de908b697a0b88f5f516f1218143ae6f9cf5322c8245d4d3ab69948a" Oct 30 21:51:17.117116 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.117097 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37b4de03de908b697a0b88f5f516f1218143ae6f9cf5322c8245d4d3ab69948a"} err="failed to get container status \"37b4de03de908b697a0b88f5f516f1218143ae6f9cf5322c8245d4d3ab69948a\": rpc error: code = NotFound desc = could not find container \"37b4de03de908b697a0b88f5f516f1218143ae6f9cf5322c8245d4d3ab69948a\": container with ID starting with 37b4de03de908b697a0b88f5f516f1218143ae6f9cf5322c8245d4d3ab69948a not found: ID does not exist" Oct 30 21:51:17.117183 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.117116 5918 scope.go:122] "RemoveContainer" containerID="6834d44b6c689c55ee085cf407a92ddb84232029fb7ed209e68358efa2240328" Oct 30 21:51:17.117344 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.117307 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6834d44b6c689c55ee085cf407a92ddb84232029fb7ed209e68358efa2240328"} err="failed to get container status \"6834d44b6c689c55ee085cf407a92ddb84232029fb7ed209e68358efa2240328\": rpc error: code = NotFound desc = could not find container \"6834d44b6c689c55ee085cf407a92ddb84232029fb7ed209e68358efa2240328\": container with ID starting with 6834d44b6c689c55ee085cf407a92ddb84232029fb7ed209e68358efa2240328 not found: ID does not exist" Oct 30 21:51:17.117418 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.117346 5918 scope.go:122] "RemoveContainer" containerID="5105322e8d9ba68c630906594410279a6ffa306bdf34adbcbf9d72aa63c2d5a2" Oct 30 21:51:17.117589 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.117570 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5105322e8d9ba68c630906594410279a6ffa306bdf34adbcbf9d72aa63c2d5a2"} err="failed to get container status \"5105322e8d9ba68c630906594410279a6ffa306bdf34adbcbf9d72aa63c2d5a2\": rpc error: code = NotFound desc = could not find container \"5105322e8d9ba68c630906594410279a6ffa306bdf34adbcbf9d72aa63c2d5a2\": container with ID starting with 5105322e8d9ba68c630906594410279a6ffa306bdf34adbcbf9d72aa63c2d5a2 not found: ID does not exist" Oct 30 21:51:17.117589 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.117588 5918 scope.go:122] "RemoveContainer" containerID="235067a08860012a63a021cead0fdc6bc25c7fe1b7fe749b46b504ae96a7e6cd" Oct 30 21:51:17.117810 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.117787 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"235067a08860012a63a021cead0fdc6bc25c7fe1b7fe749b46b504ae96a7e6cd"} err="failed to get container status \"235067a08860012a63a021cead0fdc6bc25c7fe1b7fe749b46b504ae96a7e6cd\": rpc error: code = NotFound desc = could not find container \"235067a08860012a63a021cead0fdc6bc25c7fe1b7fe749b46b504ae96a7e6cd\": container with ID starting with 235067a08860012a63a021cead0fdc6bc25c7fe1b7fe749b46b504ae96a7e6cd not found: ID does not exist" Oct 30 21:51:17.117810 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.117810 5918 scope.go:122] "RemoveContainer" containerID="080d93f93f7820f56af52a80a4e3d83c84dac6d82b71772824d68d0457d5bef7" Oct 30 21:51:17.118003 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.117989 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"080d93f93f7820f56af52a80a4e3d83c84dac6d82b71772824d68d0457d5bef7"} err="failed to get container status \"080d93f93f7820f56af52a80a4e3d83c84dac6d82b71772824d68d0457d5bef7\": rpc error: code = NotFound desc = could not find container \"080d93f93f7820f56af52a80a4e3d83c84dac6d82b71772824d68d0457d5bef7\": container with ID starting with 080d93f93f7820f56af52a80a4e3d83c84dac6d82b71772824d68d0457d5bef7 not found: ID does not exist" Oct 30 21:51:17.118043 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.118003 5918 scope.go:122] "RemoveContainer" containerID="ce58dbf55668090f8fd32f2003f91b73cf879f6b4b2c2073b1f40f9c401d6978" Oct 30 21:51:17.118194 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.118171 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce58dbf55668090f8fd32f2003f91b73cf879f6b4b2c2073b1f40f9c401d6978"} err="failed to get container status \"ce58dbf55668090f8fd32f2003f91b73cf879f6b4b2c2073b1f40f9c401d6978\": rpc error: code = NotFound desc = could not find container \"ce58dbf55668090f8fd32f2003f91b73cf879f6b4b2c2073b1f40f9c401d6978\": container with ID starting with ce58dbf55668090f8fd32f2003f91b73cf879f6b4b2c2073b1f40f9c401d6978 not found: ID does not exist" Oct 30 21:51:17.118266 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.118196 5918 scope.go:122] "RemoveContainer" containerID="b7029e8c3647fc8cf124a1d4e669b174d5c966d763a6a2cbc18befc88e3d1ffc" Oct 30 21:51:17.118429 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.118412 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7029e8c3647fc8cf124a1d4e669b174d5c966d763a6a2cbc18befc88e3d1ffc"} err="failed to get container status \"b7029e8c3647fc8cf124a1d4e669b174d5c966d763a6a2cbc18befc88e3d1ffc\": rpc error: code = NotFound desc = could not find container \"b7029e8c3647fc8cf124a1d4e669b174d5c966d763a6a2cbc18befc88e3d1ffc\": container with ID starting with b7029e8c3647fc8cf124a1d4e669b174d5c966d763a6a2cbc18befc88e3d1ffc not found: ID does not exist" Oct 30 21:51:17.118483 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.118429 5918 scope.go:122] "RemoveContainer" containerID="37b4de03de908b697a0b88f5f516f1218143ae6f9cf5322c8245d4d3ab69948a" Oct 30 21:51:17.118634 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.118614 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37b4de03de908b697a0b88f5f516f1218143ae6f9cf5322c8245d4d3ab69948a"} err="failed to get container status \"37b4de03de908b697a0b88f5f516f1218143ae6f9cf5322c8245d4d3ab69948a\": rpc error: code = NotFound desc = could not find container \"37b4de03de908b697a0b88f5f516f1218143ae6f9cf5322c8245d4d3ab69948a\": container with ID starting with 37b4de03de908b697a0b88f5f516f1218143ae6f9cf5322c8245d4d3ab69948a not found: ID does not exist" Oct 30 21:51:17.118711 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.118646 5918 scope.go:122] "RemoveContainer" containerID="6834d44b6c689c55ee085cf407a92ddb84232029fb7ed209e68358efa2240328" Oct 30 21:51:17.118894 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.118874 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6834d44b6c689c55ee085cf407a92ddb84232029fb7ed209e68358efa2240328"} err="failed to get container status \"6834d44b6c689c55ee085cf407a92ddb84232029fb7ed209e68358efa2240328\": rpc error: code = NotFound desc = could not find container \"6834d44b6c689c55ee085cf407a92ddb84232029fb7ed209e68358efa2240328\": container with ID starting with 6834d44b6c689c55ee085cf407a92ddb84232029fb7ed209e68358efa2240328 not found: ID does not exist" Oct 30 21:51:17.118894 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.118892 5918 scope.go:122] "RemoveContainer" containerID="5105322e8d9ba68c630906594410279a6ffa306bdf34adbcbf9d72aa63c2d5a2" Oct 30 21:51:17.119141 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.119126 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5105322e8d9ba68c630906594410279a6ffa306bdf34adbcbf9d72aa63c2d5a2"} err="failed to get container status \"5105322e8d9ba68c630906594410279a6ffa306bdf34adbcbf9d72aa63c2d5a2\": rpc error: code = NotFound desc = could not find container \"5105322e8d9ba68c630906594410279a6ffa306bdf34adbcbf9d72aa63c2d5a2\": container with ID starting with 5105322e8d9ba68c630906594410279a6ffa306bdf34adbcbf9d72aa63c2d5a2 not found: ID does not exist" Oct 30 21:51:17.119141 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.119141 5918 scope.go:122] "RemoveContainer" containerID="235067a08860012a63a021cead0fdc6bc25c7fe1b7fe749b46b504ae96a7e6cd" Oct 30 21:51:17.119372 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.119353 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"235067a08860012a63a021cead0fdc6bc25c7fe1b7fe749b46b504ae96a7e6cd"} err="failed to get container status \"235067a08860012a63a021cead0fdc6bc25c7fe1b7fe749b46b504ae96a7e6cd\": rpc error: code = NotFound desc = could not find container \"235067a08860012a63a021cead0fdc6bc25c7fe1b7fe749b46b504ae96a7e6cd\": container with ID starting with 235067a08860012a63a021cead0fdc6bc25c7fe1b7fe749b46b504ae96a7e6cd not found: ID does not exist" Oct 30 21:51:17.119467 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.119373 5918 scope.go:122] "RemoveContainer" containerID="080d93f93f7820f56af52a80a4e3d83c84dac6d82b71772824d68d0457d5bef7" Oct 30 21:51:17.119606 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.119585 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"080d93f93f7820f56af52a80a4e3d83c84dac6d82b71772824d68d0457d5bef7"} err="failed to get container status \"080d93f93f7820f56af52a80a4e3d83c84dac6d82b71772824d68d0457d5bef7\": rpc error: code = NotFound desc = could not find container \"080d93f93f7820f56af52a80a4e3d83c84dac6d82b71772824d68d0457d5bef7\": container with ID starting with 080d93f93f7820f56af52a80a4e3d83c84dac6d82b71772824d68d0457d5bef7 not found: ID does not exist" Oct 30 21:51:17.119676 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.119608 5918 scope.go:122] "RemoveContainer" containerID="ce58dbf55668090f8fd32f2003f91b73cf879f6b4b2c2073b1f40f9c401d6978" Oct 30 21:51:17.119843 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.119825 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce58dbf55668090f8fd32f2003f91b73cf879f6b4b2c2073b1f40f9c401d6978"} err="failed to get container status \"ce58dbf55668090f8fd32f2003f91b73cf879f6b4b2c2073b1f40f9c401d6978\": rpc error: code = NotFound desc = could not find container \"ce58dbf55668090f8fd32f2003f91b73cf879f6b4b2c2073b1f40f9c401d6978\": container with ID starting with ce58dbf55668090f8fd32f2003f91b73cf879f6b4b2c2073b1f40f9c401d6978 not found: ID does not exist" Oct 30 21:51:17.119900 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.119844 5918 scope.go:122] "RemoveContainer" containerID="b7029e8c3647fc8cf124a1d4e669b174d5c966d763a6a2cbc18befc88e3d1ffc" Oct 30 21:51:17.120087 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.120068 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7029e8c3647fc8cf124a1d4e669b174d5c966d763a6a2cbc18befc88e3d1ffc"} err="failed to get container status \"b7029e8c3647fc8cf124a1d4e669b174d5c966d763a6a2cbc18befc88e3d1ffc\": rpc error: code = NotFound desc = could not find container \"b7029e8c3647fc8cf124a1d4e669b174d5c966d763a6a2cbc18befc88e3d1ffc\": container with ID starting with b7029e8c3647fc8cf124a1d4e669b174d5c966d763a6a2cbc18befc88e3d1ffc not found: ID does not exist" Oct 30 21:51:17.120087 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.120086 5918 scope.go:122] "RemoveContainer" containerID="37b4de03de908b697a0b88f5f516f1218143ae6f9cf5322c8245d4d3ab69948a" Oct 30 21:51:17.120324 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.120308 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37b4de03de908b697a0b88f5f516f1218143ae6f9cf5322c8245d4d3ab69948a"} err="failed to get container status \"37b4de03de908b697a0b88f5f516f1218143ae6f9cf5322c8245d4d3ab69948a\": rpc error: code = NotFound desc = could not find container \"37b4de03de908b697a0b88f5f516f1218143ae6f9cf5322c8245d4d3ab69948a\": container with ID starting with 37b4de03de908b697a0b88f5f516f1218143ae6f9cf5322c8245d4d3ab69948a not found: ID does not exist" Oct 30 21:51:17.120369 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.120325 5918 scope.go:122] "RemoveContainer" containerID="6834d44b6c689c55ee085cf407a92ddb84232029fb7ed209e68358efa2240328" Oct 30 21:51:17.120520 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.120505 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6834d44b6c689c55ee085cf407a92ddb84232029fb7ed209e68358efa2240328"} err="failed to get container status \"6834d44b6c689c55ee085cf407a92ddb84232029fb7ed209e68358efa2240328\": rpc error: code = NotFound desc = could not find container \"6834d44b6c689c55ee085cf407a92ddb84232029fb7ed209e68358efa2240328\": container with ID starting with 6834d44b6c689c55ee085cf407a92ddb84232029fb7ed209e68358efa2240328 not found: ID does not exist" Oct 30 21:51:17.120559 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.120521 5918 scope.go:122] "RemoveContainer" containerID="5105322e8d9ba68c630906594410279a6ffa306bdf34adbcbf9d72aa63c2d5a2" Oct 30 21:51:17.120725 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.120704 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5105322e8d9ba68c630906594410279a6ffa306bdf34adbcbf9d72aa63c2d5a2"} err="failed to get container status \"5105322e8d9ba68c630906594410279a6ffa306bdf34adbcbf9d72aa63c2d5a2\": rpc error: code = NotFound desc = could not find container \"5105322e8d9ba68c630906594410279a6ffa306bdf34adbcbf9d72aa63c2d5a2\": container with ID starting with 5105322e8d9ba68c630906594410279a6ffa306bdf34adbcbf9d72aa63c2d5a2 not found: ID does not exist" Oct 30 21:51:17.120803 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.120726 5918 scope.go:122] "RemoveContainer" containerID="235067a08860012a63a021cead0fdc6bc25c7fe1b7fe749b46b504ae96a7e6cd" Oct 30 21:51:17.120931 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.120917 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"235067a08860012a63a021cead0fdc6bc25c7fe1b7fe749b46b504ae96a7e6cd"} err="failed to get container status \"235067a08860012a63a021cead0fdc6bc25c7fe1b7fe749b46b504ae96a7e6cd\": rpc error: code = NotFound desc = could not find container \"235067a08860012a63a021cead0fdc6bc25c7fe1b7fe749b46b504ae96a7e6cd\": container with ID starting with 235067a08860012a63a021cead0fdc6bc25c7fe1b7fe749b46b504ae96a7e6cd not found: ID does not exist" Oct 30 21:51:17.120976 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.120932 5918 scope.go:122] "RemoveContainer" containerID="080d93f93f7820f56af52a80a4e3d83c84dac6d82b71772824d68d0457d5bef7" Oct 30 21:51:17.121144 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.121127 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"080d93f93f7820f56af52a80a4e3d83c84dac6d82b71772824d68d0457d5bef7"} err="failed to get container status \"080d93f93f7820f56af52a80a4e3d83c84dac6d82b71772824d68d0457d5bef7\": rpc error: code = NotFound desc = could not find container \"080d93f93f7820f56af52a80a4e3d83c84dac6d82b71772824d68d0457d5bef7\": container with ID starting with 080d93f93f7820f56af52a80a4e3d83c84dac6d82b71772824d68d0457d5bef7 not found: ID does not exist" Oct 30 21:51:17.121208 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.121147 5918 scope.go:122] "RemoveContainer" containerID="ce58dbf55668090f8fd32f2003f91b73cf879f6b4b2c2073b1f40f9c401d6978" Oct 30 21:51:17.121368 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.121348 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce58dbf55668090f8fd32f2003f91b73cf879f6b4b2c2073b1f40f9c401d6978"} err="failed to get container status \"ce58dbf55668090f8fd32f2003f91b73cf879f6b4b2c2073b1f40f9c401d6978\": rpc error: code = NotFound desc = could not find container \"ce58dbf55668090f8fd32f2003f91b73cf879f6b4b2c2073b1f40f9c401d6978\": container with ID starting with ce58dbf55668090f8fd32f2003f91b73cf879f6b4b2c2073b1f40f9c401d6978 not found: ID does not exist" Oct 30 21:51:17.121421 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.121369 5918 scope.go:122] "RemoveContainer" containerID="b7029e8c3647fc8cf124a1d4e669b174d5c966d763a6a2cbc18befc88e3d1ffc" Oct 30 21:51:17.121570 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.121555 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7029e8c3647fc8cf124a1d4e669b174d5c966d763a6a2cbc18befc88e3d1ffc"} err="failed to get container status \"b7029e8c3647fc8cf124a1d4e669b174d5c966d763a6a2cbc18befc88e3d1ffc\": rpc error: code = NotFound desc = could not find container \"b7029e8c3647fc8cf124a1d4e669b174d5c966d763a6a2cbc18befc88e3d1ffc\": container with ID starting with b7029e8c3647fc8cf124a1d4e669b174d5c966d763a6a2cbc18befc88e3d1ffc not found: ID does not exist" Oct 30 21:51:17.121611 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.121570 5918 scope.go:122] "RemoveContainer" containerID="37b4de03de908b697a0b88f5f516f1218143ae6f9cf5322c8245d4d3ab69948a" Oct 30 21:51:17.121768 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.121753 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37b4de03de908b697a0b88f5f516f1218143ae6f9cf5322c8245d4d3ab69948a"} err="failed to get container status \"37b4de03de908b697a0b88f5f516f1218143ae6f9cf5322c8245d4d3ab69948a\": rpc error: code = NotFound desc = could not find container \"37b4de03de908b697a0b88f5f516f1218143ae6f9cf5322c8245d4d3ab69948a\": container with ID starting with 37b4de03de908b697a0b88f5f516f1218143ae6f9cf5322c8245d4d3ab69948a not found: ID does not exist" Oct 30 21:51:17.121808 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.121768 5918 scope.go:122] "RemoveContainer" containerID="6834d44b6c689c55ee085cf407a92ddb84232029fb7ed209e68358efa2240328" Oct 30 21:51:17.121991 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.121975 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6834d44b6c689c55ee085cf407a92ddb84232029fb7ed209e68358efa2240328"} err="failed to get container status \"6834d44b6c689c55ee085cf407a92ddb84232029fb7ed209e68358efa2240328\": rpc error: code = NotFound desc = could not find container \"6834d44b6c689c55ee085cf407a92ddb84232029fb7ed209e68358efa2240328\": container with ID starting with 6834d44b6c689c55ee085cf407a92ddb84232029fb7ed209e68358efa2240328 not found: ID does not exist" Oct 30 21:51:17.121991 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.121990 5918 scope.go:122] "RemoveContainer" containerID="5105322e8d9ba68c630906594410279a6ffa306bdf34adbcbf9d72aa63c2d5a2" Oct 30 21:51:17.122193 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.122177 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5105322e8d9ba68c630906594410279a6ffa306bdf34adbcbf9d72aa63c2d5a2"} err="failed to get container status \"5105322e8d9ba68c630906594410279a6ffa306bdf34adbcbf9d72aa63c2d5a2\": rpc error: code = NotFound desc = could not find container \"5105322e8d9ba68c630906594410279a6ffa306bdf34adbcbf9d72aa63c2d5a2\": container with ID starting with 5105322e8d9ba68c630906594410279a6ffa306bdf34adbcbf9d72aa63c2d5a2 not found: ID does not exist" Oct 30 21:51:17.122269 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.122194 5918 scope.go:122] "RemoveContainer" containerID="235067a08860012a63a021cead0fdc6bc25c7fe1b7fe749b46b504ae96a7e6cd" Oct 30 21:51:17.122417 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.122402 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"235067a08860012a63a021cead0fdc6bc25c7fe1b7fe749b46b504ae96a7e6cd"} err="failed to get container status \"235067a08860012a63a021cead0fdc6bc25c7fe1b7fe749b46b504ae96a7e6cd\": rpc error: code = NotFound desc = could not find container \"235067a08860012a63a021cead0fdc6bc25c7fe1b7fe749b46b504ae96a7e6cd\": container with ID starting with 235067a08860012a63a021cead0fdc6bc25c7fe1b7fe749b46b504ae96a7e6cd not found: ID does not exist" Oct 30 21:51:17.122461 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.122415 5918 scope.go:122] "RemoveContainer" containerID="080d93f93f7820f56af52a80a4e3d83c84dac6d82b71772824d68d0457d5bef7" Oct 30 21:51:17.122643 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.122625 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"080d93f93f7820f56af52a80a4e3d83c84dac6d82b71772824d68d0457d5bef7"} err="failed to get container status \"080d93f93f7820f56af52a80a4e3d83c84dac6d82b71772824d68d0457d5bef7\": rpc error: code = NotFound desc = could not find container \"080d93f93f7820f56af52a80a4e3d83c84dac6d82b71772824d68d0457d5bef7\": container with ID starting with 080d93f93f7820f56af52a80a4e3d83c84dac6d82b71772824d68d0457d5bef7 not found: ID does not exist" Oct 30 21:51:17.122643 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.122642 5918 scope.go:122] "RemoveContainer" containerID="ce58dbf55668090f8fd32f2003f91b73cf879f6b4b2c2073b1f40f9c401d6978" Oct 30 21:51:17.122852 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.122833 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce58dbf55668090f8fd32f2003f91b73cf879f6b4b2c2073b1f40f9c401d6978"} err="failed to get container status \"ce58dbf55668090f8fd32f2003f91b73cf879f6b4b2c2073b1f40f9c401d6978\": rpc error: code = NotFound desc = could not find container \"ce58dbf55668090f8fd32f2003f91b73cf879f6b4b2c2073b1f40f9c401d6978\": container with ID starting with ce58dbf55668090f8fd32f2003f91b73cf879f6b4b2c2073b1f40f9c401d6978 not found: ID does not exist" Oct 30 21:51:17.122923 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.122853 5918 scope.go:122] "RemoveContainer" containerID="b7029e8c3647fc8cf124a1d4e669b174d5c966d763a6a2cbc18befc88e3d1ffc" Oct 30 21:51:17.123069 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.123048 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7029e8c3647fc8cf124a1d4e669b174d5c966d763a6a2cbc18befc88e3d1ffc"} err="failed to get container status \"b7029e8c3647fc8cf124a1d4e669b174d5c966d763a6a2cbc18befc88e3d1ffc\": rpc error: code = NotFound desc = could not find container \"b7029e8c3647fc8cf124a1d4e669b174d5c966d763a6a2cbc18befc88e3d1ffc\": container with ID starting with b7029e8c3647fc8cf124a1d4e669b174d5c966d763a6a2cbc18befc88e3d1ffc not found: ID does not exist" Oct 30 21:51:17.233140 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.233107 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 21:51:17.283830 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.283797 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 21:51:17.284200 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.284174 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 21:51:17.284467 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.284454 5918 cpu_manager.go:414] "containerMap: removing container" podUID="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" containerName="alertmanager" Oct 30 21:51:17.284512 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.284468 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" containerName="alertmanager" Oct 30 21:51:17.284512 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.284483 5918 cpu_manager.go:414] "containerMap: removing container" podUID="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" containerName="config-reloader" Oct 30 21:51:17.284512 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.284488 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" containerName="config-reloader" Oct 30 21:51:17.284512 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.284507 5918 cpu_manager.go:414] "containerMap: removing container" podUID="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" containerName="prom-label-proxy" Oct 30 21:51:17.284512 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.284512 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" containerName="prom-label-proxy" Oct 30 21:51:17.284658 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.284523 5918 cpu_manager.go:414] "containerMap: removing container" podUID="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" containerName="kube-rbac-proxy-metric" Oct 30 21:51:17.284658 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.284528 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" containerName="kube-rbac-proxy-metric" Oct 30 21:51:17.284658 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.284535 5918 cpu_manager.go:414] "containerMap: removing container" podUID="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" containerName="kube-rbac-proxy-web" Oct 30 21:51:17.284658 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.284539 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" containerName="kube-rbac-proxy-web" Oct 30 21:51:17.284658 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.284546 5918 cpu_manager.go:414] "containerMap: removing container" podUID="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" containerName="kube-rbac-proxy" Oct 30 21:51:17.284658 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.284550 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" containerName="kube-rbac-proxy" Oct 30 21:51:17.284658 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.284555 5918 cpu_manager.go:414] "containerMap: removing container" podUID="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" containerName="init-config-reloader" Oct 30 21:51:17.284658 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.284559 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" containerName="init-config-reloader" Oct 30 21:51:17.284658 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.284601 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" containerName="prom-label-proxy" Oct 30 21:51:17.284658 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.284609 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" containerName="kube-rbac-proxy-metric" Oct 30 21:51:17.284658 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.284615 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" containerName="kube-rbac-proxy-web" Oct 30 21:51:17.284658 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.284620 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" containerName="kube-rbac-proxy" Oct 30 21:51:17.284658 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.284624 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" containerName="init-config-reloader" Oct 30 21:51:17.284658 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.284630 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" containerName="alertmanager" Oct 30 21:51:17.284658 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.284636 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" containerName="config-reloader" Oct 30 21:51:17.287836 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.287819 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.343309 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.343179 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 21:51:17.343484 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.343357 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 21:51:17.343484 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.343372 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 21:51:17.343484 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.343405 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 21:51:17.343484 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.343434 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 21:51:17.343484 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.343437 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 21:51:17.343742 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.343499 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-dhvch\"" Oct 30 21:51:17.343742 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.343508 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 21:51:17.343742 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.343607 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 21:51:17.352029 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.352003 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b077ad14-ff57-45ba-87c3-853a2e104003-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.352107 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.352043 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b077ad14-ff57-45ba-87c3-853a2e104003-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.352107 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.352083 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b077ad14-ff57-45ba-87c3-853a2e104003-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.352187 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.352121 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktvhg\" (UniqueName: \"kubernetes.io/projected/b077ad14-ff57-45ba-87c3-853a2e104003-kube-api-access-ktvhg\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.352270 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.352256 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b077ad14-ff57-45ba-87c3-853a2e104003-config-volume\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.352310 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.352294 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b077ad14-ff57-45ba-87c3-853a2e104003-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.352350 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.352310 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b077ad14-ff57-45ba-87c3-853a2e104003-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.352350 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.352329 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b077ad14-ff57-45ba-87c3-853a2e104003-config-out\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.352350 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.352344 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b077ad14-ff57-45ba-87c3-853a2e104003-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.352456 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.352388 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b077ad14-ff57-45ba-87c3-853a2e104003-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.352534 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.352518 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b077ad14-ff57-45ba-87c3-853a2e104003-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.352567 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.352542 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b077ad14-ff57-45ba-87c3-853a2e104003-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.352567 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.352563 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b077ad14-ff57-45ba-87c3-853a2e104003-web-config\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.393331 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.393299 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 21:51:17.405159 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.405127 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 21:51:17.453465 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.453427 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b077ad14-ff57-45ba-87c3-853a2e104003-config-out\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.453625 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.453476 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b077ad14-ff57-45ba-87c3-853a2e104003-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.453625 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.453512 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b077ad14-ff57-45ba-87c3-853a2e104003-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.453625 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.453558 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b077ad14-ff57-45ba-87c3-853a2e104003-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.453625 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.453582 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b077ad14-ff57-45ba-87c3-853a2e104003-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.453625 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.453609 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b077ad14-ff57-45ba-87c3-853a2e104003-web-config\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.453804 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.453666 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b077ad14-ff57-45ba-87c3-853a2e104003-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.453804 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.453699 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b077ad14-ff57-45ba-87c3-853a2e104003-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.453804 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.453742 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b077ad14-ff57-45ba-87c3-853a2e104003-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.453804 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.453765 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktvhg\" (UniqueName: \"kubernetes.io/projected/b077ad14-ff57-45ba-87c3-853a2e104003-kube-api-access-ktvhg\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.453997 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.453808 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b077ad14-ff57-45ba-87c3-853a2e104003-config-volume\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.453997 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.453857 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b077ad14-ff57-45ba-87c3-853a2e104003-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.453997 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.453868 5918 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:51:17.453997 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.453884 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b077ad14-ff57-45ba-87c3-853a2e104003-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.453997 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.453934 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b077ad14-ff57-45ba-87c3-853a2e104003-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.455006 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.454975 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b077ad14-ff57-45ba-87c3-853a2e104003-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.455310 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.455161 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b077ad14-ff57-45ba-87c3-853a2e104003-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.457108 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.457007 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b077ad14-ff57-45ba-87c3-853a2e104003-config-out\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.457377 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.457353 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b077ad14-ff57-45ba-87c3-853a2e104003-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.458018 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.457861 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b077ad14-ff57-45ba-87c3-853a2e104003-web-config\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.458018 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.457962 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b077ad14-ff57-45ba-87c3-853a2e104003-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.458018 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.457978 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b077ad14-ff57-45ba-87c3-853a2e104003-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.458233 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.458144 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b077ad14-ff57-45ba-87c3-853a2e104003-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.458480 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.458460 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b077ad14-ff57-45ba-87c3-853a2e104003-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.458822 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.458801 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b077ad14-ff57-45ba-87c3-853a2e104003-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.459262 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.459244 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b077ad14-ff57-45ba-87c3-853a2e104003-config-volume\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.517813 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.517789 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktvhg\" (UniqueName: \"kubernetes.io/projected/b077ad14-ff57-45ba-87c3-853a2e104003-kube-api-access-ktvhg\") pod \"alertmanager-main-0\" (UID: \"b077ad14-ff57-45ba-87c3-853a2e104003\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.609543 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.609486 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:51:17.848454 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:17.848411 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e62c7d9-71b5-4f9b-a587-aaf06c7b0735" path="/var/lib/kubelet/pods/9e62c7d9-71b5-4f9b-a587-aaf06c7b0735/volumes" Oct 30 21:51:18.038351 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:18.038322 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 21:51:18.041675 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:51:18.041648 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb077ad14_ff57_45ba_87c3_853a2e104003.slice/crio-d3b08c88bb6136789b35f72d0822364ea1f69f56ff437cef9071c3ed608ac15d WatchSource:0}: Error finding container d3b08c88bb6136789b35f72d0822364ea1f69f56ff437cef9071c3ed608ac15d: Status 404 returned error can't find the container with id d3b08c88bb6136789b35f72d0822364ea1f69f56ff437cef9071c3ed608ac15d Oct 30 21:51:18.066026 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:18.065999 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b077ad14-ff57-45ba-87c3-853a2e104003","Type":"ContainerStarted","Data":"d3b08c88bb6136789b35f72d0822364ea1f69f56ff437cef9071c3ed608ac15d"} Oct 30 21:51:19.071776 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:19.071741 5918 generic.go:356] "Generic (PLEG): container finished" podID="b077ad14-ff57-45ba-87c3-853a2e104003" containerID="d4ca0f3e4498e5ec6214b575bb7f6030dd7736a5a07b4a906557ed993f5ba9dd" exitCode=0 Oct 30 21:51:19.072288 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:19.071798 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b077ad14-ff57-45ba-87c3-853a2e104003","Type":"ContainerDied","Data":"d4ca0f3e4498e5ec6214b575bb7f6030dd7736a5a07b4a906557ed993f5ba9dd"} Oct 30 21:51:20.078374 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:20.078340 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b077ad14-ff57-45ba-87c3-853a2e104003","Type":"ContainerStarted","Data":"40e54a38160bc36b173a1241feb9496a08b2fbff16f2420a500734bd17fd3033"} Oct 30 21:51:20.078822 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:20.078401 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b077ad14-ff57-45ba-87c3-853a2e104003","Type":"ContainerStarted","Data":"e8adfbd94cacaa5e9d9af8dead8bb951248b705170a0ede9ea2a1382d5c83a77"} Oct 30 21:51:20.078822 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:20.078426 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b077ad14-ff57-45ba-87c3-853a2e104003","Type":"ContainerStarted","Data":"b9b7329efeef615c8b900f8e8f867b1a42941aeb0d491cf1f9824930fac51618"} Oct 30 21:51:20.078822 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:20.078461 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b077ad14-ff57-45ba-87c3-853a2e104003","Type":"ContainerStarted","Data":"566126f87b39014cb23ba5a5f2e5e40e7e1a740bbfc4e812ed7a2f03204d75f5"} Oct 30 21:51:20.078822 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:20.078486 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b077ad14-ff57-45ba-87c3-853a2e104003","Type":"ContainerStarted","Data":"2c8996eab759ddd1611025477c350f056be62383f47167862c583f6af7868c1c"} Oct 30 21:51:20.078822 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:20.078508 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b077ad14-ff57-45ba-87c3-853a2e104003","Type":"ContainerStarted","Data":"3e5557d7ca60f0285ffa6592a0cd5e1443a710815852dfb2cf86581985e9cb1c"} Oct 30 21:51:20.211446 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:20.211397 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.198858452 podStartE2EDuration="3.211385648s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.527196ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:20.210838977 +0000 UTC m=+40.481537057" watchObservedRunningTime="2026-10-30 21:51:20.211385648 +0000 UTC m=+40.482083726" Oct 30 21:51:21.274371 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:51:21.274339 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 21:51:22.099114 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.099082 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:51:22.099557 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.099390 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="736951e5-1596-4606-94d9-80d8b50d35c3" containerName="prometheus" containerID="cri-o://03bf42adb14c6eafce0516556870e7dd844847b9277150c44db7514cc4eb96cd" gracePeriod=600 Oct 30 21:51:22.099557 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.099438 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="736951e5-1596-4606-94d9-80d8b50d35c3" containerName="thanos-sidecar" containerID="cri-o://c436cf179edf23782012024b058c39f01184ff419de08fd9c33ba0cf17321ed0" gracePeriod=600 Oct 30 21:51:22.099557 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.099435 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="736951e5-1596-4606-94d9-80d8b50d35c3" containerName="kube-rbac-proxy" containerID="cri-o://b1ddaa6943a6a9fa826e141df1d6d73be3a699d6cdbd5c118deb80cd2f007ee3" gracePeriod=600 Oct 30 21:51:22.099557 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.099445 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="736951e5-1596-4606-94d9-80d8b50d35c3" containerName="kube-rbac-proxy-web" containerID="cri-o://4d7dc66cbc297b493c2282a3d67bdb66bdeabbe7484d47c4b8d8c6c5ca9d70f1" gracePeriod=600 Oct 30 21:51:22.099557 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.099439 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="736951e5-1596-4606-94d9-80d8b50d35c3" containerName="kube-rbac-proxy-thanos" containerID="cri-o://6e673397a1f3c2825a68224eb4b922c2f0a5f03c6051241bba2c74b99abb1f6d" gracePeriod=600 Oct 30 21:51:22.099557 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.099454 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="736951e5-1596-4606-94d9-80d8b50d35c3" containerName="config-reloader" containerID="cri-o://7f6a9182c1c60364249dd1d140565cfb4a5f1960e32a8bf97d90413b5212beb1" gracePeriod=600 Oct 30 21:51:22.336289 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.336261 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:22.396844 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.396813 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/736951e5-1596-4606-94d9-80d8b50d35c3-kube-api-access-6nd2d\" (UniqueName: \"kubernetes.io/projected/736951e5-1596-4606-94d9-80d8b50d35c3-kube-api-access-6nd2d\") pod \"736951e5-1596-4606-94d9-80d8b50d35c3\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " Oct 30 21:51:22.396991 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.396853 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-trusted-ca-bundle\") pod \"736951e5-1596-4606-94d9-80d8b50d35c3\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " Oct 30 21:51:22.396991 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.396882 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-config\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-config\") pod \"736951e5-1596-4606-94d9-80d8b50d35c3\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " Oct 30 21:51:22.396991 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.396925 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"736951e5-1596-4606-94d9-80d8b50d35c3\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " Oct 30 21:51:22.396991 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.396957 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-configmap-serving-certs-ca-bundle\") pod \"736951e5-1596-4606-94d9-80d8b50d35c3\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " Oct 30 21:51:22.396991 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.396981 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-metrics-client-certs\") pod \"736951e5-1596-4606-94d9-80d8b50d35c3\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " Oct 30 21:51:22.397283 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.397025 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-rulefiles-2\") pod \"736951e5-1596-4606-94d9-80d8b50d35c3\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " Oct 30 21:51:22.397283 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.397086 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/736951e5-1596-4606-94d9-80d8b50d35c3-tls-assets\" (UniqueName: \"kubernetes.io/projected/736951e5-1596-4606-94d9-80d8b50d35c3-tls-assets\") pod \"736951e5-1596-4606-94d9-80d8b50d35c3\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " Oct 30 21:51:22.397283 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.397128 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"736951e5-1596-4606-94d9-80d8b50d35c3\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " Oct 30 21:51:22.397283 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.397158 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/736951e5-1596-4606-94d9-80d8b50d35c3-config-out\" (UniqueName: \"kubernetes.io/empty-dir/736951e5-1596-4606-94d9-80d8b50d35c3-config-out\") pod \"736951e5-1596-4606-94d9-80d8b50d35c3\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " Oct 30 21:51:22.397283 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.397195 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-rulefiles-0\") pod \"736951e5-1596-4606-94d9-80d8b50d35c3\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " Oct 30 21:51:22.397283 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.397254 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-rulefiles-1\") pod \"736951e5-1596-4606-94d9-80d8b50d35c3\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " Oct 30 21:51:22.397553 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.397285 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-configmap-kubelet-serving-ca-bundle\") pod \"736951e5-1596-4606-94d9-80d8b50d35c3\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " Oct 30 21:51:22.397553 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.397317 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-prometheus-k8s-tls\") pod \"736951e5-1596-4606-94d9-80d8b50d35c3\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " Oct 30 21:51:22.397553 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.397367 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-thanos-prometheus-http-client-file\") pod \"736951e5-1596-4606-94d9-80d8b50d35c3\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " Oct 30 21:51:22.397553 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.397397 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-configmap-metrics-client-ca\") pod \"736951e5-1596-4606-94d9-80d8b50d35c3\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " Oct 30 21:51:22.397553 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.397431 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-db\") pod \"736951e5-1596-4606-94d9-80d8b50d35c3\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " Oct 30 21:51:22.397927 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.397907 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-web-config\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-web-config\") pod \"736951e5-1596-4606-94d9-80d8b50d35c3\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " Oct 30 21:51:22.397989 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.397965 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-grpc-tls\") pod \"736951e5-1596-4606-94d9-80d8b50d35c3\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " Oct 30 21:51:22.398134 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.398119 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-kube-rbac-proxy\") pod \"736951e5-1596-4606-94d9-80d8b50d35c3\" (UID: \"736951e5-1596-4606-94d9-80d8b50d35c3\") " Oct 30 21:51:22.399632 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.397277 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-trusted-ca-bundle" pod "736951e5-1596-4606-94d9-80d8b50d35c3" (UID: "736951e5-1596-4606-94d9-80d8b50d35c3"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:22.399632 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.398311 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-rulefiles-2" pod "736951e5-1596-4606-94d9-80d8b50d35c3" (UID: "736951e5-1596-4606-94d9-80d8b50d35c3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:22.399632 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.398433 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-configmap-serving-certs-ca-bundle" pod "736951e5-1596-4606-94d9-80d8b50d35c3" (UID: "736951e5-1596-4606-94d9-80d8b50d35c3"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:22.399882 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.399856 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-rulefiles-1" pod "736951e5-1596-4606-94d9-80d8b50d35c3" (UID: "736951e5-1596-4606-94d9-80d8b50d35c3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:22.399956 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.399918 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-configmap-metrics-client-ca" pod "736951e5-1596-4606-94d9-80d8b50d35c3" (UID: "736951e5-1596-4606-94d9-80d8b50d35c3"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:22.400011 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.399960 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-rulefiles-0" pod "736951e5-1596-4606-94d9-80d8b50d35c3" (UID: "736951e5-1596-4606-94d9-80d8b50d35c3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:22.400625 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.400593 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-configmap-kubelet-serving-ca-bundle" pod "736951e5-1596-4606-94d9-80d8b50d35c3" (UID: "736951e5-1596-4606-94d9-80d8b50d35c3"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:22.401589 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.401324 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-db" pod "736951e5-1596-4606-94d9-80d8b50d35c3" (UID: "736951e5-1596-4606-94d9-80d8b50d35c3"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:22.401759 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.401651 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/736951e5-1596-4606-94d9-80d8b50d35c3-kube-api-access-6nd2d" pod "736951e5-1596-4606-94d9-80d8b50d35c3" (UID: "736951e5-1596-4606-94d9-80d8b50d35c3"). InnerVolumeSpecName "kube-api-access-6nd2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:22.405969 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.405809 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-kube-rbac-proxy" pod "736951e5-1596-4606-94d9-80d8b50d35c3" (UID: "736951e5-1596-4606-94d9-80d8b50d35c3"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:22.405969 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.405854 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-config" pod "736951e5-1596-4606-94d9-80d8b50d35c3" (UID: "736951e5-1596-4606-94d9-80d8b50d35c3"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:22.406126 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.406075 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-prometheus-k8s-thanos-sidecar-tls" pod "736951e5-1596-4606-94d9-80d8b50d35c3" (UID: "736951e5-1596-4606-94d9-80d8b50d35c3"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:22.406370 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.406342 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-metrics-client-certs" pod "736951e5-1596-4606-94d9-80d8b50d35c3" (UID: "736951e5-1596-4606-94d9-80d8b50d35c3"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:22.406506 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.406483 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-grpc-tls" pod "736951e5-1596-4606-94d9-80d8b50d35c3" (UID: "736951e5-1596-4606-94d9-80d8b50d35c3"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:22.406615 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.406411 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/736951e5-1596-4606-94d9-80d8b50d35c3-tls-assets" pod "736951e5-1596-4606-94d9-80d8b50d35c3" (UID: "736951e5-1596-4606-94d9-80d8b50d35c3"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:22.406765 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.406640 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-thanos-prometheus-http-client-file" pod "736951e5-1596-4606-94d9-80d8b50d35c3" (UID: "736951e5-1596-4606-94d9-80d8b50d35c3"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:22.406765 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.406686 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736951e5-1596-4606-94d9-80d8b50d35c3-config-out" pod "736951e5-1596-4606-94d9-80d8b50d35c3" (UID: "736951e5-1596-4606-94d9-80d8b50d35c3"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:22.406872 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.406786 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-prometheus-k8s-tls" pod "736951e5-1596-4606-94d9-80d8b50d35c3" (UID: "736951e5-1596-4606-94d9-80d8b50d35c3"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:22.407281 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.407200 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-prometheus-k8s-kube-rbac-proxy-web" pod "736951e5-1596-4606-94d9-80d8b50d35c3" (UID: "736951e5-1596-4606-94d9-80d8b50d35c3"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:22.415778 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.415758 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-web-config" pod "736951e5-1596-4606-94d9-80d8b50d35c3" (UID: "736951e5-1596-4606-94d9-80d8b50d35c3"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:22.502260 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.502210 5918 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.502260 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.502253 5918 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/736951e5-1596-4606-94d9-80d8b50d35c3-tls-assets\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.502260 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.502264 5918 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.502472 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.502274 5918 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/736951e5-1596-4606-94d9-80d8b50d35c3-config-out\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.502472 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.502284 5918 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.502472 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.502292 5918 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.502472 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.502301 5918 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.502472 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.502311 5918 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.502472 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.502319 5918 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.502472 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.502327 5918 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-configmap-metrics-client-ca\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.502472 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.502335 5918 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-k8s-db\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.502472 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.502343 5918 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-web-config\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.502472 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.502352 5918 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-grpc-tls\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.502472 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.502360 5918 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-kube-rbac-proxy\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.502472 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.502368 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nd2d\" (UniqueName: \"kubernetes.io/projected/736951e5-1596-4606-94d9-80d8b50d35c3-kube-api-access-6nd2d\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.502472 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.502377 5918 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.502472 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.502385 5918 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-config\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.502472 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.502394 5918 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.502472 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.502402 5918 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/736951e5-1596-4606-94d9-80d8b50d35c3-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:22.502472 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:22.502410 5918 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/736951e5-1596-4606-94d9-80d8b50d35c3-secret-metrics-client-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:23.092057 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.092024 5918 generic.go:356] "Generic (PLEG): container finished" podID="736951e5-1596-4606-94d9-80d8b50d35c3" containerID="6e673397a1f3c2825a68224eb4b922c2f0a5f03c6051241bba2c74b99abb1f6d" exitCode=0 Oct 30 21:51:23.092057 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.092049 5918 generic.go:356] "Generic (PLEG): container finished" podID="736951e5-1596-4606-94d9-80d8b50d35c3" containerID="b1ddaa6943a6a9fa826e141df1d6d73be3a699d6cdbd5c118deb80cd2f007ee3" exitCode=0 Oct 30 21:51:23.092057 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.092058 5918 generic.go:356] "Generic (PLEG): container finished" podID="736951e5-1596-4606-94d9-80d8b50d35c3" containerID="4d7dc66cbc297b493c2282a3d67bdb66bdeabbe7484d47c4b8d8c6c5ca9d70f1" exitCode=0 Oct 30 21:51:23.092057 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.092063 5918 generic.go:356] "Generic (PLEG): container finished" podID="736951e5-1596-4606-94d9-80d8b50d35c3" containerID="c436cf179edf23782012024b058c39f01184ff419de08fd9c33ba0cf17321ed0" exitCode=0 Oct 30 21:51:23.092364 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.092069 5918 generic.go:356] "Generic (PLEG): container finished" podID="736951e5-1596-4606-94d9-80d8b50d35c3" containerID="7f6a9182c1c60364249dd1d140565cfb4a5f1960e32a8bf97d90413b5212beb1" exitCode=0 Oct 30 21:51:23.092364 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.092075 5918 generic.go:356] "Generic (PLEG): container finished" podID="736951e5-1596-4606-94d9-80d8b50d35c3" containerID="03bf42adb14c6eafce0516556870e7dd844847b9277150c44db7514cc4eb96cd" exitCode=0 Oct 30 21:51:23.092364 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.092099 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"736951e5-1596-4606-94d9-80d8b50d35c3","Type":"ContainerDied","Data":"6e673397a1f3c2825a68224eb4b922c2f0a5f03c6051241bba2c74b99abb1f6d"} Oct 30 21:51:23.092364 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.092118 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.092364 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.092174 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"736951e5-1596-4606-94d9-80d8b50d35c3","Type":"ContainerDied","Data":"b1ddaa6943a6a9fa826e141df1d6d73be3a699d6cdbd5c118deb80cd2f007ee3"} Oct 30 21:51:23.092364 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.092204 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"736951e5-1596-4606-94d9-80d8b50d35c3","Type":"ContainerDied","Data":"4d7dc66cbc297b493c2282a3d67bdb66bdeabbe7484d47c4b8d8c6c5ca9d70f1"} Oct 30 21:51:23.092364 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.092238 5918 scope.go:122] "RemoveContainer" containerID="6e673397a1f3c2825a68224eb4b922c2f0a5f03c6051241bba2c74b99abb1f6d" Oct 30 21:51:23.092364 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.092241 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"736951e5-1596-4606-94d9-80d8b50d35c3","Type":"ContainerDied","Data":"c436cf179edf23782012024b058c39f01184ff419de08fd9c33ba0cf17321ed0"} Oct 30 21:51:23.092364 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.092266 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"736951e5-1596-4606-94d9-80d8b50d35c3","Type":"ContainerDied","Data":"7f6a9182c1c60364249dd1d140565cfb4a5f1960e32a8bf97d90413b5212beb1"} Oct 30 21:51:23.092364 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.092289 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"736951e5-1596-4606-94d9-80d8b50d35c3","Type":"ContainerDied","Data":"03bf42adb14c6eafce0516556870e7dd844847b9277150c44db7514cc4eb96cd"} Oct 30 21:51:23.092364 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.092312 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"736951e5-1596-4606-94d9-80d8b50d35c3","Type":"ContainerDied","Data":"f9fae51e3797f359f826f7d5c8436433cd1dae5fa3210c0f63881d21e1a27259"} Oct 30 21:51:23.100809 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.100667 5918 scope.go:122] "RemoveContainer" containerID="b1ddaa6943a6a9fa826e141df1d6d73be3a699d6cdbd5c118deb80cd2f007ee3" Oct 30 21:51:23.108178 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.108162 5918 scope.go:122] "RemoveContainer" containerID="4d7dc66cbc297b493c2282a3d67bdb66bdeabbe7484d47c4b8d8c6c5ca9d70f1" Oct 30 21:51:23.122655 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.122639 5918 scope.go:122] "RemoveContainer" containerID="c436cf179edf23782012024b058c39f01184ff419de08fd9c33ba0cf17321ed0" Oct 30 21:51:23.129449 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.129427 5918 scope.go:122] "RemoveContainer" containerID="7f6a9182c1c60364249dd1d140565cfb4a5f1960e32a8bf97d90413b5212beb1" Oct 30 21:51:23.135970 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.135953 5918 scope.go:122] "RemoveContainer" containerID="03bf42adb14c6eafce0516556870e7dd844847b9277150c44db7514cc4eb96cd" Oct 30 21:51:23.142825 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.142809 5918 scope.go:122] "RemoveContainer" containerID="6ecd79a7c6d356f8e64253560432de55058bb4c2da2408ddf9e73b11af211256" Oct 30 21:51:23.149228 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.149202 5918 scope.go:122] "RemoveContainer" containerID="6e673397a1f3c2825a68224eb4b922c2f0a5f03c6051241bba2c74b99abb1f6d" Oct 30 21:51:23.149499 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:51:23.149479 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e673397a1f3c2825a68224eb4b922c2f0a5f03c6051241bba2c74b99abb1f6d\": container with ID starting with 6e673397a1f3c2825a68224eb4b922c2f0a5f03c6051241bba2c74b99abb1f6d not found: ID does not exist" containerID="6e673397a1f3c2825a68224eb4b922c2f0a5f03c6051241bba2c74b99abb1f6d" Oct 30 21:51:23.149559 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.149512 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e673397a1f3c2825a68224eb4b922c2f0a5f03c6051241bba2c74b99abb1f6d"} err="failed to get container status \"6e673397a1f3c2825a68224eb4b922c2f0a5f03c6051241bba2c74b99abb1f6d\": rpc error: code = NotFound desc = could not find container \"6e673397a1f3c2825a68224eb4b922c2f0a5f03c6051241bba2c74b99abb1f6d\": container with ID starting with 6e673397a1f3c2825a68224eb4b922c2f0a5f03c6051241bba2c74b99abb1f6d not found: ID does not exist" Oct 30 21:51:23.149559 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.149529 5918 scope.go:122] "RemoveContainer" containerID="b1ddaa6943a6a9fa826e141df1d6d73be3a699d6cdbd5c118deb80cd2f007ee3" Oct 30 21:51:23.149764 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:51:23.149747 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1ddaa6943a6a9fa826e141df1d6d73be3a699d6cdbd5c118deb80cd2f007ee3\": container with ID starting with b1ddaa6943a6a9fa826e141df1d6d73be3a699d6cdbd5c118deb80cd2f007ee3 not found: ID does not exist" containerID="b1ddaa6943a6a9fa826e141df1d6d73be3a699d6cdbd5c118deb80cd2f007ee3" Oct 30 21:51:23.149806 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.149770 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1ddaa6943a6a9fa826e141df1d6d73be3a699d6cdbd5c118deb80cd2f007ee3"} err="failed to get container status \"b1ddaa6943a6a9fa826e141df1d6d73be3a699d6cdbd5c118deb80cd2f007ee3\": rpc error: code = NotFound desc = could not find container \"b1ddaa6943a6a9fa826e141df1d6d73be3a699d6cdbd5c118deb80cd2f007ee3\": container with ID starting with b1ddaa6943a6a9fa826e141df1d6d73be3a699d6cdbd5c118deb80cd2f007ee3 not found: ID does not exist" Oct 30 21:51:23.149806 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.149784 5918 scope.go:122] "RemoveContainer" containerID="4d7dc66cbc297b493c2282a3d67bdb66bdeabbe7484d47c4b8d8c6c5ca9d70f1" Oct 30 21:51:23.150041 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:51:23.150027 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d7dc66cbc297b493c2282a3d67bdb66bdeabbe7484d47c4b8d8c6c5ca9d70f1\": container with ID starting with 4d7dc66cbc297b493c2282a3d67bdb66bdeabbe7484d47c4b8d8c6c5ca9d70f1 not found: ID does not exist" containerID="4d7dc66cbc297b493c2282a3d67bdb66bdeabbe7484d47c4b8d8c6c5ca9d70f1" Oct 30 21:51:23.150082 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.150044 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d7dc66cbc297b493c2282a3d67bdb66bdeabbe7484d47c4b8d8c6c5ca9d70f1"} err="failed to get container status \"4d7dc66cbc297b493c2282a3d67bdb66bdeabbe7484d47c4b8d8c6c5ca9d70f1\": rpc error: code = NotFound desc = could not find container \"4d7dc66cbc297b493c2282a3d67bdb66bdeabbe7484d47c4b8d8c6c5ca9d70f1\": container with ID starting with 4d7dc66cbc297b493c2282a3d67bdb66bdeabbe7484d47c4b8d8c6c5ca9d70f1 not found: ID does not exist" Oct 30 21:51:23.150082 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.150056 5918 scope.go:122] "RemoveContainer" containerID="c436cf179edf23782012024b058c39f01184ff419de08fd9c33ba0cf17321ed0" Oct 30 21:51:23.150273 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:51:23.150257 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c436cf179edf23782012024b058c39f01184ff419de08fd9c33ba0cf17321ed0\": container with ID starting with c436cf179edf23782012024b058c39f01184ff419de08fd9c33ba0cf17321ed0 not found: ID does not exist" containerID="c436cf179edf23782012024b058c39f01184ff419de08fd9c33ba0cf17321ed0" Oct 30 21:51:23.150322 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.150276 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c436cf179edf23782012024b058c39f01184ff419de08fd9c33ba0cf17321ed0"} err="failed to get container status \"c436cf179edf23782012024b058c39f01184ff419de08fd9c33ba0cf17321ed0\": rpc error: code = NotFound desc = could not find container \"c436cf179edf23782012024b058c39f01184ff419de08fd9c33ba0cf17321ed0\": container with ID starting with c436cf179edf23782012024b058c39f01184ff419de08fd9c33ba0cf17321ed0 not found: ID does not exist" Oct 30 21:51:23.150322 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.150289 5918 scope.go:122] "RemoveContainer" containerID="7f6a9182c1c60364249dd1d140565cfb4a5f1960e32a8bf97d90413b5212beb1" Oct 30 21:51:23.150483 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:51:23.150468 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f6a9182c1c60364249dd1d140565cfb4a5f1960e32a8bf97d90413b5212beb1\": container with ID starting with 7f6a9182c1c60364249dd1d140565cfb4a5f1960e32a8bf97d90413b5212beb1 not found: ID does not exist" containerID="7f6a9182c1c60364249dd1d140565cfb4a5f1960e32a8bf97d90413b5212beb1" Oct 30 21:51:23.150518 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.150486 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f6a9182c1c60364249dd1d140565cfb4a5f1960e32a8bf97d90413b5212beb1"} err="failed to get container status \"7f6a9182c1c60364249dd1d140565cfb4a5f1960e32a8bf97d90413b5212beb1\": rpc error: code = NotFound desc = could not find container \"7f6a9182c1c60364249dd1d140565cfb4a5f1960e32a8bf97d90413b5212beb1\": container with ID starting with 7f6a9182c1c60364249dd1d140565cfb4a5f1960e32a8bf97d90413b5212beb1 not found: ID does not exist" Oct 30 21:51:23.150518 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.150498 5918 scope.go:122] "RemoveContainer" containerID="03bf42adb14c6eafce0516556870e7dd844847b9277150c44db7514cc4eb96cd" Oct 30 21:51:23.150706 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:51:23.150691 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03bf42adb14c6eafce0516556870e7dd844847b9277150c44db7514cc4eb96cd\": container with ID starting with 03bf42adb14c6eafce0516556870e7dd844847b9277150c44db7514cc4eb96cd not found: ID does not exist" containerID="03bf42adb14c6eafce0516556870e7dd844847b9277150c44db7514cc4eb96cd" Oct 30 21:51:23.150867 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.150711 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03bf42adb14c6eafce0516556870e7dd844847b9277150c44db7514cc4eb96cd"} err="failed to get container status \"03bf42adb14c6eafce0516556870e7dd844847b9277150c44db7514cc4eb96cd\": rpc error: code = NotFound desc = could not find container \"03bf42adb14c6eafce0516556870e7dd844847b9277150c44db7514cc4eb96cd\": container with ID starting with 03bf42adb14c6eafce0516556870e7dd844847b9277150c44db7514cc4eb96cd not found: ID does not exist" Oct 30 21:51:23.150867 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.150721 5918 scope.go:122] "RemoveContainer" containerID="6ecd79a7c6d356f8e64253560432de55058bb4c2da2408ddf9e73b11af211256" Oct 30 21:51:23.150946 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:51:23.150933 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ecd79a7c6d356f8e64253560432de55058bb4c2da2408ddf9e73b11af211256\": container with ID starting with 6ecd79a7c6d356f8e64253560432de55058bb4c2da2408ddf9e73b11af211256 not found: ID does not exist" containerID="6ecd79a7c6d356f8e64253560432de55058bb4c2da2408ddf9e73b11af211256" Oct 30 21:51:23.150993 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.150951 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ecd79a7c6d356f8e64253560432de55058bb4c2da2408ddf9e73b11af211256"} err="failed to get container status \"6ecd79a7c6d356f8e64253560432de55058bb4c2da2408ddf9e73b11af211256\": rpc error: code = NotFound desc = could not find container \"6ecd79a7c6d356f8e64253560432de55058bb4c2da2408ddf9e73b11af211256\": container with ID starting with 6ecd79a7c6d356f8e64253560432de55058bb4c2da2408ddf9e73b11af211256 not found: ID does not exist" Oct 30 21:51:23.150993 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.150966 5918 scope.go:122] "RemoveContainer" containerID="6e673397a1f3c2825a68224eb4b922c2f0a5f03c6051241bba2c74b99abb1f6d" Oct 30 21:51:23.151184 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.151164 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e673397a1f3c2825a68224eb4b922c2f0a5f03c6051241bba2c74b99abb1f6d"} err="failed to get container status \"6e673397a1f3c2825a68224eb4b922c2f0a5f03c6051241bba2c74b99abb1f6d\": rpc error: code = NotFound desc = could not find container \"6e673397a1f3c2825a68224eb4b922c2f0a5f03c6051241bba2c74b99abb1f6d\": container with ID starting with 6e673397a1f3c2825a68224eb4b922c2f0a5f03c6051241bba2c74b99abb1f6d not found: ID does not exist" Oct 30 21:51:23.151375 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.151185 5918 scope.go:122] "RemoveContainer" containerID="b1ddaa6943a6a9fa826e141df1d6d73be3a699d6cdbd5c118deb80cd2f007ee3" Oct 30 21:51:23.151427 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.151415 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1ddaa6943a6a9fa826e141df1d6d73be3a699d6cdbd5c118deb80cd2f007ee3"} err="failed to get container status \"b1ddaa6943a6a9fa826e141df1d6d73be3a699d6cdbd5c118deb80cd2f007ee3\": rpc error: code = NotFound desc = could not find container \"b1ddaa6943a6a9fa826e141df1d6d73be3a699d6cdbd5c118deb80cd2f007ee3\": container with ID starting with b1ddaa6943a6a9fa826e141df1d6d73be3a699d6cdbd5c118deb80cd2f007ee3 not found: ID does not exist" Oct 30 21:51:23.151471 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.151430 5918 scope.go:122] "RemoveContainer" containerID="4d7dc66cbc297b493c2282a3d67bdb66bdeabbe7484d47c4b8d8c6c5ca9d70f1" Oct 30 21:51:23.151655 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.151633 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d7dc66cbc297b493c2282a3d67bdb66bdeabbe7484d47c4b8d8c6c5ca9d70f1"} err="failed to get container status \"4d7dc66cbc297b493c2282a3d67bdb66bdeabbe7484d47c4b8d8c6c5ca9d70f1\": rpc error: code = NotFound desc = could not find container \"4d7dc66cbc297b493c2282a3d67bdb66bdeabbe7484d47c4b8d8c6c5ca9d70f1\": container with ID starting with 4d7dc66cbc297b493c2282a3d67bdb66bdeabbe7484d47c4b8d8c6c5ca9d70f1 not found: ID does not exist" Oct 30 21:51:23.151729 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.151656 5918 scope.go:122] "RemoveContainer" containerID="c436cf179edf23782012024b058c39f01184ff419de08fd9c33ba0cf17321ed0" Oct 30 21:51:23.151893 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.151877 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c436cf179edf23782012024b058c39f01184ff419de08fd9c33ba0cf17321ed0"} err="failed to get container status \"c436cf179edf23782012024b058c39f01184ff419de08fd9c33ba0cf17321ed0\": rpc error: code = NotFound desc = could not find container \"c436cf179edf23782012024b058c39f01184ff419de08fd9c33ba0cf17321ed0\": container with ID starting with c436cf179edf23782012024b058c39f01184ff419de08fd9c33ba0cf17321ed0 not found: ID does not exist" Oct 30 21:51:23.151938 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.151893 5918 scope.go:122] "RemoveContainer" containerID="7f6a9182c1c60364249dd1d140565cfb4a5f1960e32a8bf97d90413b5212beb1" Oct 30 21:51:23.152113 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.152095 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f6a9182c1c60364249dd1d140565cfb4a5f1960e32a8bf97d90413b5212beb1"} err="failed to get container status \"7f6a9182c1c60364249dd1d140565cfb4a5f1960e32a8bf97d90413b5212beb1\": rpc error: code = NotFound desc = could not find container \"7f6a9182c1c60364249dd1d140565cfb4a5f1960e32a8bf97d90413b5212beb1\": container with ID starting with 7f6a9182c1c60364249dd1d140565cfb4a5f1960e32a8bf97d90413b5212beb1 not found: ID does not exist" Oct 30 21:51:23.152180 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.152114 5918 scope.go:122] "RemoveContainer" containerID="03bf42adb14c6eafce0516556870e7dd844847b9277150c44db7514cc4eb96cd" Oct 30 21:51:23.152337 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.152322 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03bf42adb14c6eafce0516556870e7dd844847b9277150c44db7514cc4eb96cd"} err="failed to get container status \"03bf42adb14c6eafce0516556870e7dd844847b9277150c44db7514cc4eb96cd\": rpc error: code = NotFound desc = could not find container \"03bf42adb14c6eafce0516556870e7dd844847b9277150c44db7514cc4eb96cd\": container with ID starting with 03bf42adb14c6eafce0516556870e7dd844847b9277150c44db7514cc4eb96cd not found: ID does not exist" Oct 30 21:51:23.152388 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.152337 5918 scope.go:122] "RemoveContainer" containerID="6ecd79a7c6d356f8e64253560432de55058bb4c2da2408ddf9e73b11af211256" Oct 30 21:51:23.152560 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.152542 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ecd79a7c6d356f8e64253560432de55058bb4c2da2408ddf9e73b11af211256"} err="failed to get container status \"6ecd79a7c6d356f8e64253560432de55058bb4c2da2408ddf9e73b11af211256\": rpc error: code = NotFound desc = could not find container \"6ecd79a7c6d356f8e64253560432de55058bb4c2da2408ddf9e73b11af211256\": container with ID starting with 6ecd79a7c6d356f8e64253560432de55058bb4c2da2408ddf9e73b11af211256 not found: ID does not exist" Oct 30 21:51:23.152603 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.152562 5918 scope.go:122] "RemoveContainer" containerID="6e673397a1f3c2825a68224eb4b922c2f0a5f03c6051241bba2c74b99abb1f6d" Oct 30 21:51:23.152776 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.152758 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e673397a1f3c2825a68224eb4b922c2f0a5f03c6051241bba2c74b99abb1f6d"} err="failed to get container status \"6e673397a1f3c2825a68224eb4b922c2f0a5f03c6051241bba2c74b99abb1f6d\": rpc error: code = NotFound desc = could not find container \"6e673397a1f3c2825a68224eb4b922c2f0a5f03c6051241bba2c74b99abb1f6d\": container with ID starting with 6e673397a1f3c2825a68224eb4b922c2f0a5f03c6051241bba2c74b99abb1f6d not found: ID does not exist" Oct 30 21:51:23.152842 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.152776 5918 scope.go:122] "RemoveContainer" containerID="b1ddaa6943a6a9fa826e141df1d6d73be3a699d6cdbd5c118deb80cd2f007ee3" Oct 30 21:51:23.152994 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.152977 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1ddaa6943a6a9fa826e141df1d6d73be3a699d6cdbd5c118deb80cd2f007ee3"} err="failed to get container status \"b1ddaa6943a6a9fa826e141df1d6d73be3a699d6cdbd5c118deb80cd2f007ee3\": rpc error: code = NotFound desc = could not find container \"b1ddaa6943a6a9fa826e141df1d6d73be3a699d6cdbd5c118deb80cd2f007ee3\": container with ID starting with b1ddaa6943a6a9fa826e141df1d6d73be3a699d6cdbd5c118deb80cd2f007ee3 not found: ID does not exist" Oct 30 21:51:23.153039 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.152995 5918 scope.go:122] "RemoveContainer" containerID="4d7dc66cbc297b493c2282a3d67bdb66bdeabbe7484d47c4b8d8c6c5ca9d70f1" Oct 30 21:51:23.153186 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.153165 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d7dc66cbc297b493c2282a3d67bdb66bdeabbe7484d47c4b8d8c6c5ca9d70f1"} err="failed to get container status \"4d7dc66cbc297b493c2282a3d67bdb66bdeabbe7484d47c4b8d8c6c5ca9d70f1\": rpc error: code = NotFound desc = could not find container \"4d7dc66cbc297b493c2282a3d67bdb66bdeabbe7484d47c4b8d8c6c5ca9d70f1\": container with ID starting with 4d7dc66cbc297b493c2282a3d67bdb66bdeabbe7484d47c4b8d8c6c5ca9d70f1 not found: ID does not exist" Oct 30 21:51:23.153250 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.153186 5918 scope.go:122] "RemoveContainer" containerID="c436cf179edf23782012024b058c39f01184ff419de08fd9c33ba0cf17321ed0" Oct 30 21:51:23.153420 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.153404 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c436cf179edf23782012024b058c39f01184ff419de08fd9c33ba0cf17321ed0"} err="failed to get container status \"c436cf179edf23782012024b058c39f01184ff419de08fd9c33ba0cf17321ed0\": rpc error: code = NotFound desc = could not find container \"c436cf179edf23782012024b058c39f01184ff419de08fd9c33ba0cf17321ed0\": container with ID starting with c436cf179edf23782012024b058c39f01184ff419de08fd9c33ba0cf17321ed0 not found: ID does not exist" Oct 30 21:51:23.153480 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.153420 5918 scope.go:122] "RemoveContainer" containerID="7f6a9182c1c60364249dd1d140565cfb4a5f1960e32a8bf97d90413b5212beb1" Oct 30 21:51:23.153632 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.153618 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f6a9182c1c60364249dd1d140565cfb4a5f1960e32a8bf97d90413b5212beb1"} err="failed to get container status \"7f6a9182c1c60364249dd1d140565cfb4a5f1960e32a8bf97d90413b5212beb1\": rpc error: code = NotFound desc = could not find container \"7f6a9182c1c60364249dd1d140565cfb4a5f1960e32a8bf97d90413b5212beb1\": container with ID starting with 7f6a9182c1c60364249dd1d140565cfb4a5f1960e32a8bf97d90413b5212beb1 not found: ID does not exist" Oct 30 21:51:23.153672 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.153632 5918 scope.go:122] "RemoveContainer" containerID="03bf42adb14c6eafce0516556870e7dd844847b9277150c44db7514cc4eb96cd" Oct 30 21:51:23.153824 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.153808 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03bf42adb14c6eafce0516556870e7dd844847b9277150c44db7514cc4eb96cd"} err="failed to get container status \"03bf42adb14c6eafce0516556870e7dd844847b9277150c44db7514cc4eb96cd\": rpc error: code = NotFound desc = could not find container \"03bf42adb14c6eafce0516556870e7dd844847b9277150c44db7514cc4eb96cd\": container with ID starting with 03bf42adb14c6eafce0516556870e7dd844847b9277150c44db7514cc4eb96cd not found: ID does not exist" Oct 30 21:51:23.153888 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.153824 5918 scope.go:122] "RemoveContainer" containerID="6ecd79a7c6d356f8e64253560432de55058bb4c2da2408ddf9e73b11af211256" Oct 30 21:51:23.154042 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.154025 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ecd79a7c6d356f8e64253560432de55058bb4c2da2408ddf9e73b11af211256"} err="failed to get container status \"6ecd79a7c6d356f8e64253560432de55058bb4c2da2408ddf9e73b11af211256\": rpc error: code = NotFound desc = could not find container \"6ecd79a7c6d356f8e64253560432de55058bb4c2da2408ddf9e73b11af211256\": container with ID starting with 6ecd79a7c6d356f8e64253560432de55058bb4c2da2408ddf9e73b11af211256 not found: ID does not exist" Oct 30 21:51:23.154085 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.154043 5918 scope.go:122] "RemoveContainer" containerID="6e673397a1f3c2825a68224eb4b922c2f0a5f03c6051241bba2c74b99abb1f6d" Oct 30 21:51:23.154279 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.154254 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e673397a1f3c2825a68224eb4b922c2f0a5f03c6051241bba2c74b99abb1f6d"} err="failed to get container status \"6e673397a1f3c2825a68224eb4b922c2f0a5f03c6051241bba2c74b99abb1f6d\": rpc error: code = NotFound desc = could not find container \"6e673397a1f3c2825a68224eb4b922c2f0a5f03c6051241bba2c74b99abb1f6d\": container with ID starting with 6e673397a1f3c2825a68224eb4b922c2f0a5f03c6051241bba2c74b99abb1f6d not found: ID does not exist" Oct 30 21:51:23.154279 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.154278 5918 scope.go:122] "RemoveContainer" containerID="b1ddaa6943a6a9fa826e141df1d6d73be3a699d6cdbd5c118deb80cd2f007ee3" Oct 30 21:51:23.154523 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.154506 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1ddaa6943a6a9fa826e141df1d6d73be3a699d6cdbd5c118deb80cd2f007ee3"} err="failed to get container status \"b1ddaa6943a6a9fa826e141df1d6d73be3a699d6cdbd5c118deb80cd2f007ee3\": rpc error: code = NotFound desc = could not find container \"b1ddaa6943a6a9fa826e141df1d6d73be3a699d6cdbd5c118deb80cd2f007ee3\": container with ID starting with b1ddaa6943a6a9fa826e141df1d6d73be3a699d6cdbd5c118deb80cd2f007ee3 not found: ID does not exist" Oct 30 21:51:23.154570 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.154523 5918 scope.go:122] "RemoveContainer" containerID="4d7dc66cbc297b493c2282a3d67bdb66bdeabbe7484d47c4b8d8c6c5ca9d70f1" Oct 30 21:51:23.154744 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.154728 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d7dc66cbc297b493c2282a3d67bdb66bdeabbe7484d47c4b8d8c6c5ca9d70f1"} err="failed to get container status \"4d7dc66cbc297b493c2282a3d67bdb66bdeabbe7484d47c4b8d8c6c5ca9d70f1\": rpc error: code = NotFound desc = could not find container \"4d7dc66cbc297b493c2282a3d67bdb66bdeabbe7484d47c4b8d8c6c5ca9d70f1\": container with ID starting with 4d7dc66cbc297b493c2282a3d67bdb66bdeabbe7484d47c4b8d8c6c5ca9d70f1 not found: ID does not exist" Oct 30 21:51:23.154805 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.154744 5918 scope.go:122] "RemoveContainer" containerID="c436cf179edf23782012024b058c39f01184ff419de08fd9c33ba0cf17321ed0" Oct 30 21:51:23.154957 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.154943 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c436cf179edf23782012024b058c39f01184ff419de08fd9c33ba0cf17321ed0"} err="failed to get container status \"c436cf179edf23782012024b058c39f01184ff419de08fd9c33ba0cf17321ed0\": rpc error: code = NotFound desc = could not find container \"c436cf179edf23782012024b058c39f01184ff419de08fd9c33ba0cf17321ed0\": container with ID starting with c436cf179edf23782012024b058c39f01184ff419de08fd9c33ba0cf17321ed0 not found: ID does not exist" Oct 30 21:51:23.154957 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.154955 5918 scope.go:122] "RemoveContainer" containerID="7f6a9182c1c60364249dd1d140565cfb4a5f1960e32a8bf97d90413b5212beb1" Oct 30 21:51:23.155150 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.155135 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f6a9182c1c60364249dd1d140565cfb4a5f1960e32a8bf97d90413b5212beb1"} err="failed to get container status \"7f6a9182c1c60364249dd1d140565cfb4a5f1960e32a8bf97d90413b5212beb1\": rpc error: code = NotFound desc = could not find container \"7f6a9182c1c60364249dd1d140565cfb4a5f1960e32a8bf97d90413b5212beb1\": container with ID starting with 7f6a9182c1c60364249dd1d140565cfb4a5f1960e32a8bf97d90413b5212beb1 not found: ID does not exist" Oct 30 21:51:23.155150 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.155149 5918 scope.go:122] "RemoveContainer" containerID="03bf42adb14c6eafce0516556870e7dd844847b9277150c44db7514cc4eb96cd" Oct 30 21:51:23.155362 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.155341 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03bf42adb14c6eafce0516556870e7dd844847b9277150c44db7514cc4eb96cd"} err="failed to get container status \"03bf42adb14c6eafce0516556870e7dd844847b9277150c44db7514cc4eb96cd\": rpc error: code = NotFound desc = could not find container \"03bf42adb14c6eafce0516556870e7dd844847b9277150c44db7514cc4eb96cd\": container with ID starting with 03bf42adb14c6eafce0516556870e7dd844847b9277150c44db7514cc4eb96cd not found: ID does not exist" Oct 30 21:51:23.155445 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.155362 5918 scope.go:122] "RemoveContainer" containerID="6ecd79a7c6d356f8e64253560432de55058bb4c2da2408ddf9e73b11af211256" Oct 30 21:51:23.155612 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.155596 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ecd79a7c6d356f8e64253560432de55058bb4c2da2408ddf9e73b11af211256"} err="failed to get container status \"6ecd79a7c6d356f8e64253560432de55058bb4c2da2408ddf9e73b11af211256\": rpc error: code = NotFound desc = could not find container \"6ecd79a7c6d356f8e64253560432de55058bb4c2da2408ddf9e73b11af211256\": container with ID starting with 6ecd79a7c6d356f8e64253560432de55058bb4c2da2408ddf9e73b11af211256 not found: ID does not exist" Oct 30 21:51:23.155656 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.155612 5918 scope.go:122] "RemoveContainer" containerID="6e673397a1f3c2825a68224eb4b922c2f0a5f03c6051241bba2c74b99abb1f6d" Oct 30 21:51:23.155822 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.155805 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e673397a1f3c2825a68224eb4b922c2f0a5f03c6051241bba2c74b99abb1f6d"} err="failed to get container status \"6e673397a1f3c2825a68224eb4b922c2f0a5f03c6051241bba2c74b99abb1f6d\": rpc error: code = NotFound desc = could not find container \"6e673397a1f3c2825a68224eb4b922c2f0a5f03c6051241bba2c74b99abb1f6d\": container with ID starting with 6e673397a1f3c2825a68224eb4b922c2f0a5f03c6051241bba2c74b99abb1f6d not found: ID does not exist" Oct 30 21:51:23.155866 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.155823 5918 scope.go:122] "RemoveContainer" containerID="b1ddaa6943a6a9fa826e141df1d6d73be3a699d6cdbd5c118deb80cd2f007ee3" Oct 30 21:51:23.156037 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.156022 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1ddaa6943a6a9fa826e141df1d6d73be3a699d6cdbd5c118deb80cd2f007ee3"} err="failed to get container status \"b1ddaa6943a6a9fa826e141df1d6d73be3a699d6cdbd5c118deb80cd2f007ee3\": rpc error: code = NotFound desc = could not find container \"b1ddaa6943a6a9fa826e141df1d6d73be3a699d6cdbd5c118deb80cd2f007ee3\": container with ID starting with b1ddaa6943a6a9fa826e141df1d6d73be3a699d6cdbd5c118deb80cd2f007ee3 not found: ID does not exist" Oct 30 21:51:23.156079 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.156038 5918 scope.go:122] "RemoveContainer" containerID="4d7dc66cbc297b493c2282a3d67bdb66bdeabbe7484d47c4b8d8c6c5ca9d70f1" Oct 30 21:51:23.156268 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.156251 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d7dc66cbc297b493c2282a3d67bdb66bdeabbe7484d47c4b8d8c6c5ca9d70f1"} err="failed to get container status \"4d7dc66cbc297b493c2282a3d67bdb66bdeabbe7484d47c4b8d8c6c5ca9d70f1\": rpc error: code = NotFound desc = could not find container \"4d7dc66cbc297b493c2282a3d67bdb66bdeabbe7484d47c4b8d8c6c5ca9d70f1\": container with ID starting with 4d7dc66cbc297b493c2282a3d67bdb66bdeabbe7484d47c4b8d8c6c5ca9d70f1 not found: ID does not exist" Oct 30 21:51:23.156320 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.156269 5918 scope.go:122] "RemoveContainer" containerID="c436cf179edf23782012024b058c39f01184ff419de08fd9c33ba0cf17321ed0" Oct 30 21:51:23.156460 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.156445 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c436cf179edf23782012024b058c39f01184ff419de08fd9c33ba0cf17321ed0"} err="failed to get container status \"c436cf179edf23782012024b058c39f01184ff419de08fd9c33ba0cf17321ed0\": rpc error: code = NotFound desc = could not find container \"c436cf179edf23782012024b058c39f01184ff419de08fd9c33ba0cf17321ed0\": container with ID starting with c436cf179edf23782012024b058c39f01184ff419de08fd9c33ba0cf17321ed0 not found: ID does not exist" Oct 30 21:51:23.156499 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.156460 5918 scope.go:122] "RemoveContainer" containerID="7f6a9182c1c60364249dd1d140565cfb4a5f1960e32a8bf97d90413b5212beb1" Oct 30 21:51:23.156687 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.156667 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f6a9182c1c60364249dd1d140565cfb4a5f1960e32a8bf97d90413b5212beb1"} err="failed to get container status \"7f6a9182c1c60364249dd1d140565cfb4a5f1960e32a8bf97d90413b5212beb1\": rpc error: code = NotFound desc = could not find container \"7f6a9182c1c60364249dd1d140565cfb4a5f1960e32a8bf97d90413b5212beb1\": container with ID starting with 7f6a9182c1c60364249dd1d140565cfb4a5f1960e32a8bf97d90413b5212beb1 not found: ID does not exist" Oct 30 21:51:23.156729 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.156689 5918 scope.go:122] "RemoveContainer" containerID="03bf42adb14c6eafce0516556870e7dd844847b9277150c44db7514cc4eb96cd" Oct 30 21:51:23.156884 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.156865 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03bf42adb14c6eafce0516556870e7dd844847b9277150c44db7514cc4eb96cd"} err="failed to get container status \"03bf42adb14c6eafce0516556870e7dd844847b9277150c44db7514cc4eb96cd\": rpc error: code = NotFound desc = could not find container \"03bf42adb14c6eafce0516556870e7dd844847b9277150c44db7514cc4eb96cd\": container with ID starting with 03bf42adb14c6eafce0516556870e7dd844847b9277150c44db7514cc4eb96cd not found: ID does not exist" Oct 30 21:51:23.156925 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.156884 5918 scope.go:122] "RemoveContainer" containerID="6ecd79a7c6d356f8e64253560432de55058bb4c2da2408ddf9e73b11af211256" Oct 30 21:51:23.157097 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.157080 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ecd79a7c6d356f8e64253560432de55058bb4c2da2408ddf9e73b11af211256"} err="failed to get container status \"6ecd79a7c6d356f8e64253560432de55058bb4c2da2408ddf9e73b11af211256\": rpc error: code = NotFound desc = could not find container \"6ecd79a7c6d356f8e64253560432de55058bb4c2da2408ddf9e73b11af211256\": container with ID starting with 6ecd79a7c6d356f8e64253560432de55058bb4c2da2408ddf9e73b11af211256 not found: ID does not exist" Oct 30 21:51:23.157137 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.157099 5918 scope.go:122] "RemoveContainer" containerID="6e673397a1f3c2825a68224eb4b922c2f0a5f03c6051241bba2c74b99abb1f6d" Oct 30 21:51:23.157315 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.157296 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e673397a1f3c2825a68224eb4b922c2f0a5f03c6051241bba2c74b99abb1f6d"} err="failed to get container status \"6e673397a1f3c2825a68224eb4b922c2f0a5f03c6051241bba2c74b99abb1f6d\": rpc error: code = NotFound desc = could not find container \"6e673397a1f3c2825a68224eb4b922c2f0a5f03c6051241bba2c74b99abb1f6d\": container with ID starting with 6e673397a1f3c2825a68224eb4b922c2f0a5f03c6051241bba2c74b99abb1f6d not found: ID does not exist" Oct 30 21:51:23.157357 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.157315 5918 scope.go:122] "RemoveContainer" containerID="b1ddaa6943a6a9fa826e141df1d6d73be3a699d6cdbd5c118deb80cd2f007ee3" Oct 30 21:51:23.157531 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.157513 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1ddaa6943a6a9fa826e141df1d6d73be3a699d6cdbd5c118deb80cd2f007ee3"} err="failed to get container status \"b1ddaa6943a6a9fa826e141df1d6d73be3a699d6cdbd5c118deb80cd2f007ee3\": rpc error: code = NotFound desc = could not find container \"b1ddaa6943a6a9fa826e141df1d6d73be3a699d6cdbd5c118deb80cd2f007ee3\": container with ID starting with b1ddaa6943a6a9fa826e141df1d6d73be3a699d6cdbd5c118deb80cd2f007ee3 not found: ID does not exist" Oct 30 21:51:23.157596 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.157531 5918 scope.go:122] "RemoveContainer" containerID="4d7dc66cbc297b493c2282a3d67bdb66bdeabbe7484d47c4b8d8c6c5ca9d70f1" Oct 30 21:51:23.157754 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.157737 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d7dc66cbc297b493c2282a3d67bdb66bdeabbe7484d47c4b8d8c6c5ca9d70f1"} err="failed to get container status \"4d7dc66cbc297b493c2282a3d67bdb66bdeabbe7484d47c4b8d8c6c5ca9d70f1\": rpc error: code = NotFound desc = could not find container \"4d7dc66cbc297b493c2282a3d67bdb66bdeabbe7484d47c4b8d8c6c5ca9d70f1\": container with ID starting with 4d7dc66cbc297b493c2282a3d67bdb66bdeabbe7484d47c4b8d8c6c5ca9d70f1 not found: ID does not exist" Oct 30 21:51:23.157798 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.157754 5918 scope.go:122] "RemoveContainer" containerID="c436cf179edf23782012024b058c39f01184ff419de08fd9c33ba0cf17321ed0" Oct 30 21:51:23.157962 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.157948 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c436cf179edf23782012024b058c39f01184ff419de08fd9c33ba0cf17321ed0"} err="failed to get container status \"c436cf179edf23782012024b058c39f01184ff419de08fd9c33ba0cf17321ed0\": rpc error: code = NotFound desc = could not find container \"c436cf179edf23782012024b058c39f01184ff419de08fd9c33ba0cf17321ed0\": container with ID starting with c436cf179edf23782012024b058c39f01184ff419de08fd9c33ba0cf17321ed0 not found: ID does not exist" Oct 30 21:51:23.157962 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.157962 5918 scope.go:122] "RemoveContainer" containerID="7f6a9182c1c60364249dd1d140565cfb4a5f1960e32a8bf97d90413b5212beb1" Oct 30 21:51:23.158165 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.158150 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f6a9182c1c60364249dd1d140565cfb4a5f1960e32a8bf97d90413b5212beb1"} err="failed to get container status \"7f6a9182c1c60364249dd1d140565cfb4a5f1960e32a8bf97d90413b5212beb1\": rpc error: code = NotFound desc = could not find container \"7f6a9182c1c60364249dd1d140565cfb4a5f1960e32a8bf97d90413b5212beb1\": container with ID starting with 7f6a9182c1c60364249dd1d140565cfb4a5f1960e32a8bf97d90413b5212beb1 not found: ID does not exist" Oct 30 21:51:23.158209 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.158166 5918 scope.go:122] "RemoveContainer" containerID="03bf42adb14c6eafce0516556870e7dd844847b9277150c44db7514cc4eb96cd" Oct 30 21:51:23.158377 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.158362 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03bf42adb14c6eafce0516556870e7dd844847b9277150c44db7514cc4eb96cd"} err="failed to get container status \"03bf42adb14c6eafce0516556870e7dd844847b9277150c44db7514cc4eb96cd\": rpc error: code = NotFound desc = could not find container \"03bf42adb14c6eafce0516556870e7dd844847b9277150c44db7514cc4eb96cd\": container with ID starting with 03bf42adb14c6eafce0516556870e7dd844847b9277150c44db7514cc4eb96cd not found: ID does not exist" Oct 30 21:51:23.158423 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.158378 5918 scope.go:122] "RemoveContainer" containerID="6ecd79a7c6d356f8e64253560432de55058bb4c2da2408ddf9e73b11af211256" Oct 30 21:51:23.158599 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.158583 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ecd79a7c6d356f8e64253560432de55058bb4c2da2408ddf9e73b11af211256"} err="failed to get container status \"6ecd79a7c6d356f8e64253560432de55058bb4c2da2408ddf9e73b11af211256\": rpc error: code = NotFound desc = could not find container \"6ecd79a7c6d356f8e64253560432de55058bb4c2da2408ddf9e73b11af211256\": container with ID starting with 6ecd79a7c6d356f8e64253560432de55058bb4c2da2408ddf9e73b11af211256 not found: ID does not exist" Oct 30 21:51:23.265122 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.265094 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:51:23.315605 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.315580 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:51:23.315953 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.315935 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:51:23.316227 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.316200 5918 cpu_manager.go:414] "containerMap: removing container" podUID="736951e5-1596-4606-94d9-80d8b50d35c3" containerName="kube-rbac-proxy-web" Oct 30 21:51:23.316271 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.316229 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="736951e5-1596-4606-94d9-80d8b50d35c3" containerName="kube-rbac-proxy-web" Oct 30 21:51:23.316271 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.316243 5918 cpu_manager.go:414] "containerMap: removing container" podUID="736951e5-1596-4606-94d9-80d8b50d35c3" containerName="config-reloader" Oct 30 21:51:23.316271 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.316248 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="736951e5-1596-4606-94d9-80d8b50d35c3" containerName="config-reloader" Oct 30 21:51:23.316271 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.316258 5918 cpu_manager.go:414] "containerMap: removing container" podUID="736951e5-1596-4606-94d9-80d8b50d35c3" containerName="thanos-sidecar" Oct 30 21:51:23.316271 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.316262 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="736951e5-1596-4606-94d9-80d8b50d35c3" containerName="thanos-sidecar" Oct 30 21:51:23.316433 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.316276 5918 cpu_manager.go:414] "containerMap: removing container" podUID="736951e5-1596-4606-94d9-80d8b50d35c3" containerName="kube-rbac-proxy" Oct 30 21:51:23.316433 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.316280 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="736951e5-1596-4606-94d9-80d8b50d35c3" containerName="kube-rbac-proxy" Oct 30 21:51:23.316433 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.316285 5918 cpu_manager.go:414] "containerMap: removing container" podUID="736951e5-1596-4606-94d9-80d8b50d35c3" containerName="init-config-reloader" Oct 30 21:51:23.316433 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.316289 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="736951e5-1596-4606-94d9-80d8b50d35c3" containerName="init-config-reloader" Oct 30 21:51:23.316433 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.316308 5918 cpu_manager.go:414] "containerMap: removing container" podUID="736951e5-1596-4606-94d9-80d8b50d35c3" containerName="prometheus" Oct 30 21:51:23.316433 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.316312 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="736951e5-1596-4606-94d9-80d8b50d35c3" containerName="prometheus" Oct 30 21:51:23.316433 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.316318 5918 cpu_manager.go:414] "containerMap: removing container" podUID="736951e5-1596-4606-94d9-80d8b50d35c3" containerName="kube-rbac-proxy-thanos" Oct 30 21:51:23.316433 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.316322 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="736951e5-1596-4606-94d9-80d8b50d35c3" containerName="kube-rbac-proxy-thanos" Oct 30 21:51:23.316433 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.316363 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="736951e5-1596-4606-94d9-80d8b50d35c3" containerName="kube-rbac-proxy" Oct 30 21:51:23.316433 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.316370 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="736951e5-1596-4606-94d9-80d8b50d35c3" containerName="init-config-reloader" Oct 30 21:51:23.316433 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.316379 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="736951e5-1596-4606-94d9-80d8b50d35c3" containerName="prometheus" Oct 30 21:51:23.316433 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.316384 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="736951e5-1596-4606-94d9-80d8b50d35c3" containerName="kube-rbac-proxy-thanos" Oct 30 21:51:23.316433 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.316391 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="736951e5-1596-4606-94d9-80d8b50d35c3" containerName="kube-rbac-proxy-web" Oct 30 21:51:23.316433 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.316398 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="736951e5-1596-4606-94d9-80d8b50d35c3" containerName="config-reloader" Oct 30 21:51:23.316433 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.316403 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="736951e5-1596-4606-94d9-80d8b50d35c3" containerName="thanos-sidecar" Oct 30 21:51:23.319673 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.319658 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.374432 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.374290 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 21:51:23.374432 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.374290 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 21:51:23.374432 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.374335 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9jiopb23p179d\"" Oct 30 21:51:23.374432 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.374348 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 21:51:23.374432 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.374343 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 21:51:23.374432 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.374366 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 21:51:23.374432 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.374349 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 21:51:23.374432 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.374385 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 21:51:23.374432 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.374397 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 21:51:23.374432 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.374376 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 21:51:23.374432 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.374362 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 21:51:23.377341 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.377328 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 21:51:23.377470 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.377458 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 21:51:23.377565 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.377553 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-t5jsh\"" Oct 30 21:51:23.408451 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.408424 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/35140ca6-78c4-49d2-8755-ba7f22efb045-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.408451 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.408452 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/35140ca6-78c4-49d2-8755-ba7f22efb045-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.408600 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.408469 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/35140ca6-78c4-49d2-8755-ba7f22efb045-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.408600 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.408491 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35140ca6-78c4-49d2-8755-ba7f22efb045-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.408600 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.408570 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/35140ca6-78c4-49d2-8755-ba7f22efb045-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.408699 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.408601 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/35140ca6-78c4-49d2-8755-ba7f22efb045-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.408699 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.408647 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/35140ca6-78c4-49d2-8755-ba7f22efb045-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.408699 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.408693 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35140ca6-78c4-49d2-8755-ba7f22efb045-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.408794 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.408716 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35140ca6-78c4-49d2-8755-ba7f22efb045-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.408794 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.408743 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35140ca6-78c4-49d2-8755-ba7f22efb045-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.408794 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.408757 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/35140ca6-78c4-49d2-8755-ba7f22efb045-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.408901 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.408840 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/35140ca6-78c4-49d2-8755-ba7f22efb045-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.408901 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.408870 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/35140ca6-78c4-49d2-8755-ba7f22efb045-config\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.408901 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.408888 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35140ca6-78c4-49d2-8755-ba7f22efb045-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.408994 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.408903 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/35140ca6-78c4-49d2-8755-ba7f22efb045-web-config\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.408994 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.408958 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/35140ca6-78c4-49d2-8755-ba7f22efb045-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.409062 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.408992 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/35140ca6-78c4-49d2-8755-ba7f22efb045-config-out\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.409062 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.409007 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/35140ca6-78c4-49d2-8755-ba7f22efb045-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.409134 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.409070 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35140ca6-78c4-49d2-8755-ba7f22efb045-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.409134 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.409093 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhpj2\" (UniqueName: \"kubernetes.io/projected/35140ca6-78c4-49d2-8755-ba7f22efb045-kube-api-access-hhpj2\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.428421 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.428402 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 21:51:23.428667 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.428652 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 21:51:23.439344 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.439320 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:51:23.510424 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.510390 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35140ca6-78c4-49d2-8755-ba7f22efb045-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.510424 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.510424 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/35140ca6-78c4-49d2-8755-ba7f22efb045-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.510660 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.510446 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/35140ca6-78c4-49d2-8755-ba7f22efb045-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.510660 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.510466 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/35140ca6-78c4-49d2-8755-ba7f22efb045-config\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.510660 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.510489 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35140ca6-78c4-49d2-8755-ba7f22efb045-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.510660 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.510511 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/35140ca6-78c4-49d2-8755-ba7f22efb045-web-config\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.510660 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.510538 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/35140ca6-78c4-49d2-8755-ba7f22efb045-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.510660 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.510564 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/35140ca6-78c4-49d2-8755-ba7f22efb045-config-out\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.510660 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.510586 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/35140ca6-78c4-49d2-8755-ba7f22efb045-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.510660 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.510616 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35140ca6-78c4-49d2-8755-ba7f22efb045-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.510660 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.510640 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhpj2\" (UniqueName: \"kubernetes.io/projected/35140ca6-78c4-49d2-8755-ba7f22efb045-kube-api-access-hhpj2\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.511057 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.510681 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/35140ca6-78c4-49d2-8755-ba7f22efb045-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.511057 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.510704 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/35140ca6-78c4-49d2-8755-ba7f22efb045-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.511057 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.510724 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/35140ca6-78c4-49d2-8755-ba7f22efb045-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.511057 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.510754 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35140ca6-78c4-49d2-8755-ba7f22efb045-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.511057 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.510777 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/35140ca6-78c4-49d2-8755-ba7f22efb045-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.511057 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.510805 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/35140ca6-78c4-49d2-8755-ba7f22efb045-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.511057 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.510834 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/35140ca6-78c4-49d2-8755-ba7f22efb045-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.511057 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.510875 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35140ca6-78c4-49d2-8755-ba7f22efb045-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.511057 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.510915 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35140ca6-78c4-49d2-8755-ba7f22efb045-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.513534 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.511136 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/35140ca6-78c4-49d2-8755-ba7f22efb045-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.513534 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.511548 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/35140ca6-78c4-49d2-8755-ba7f22efb045-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.513534 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.511635 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35140ca6-78c4-49d2-8755-ba7f22efb045-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.513534 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.511893 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/35140ca6-78c4-49d2-8755-ba7f22efb045-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.513534 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.512394 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35140ca6-78c4-49d2-8755-ba7f22efb045-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.513534 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.512914 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35140ca6-78c4-49d2-8755-ba7f22efb045-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.513534 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.513133 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35140ca6-78c4-49d2-8755-ba7f22efb045-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.514474 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.514428 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/35140ca6-78c4-49d2-8755-ba7f22efb045-config-out\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.514898 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.514859 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35140ca6-78c4-49d2-8755-ba7f22efb045-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.515902 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.515352 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/35140ca6-78c4-49d2-8755-ba7f22efb045-web-config\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.515902 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.515362 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/35140ca6-78c4-49d2-8755-ba7f22efb045-config\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.515902 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.515691 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/35140ca6-78c4-49d2-8755-ba7f22efb045-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.515902 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.515771 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/35140ca6-78c4-49d2-8755-ba7f22efb045-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.515902 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.515896 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/35140ca6-78c4-49d2-8755-ba7f22efb045-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.516363 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.516346 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/35140ca6-78c4-49d2-8755-ba7f22efb045-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.516639 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.516624 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/35140ca6-78c4-49d2-8755-ba7f22efb045-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.517164 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.517144 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/35140ca6-78c4-49d2-8755-ba7f22efb045-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.517257 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.517149 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35140ca6-78c4-49d2-8755-ba7f22efb045-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.517322 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.517291 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/35140ca6-78c4-49d2-8755-ba7f22efb045-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.574995 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.574965 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhpj2\" (UniqueName: \"kubernetes.io/projected/35140ca6-78c4-49d2-8755-ba7f22efb045-kube-api-access-hhpj2\") pod \"prometheus-k8s-0\" (UID: \"35140ca6-78c4-49d2-8755-ba7f22efb045\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.634772 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.634710 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:23.848534 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:23.848503 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="736951e5-1596-4606-94d9-80d8b50d35c3" path="/var/lib/kubelet/pods/736951e5-1596-4606-94d9-80d8b50d35c3/volumes" Oct 30 21:51:24.067797 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:24.067762 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:51:24.070514 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:51:24.070488 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35140ca6_78c4_49d2_8755_ba7f22efb045.slice/crio-8b8971600ec5ccbd9e61813449145fd118ee05554fb677964ecbd501f4636a7a WatchSource:0}: Error finding container 8b8971600ec5ccbd9e61813449145fd118ee05554fb677964ecbd501f4636a7a: Status 404 returned error can't find the container with id 8b8971600ec5ccbd9e61813449145fd118ee05554fb677964ecbd501f4636a7a Oct 30 21:51:24.097430 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:24.097405 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"35140ca6-78c4-49d2-8755-ba7f22efb045","Type":"ContainerStarted","Data":"8b8971600ec5ccbd9e61813449145fd118ee05554fb677964ecbd501f4636a7a"} Oct 30 21:51:25.102886 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:25.102844 5918 generic.go:356] "Generic (PLEG): container finished" podID="35140ca6-78c4-49d2-8755-ba7f22efb045" containerID="c9329b8d4acee1a484bfdb05dc7ee84ff4bafa9ee44c15a5a5d8a235338df78e" exitCode=0 Oct 30 21:51:25.103363 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:25.102899 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"35140ca6-78c4-49d2-8755-ba7f22efb045","Type":"ContainerDied","Data":"c9329b8d4acee1a484bfdb05dc7ee84ff4bafa9ee44c15a5a5d8a235338df78e"} Oct 30 21:51:26.110789 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:26.110756 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"35140ca6-78c4-49d2-8755-ba7f22efb045","Type":"ContainerStarted","Data":"024bc579d0a04769b08c1c143506bbdf75cbd387ae86cc68cff7ce69a7cf5386"} Oct 30 21:51:26.111196 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:26.110822 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"35140ca6-78c4-49d2-8755-ba7f22efb045","Type":"ContainerStarted","Data":"5843464ee766388171d219488561a75cc40771df1432dd48934d2ef735dd8089"} Oct 30 21:51:26.111196 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:26.110853 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"35140ca6-78c4-49d2-8755-ba7f22efb045","Type":"ContainerStarted","Data":"824aede121326fa51d7cce516c54f75e2def33447b5072a60a79f39d0aa1c73c"} Oct 30 21:51:26.111196 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:26.110880 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"35140ca6-78c4-49d2-8755-ba7f22efb045","Type":"ContainerStarted","Data":"3de76fe200e186cdde4f194a1746a392074ce3ca4a056ee1f994d79558f559e4"} Oct 30 21:51:26.111196 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:26.110902 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"35140ca6-78c4-49d2-8755-ba7f22efb045","Type":"ContainerStarted","Data":"e8a57a38fe04980c8202ebaa2268e31010bcdc16cbbe4058044282e79c45e298"} Oct 30 21:51:26.111196 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:26.110932 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"35140ca6-78c4-49d2-8755-ba7f22efb045","Type":"ContainerStarted","Data":"5909d3db650c6ecd1d5417d85696bfa12219ff831c9409d5fd7b2c0c5c4aec19"} Oct 30 21:51:26.248431 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:26.248377 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.229757259 podStartE2EDuration="3.248363311s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.606052ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:26.247647895 +0000 UTC m=+46.518345971" watchObservedRunningTime="2026-10-30 21:51:26.248363311 +0000 UTC m=+46.519061394" Oct 30 21:51:28.635033 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:51:28.635001 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:51:51.274167 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:51:51.274130 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 21:52:17.842261 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:52:17.842191 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:52:17.858681 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:52:17.858659 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:52:18.315017 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:52:18.314991 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:52:21.273546 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:52:21.273472 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 21:52:51.273563 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:52:51.273524 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 21:52:58.300419 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:52:58.300388 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71msl55"] Oct 30 21:52:58.303900 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:52:58.303880 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71msl55" Oct 30 21:52:58.409616 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:52:58.409592 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-r29x9\"" Oct 30 21:52:58.409798 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:52:58.409777 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 21:52:58.409798 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:52:58.409782 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 21:52:58.410588 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:52:58.410567 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71msl55"] Oct 30 21:52:58.425174 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:52:58.425152 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/41d8ed2e-6384-44a0-978f-5c463b551153-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71msl55\" (UID: \"41d8ed2e-6384-44a0-978f-5c463b551153\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71msl55" Oct 30 21:52:58.425236 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:52:58.425205 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb2mz\" (UniqueName: \"kubernetes.io/projected/41d8ed2e-6384-44a0-978f-5c463b551153-kube-api-access-pb2mz\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71msl55\" (UID: \"41d8ed2e-6384-44a0-978f-5c463b551153\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71msl55" Oct 30 21:52:58.425316 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:52:58.425295 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/41d8ed2e-6384-44a0-978f-5c463b551153-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71msl55\" (UID: \"41d8ed2e-6384-44a0-978f-5c463b551153\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71msl55" Oct 30 21:52:58.525923 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:52:58.525899 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/41d8ed2e-6384-44a0-978f-5c463b551153-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71msl55\" (UID: \"41d8ed2e-6384-44a0-978f-5c463b551153\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71msl55" Oct 30 21:52:58.526076 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:52:58.525951 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb2mz\" (UniqueName: \"kubernetes.io/projected/41d8ed2e-6384-44a0-978f-5c463b551153-kube-api-access-pb2mz\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71msl55\" (UID: \"41d8ed2e-6384-44a0-978f-5c463b551153\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71msl55" Oct 30 21:52:58.526076 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:52:58.525989 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/41d8ed2e-6384-44a0-978f-5c463b551153-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71msl55\" (UID: \"41d8ed2e-6384-44a0-978f-5c463b551153\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71msl55" Oct 30 21:52:58.526341 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:52:58.526323 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/41d8ed2e-6384-44a0-978f-5c463b551153-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71msl55\" (UID: \"41d8ed2e-6384-44a0-978f-5c463b551153\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71msl55" Oct 30 21:52:58.526382 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:52:58.526357 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/41d8ed2e-6384-44a0-978f-5c463b551153-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71msl55\" (UID: \"41d8ed2e-6384-44a0-978f-5c463b551153\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71msl55" Oct 30 21:52:58.600289 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:52:58.600206 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb2mz\" (UniqueName: \"kubernetes.io/projected/41d8ed2e-6384-44a0-978f-5c463b551153-kube-api-access-pb2mz\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71msl55\" (UID: \"41d8ed2e-6384-44a0-978f-5c463b551153\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71msl55" Oct 30 21:52:58.619906 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:52:58.619881 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71msl55" Oct 30 21:52:58.994918 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:52:58.994889 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71msl55"] Oct 30 21:52:58.998713 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:52:58.998684 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41d8ed2e_6384_44a0_978f_5c463b551153.slice/crio-66328a95d11df149930084d3453381ac7b84c45244eb76a1493b18d419401433 WatchSource:0}: Error finding container 66328a95d11df149930084d3453381ac7b84c45244eb76a1493b18d419401433: Status 404 returned error can't find the container with id 66328a95d11df149930084d3453381ac7b84c45244eb76a1493b18d419401433 Oct 30 21:52:59.001185 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:52:59.001165 5918 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:52:59.450776 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:52:59.450745 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71msl55" event={"ID":"41d8ed2e-6384-44a0-978f-5c463b551153","Type":"ContainerStarted","Data":"66328a95d11df149930084d3453381ac7b84c45244eb76a1493b18d419401433"} Oct 30 21:53:01.564054 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.564019 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2r27"] Oct 30 21:53:01.568880 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.568858 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2r27" Oct 30 21:53:01.600764 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.600731 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7zbx"] Oct 30 21:53:01.604422 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.604400 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7zbx" Oct 30 21:53:01.650463 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.650429 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-qz8st\"" Oct 30 21:53:01.651338 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.651301 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2r27"] Oct 30 21:53:01.653820 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.653796 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3d268d9-feff-4875-bf43-d7a44f2b81f9-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2r27\" (UID: \"b3d268d9-feff-4875-bf43-d7a44f2b81f9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2r27" Oct 30 21:53:01.653943 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.653848 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv2v6\" (UniqueName: \"kubernetes.io/projected/b3d268d9-feff-4875-bf43-d7a44f2b81f9-kube-api-access-gv2v6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2r27\" (UID: \"b3d268d9-feff-4875-bf43-d7a44f2b81f9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2r27" Oct 30 21:53:01.653943 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.653884 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3d268d9-feff-4875-bf43-d7a44f2b81f9-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2r27\" (UID: \"b3d268d9-feff-4875-bf43-d7a44f2b81f9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2r27" Oct 30 21:53:01.673914 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.673882 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2zwv9"] Oct 30 21:53:01.677598 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.677569 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7zbx"] Oct 30 21:53:01.677750 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.677703 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2zwv9" Oct 30 21:53:01.702976 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.702950 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2zwv9"] Oct 30 21:53:01.724121 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.724086 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-t7vbt\"" Oct 30 21:53:01.731572 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.731540 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-xqptj\"" Oct 30 21:53:01.754494 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.754456 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01521af4-ecff-4b27-8a51-6d7ec3bb3efb-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7zbx\" (UID: \"01521af4-ecff-4b27-8a51-6d7ec3bb3efb\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7zbx" Oct 30 21:53:01.754739 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.754509 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmrph\" (UniqueName: \"kubernetes.io/projected/01521af4-ecff-4b27-8a51-6d7ec3bb3efb-kube-api-access-cmrph\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7zbx\" (UID: \"01521af4-ecff-4b27-8a51-6d7ec3bb3efb\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7zbx" Oct 30 21:53:01.754739 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.754664 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv8vv\" (UniqueName: \"kubernetes.io/projected/4ee1ae12-97e4-49f4-8f95-f92746f709b6-kube-api-access-hv8vv\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2zwv9\" (UID: \"4ee1ae12-97e4-49f4-8f95-f92746f709b6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2zwv9" Oct 30 21:53:01.754739 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.754692 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3d268d9-feff-4875-bf43-d7a44f2b81f9-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2r27\" (UID: \"b3d268d9-feff-4875-bf43-d7a44f2b81f9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2r27" Oct 30 21:53:01.754992 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.754745 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01521af4-ecff-4b27-8a51-6d7ec3bb3efb-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7zbx\" (UID: \"01521af4-ecff-4b27-8a51-6d7ec3bb3efb\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7zbx" Oct 30 21:53:01.754992 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.754772 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv2v6\" (UniqueName: \"kubernetes.io/projected/b3d268d9-feff-4875-bf43-d7a44f2b81f9-kube-api-access-gv2v6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2r27\" (UID: \"b3d268d9-feff-4875-bf43-d7a44f2b81f9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2r27" Oct 30 21:53:01.754992 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.754797 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3d268d9-feff-4875-bf43-d7a44f2b81f9-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2r27\" (UID: \"b3d268d9-feff-4875-bf43-d7a44f2b81f9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2r27" Oct 30 21:53:01.754992 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.754816 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ee1ae12-97e4-49f4-8f95-f92746f709b6-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2zwv9\" (UID: \"4ee1ae12-97e4-49f4-8f95-f92746f709b6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2zwv9" Oct 30 21:53:01.754992 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.754891 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ee1ae12-97e4-49f4-8f95-f92746f709b6-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2zwv9\" (UID: \"4ee1ae12-97e4-49f4-8f95-f92746f709b6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2zwv9" Oct 30 21:53:01.755199 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.755039 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3d268d9-feff-4875-bf43-d7a44f2b81f9-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2r27\" (UID: \"b3d268d9-feff-4875-bf43-d7a44f2b81f9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2r27" Oct 30 21:53:01.755199 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.755115 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3d268d9-feff-4875-bf43-d7a44f2b81f9-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2r27\" (UID: \"b3d268d9-feff-4875-bf43-d7a44f2b81f9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2r27" Oct 30 21:53:01.818881 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.818803 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv2v6\" (UniqueName: \"kubernetes.io/projected/b3d268d9-feff-4875-bf43-d7a44f2b81f9-kube-api-access-gv2v6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2r27\" (UID: \"b3d268d9-feff-4875-bf43-d7a44f2b81f9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2r27" Oct 30 21:53:01.856424 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.856393 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ee1ae12-97e4-49f4-8f95-f92746f709b6-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2zwv9\" (UID: \"4ee1ae12-97e4-49f4-8f95-f92746f709b6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2zwv9" Oct 30 21:53:01.856622 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.856450 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ee1ae12-97e4-49f4-8f95-f92746f709b6-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2zwv9\" (UID: \"4ee1ae12-97e4-49f4-8f95-f92746f709b6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2zwv9" Oct 30 21:53:01.856622 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.856493 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01521af4-ecff-4b27-8a51-6d7ec3bb3efb-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7zbx\" (UID: \"01521af4-ecff-4b27-8a51-6d7ec3bb3efb\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7zbx" Oct 30 21:53:01.856622 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.856530 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmrph\" (UniqueName: \"kubernetes.io/projected/01521af4-ecff-4b27-8a51-6d7ec3bb3efb-kube-api-access-cmrph\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7zbx\" (UID: \"01521af4-ecff-4b27-8a51-6d7ec3bb3efb\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7zbx" Oct 30 21:53:01.856622 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.856584 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv8vv\" (UniqueName: \"kubernetes.io/projected/4ee1ae12-97e4-49f4-8f95-f92746f709b6-kube-api-access-hv8vv\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2zwv9\" (UID: \"4ee1ae12-97e4-49f4-8f95-f92746f709b6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2zwv9" Oct 30 21:53:01.856892 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.856627 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01521af4-ecff-4b27-8a51-6d7ec3bb3efb-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7zbx\" (UID: \"01521af4-ecff-4b27-8a51-6d7ec3bb3efb\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7zbx" Oct 30 21:53:01.856892 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.856798 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ee1ae12-97e4-49f4-8f95-f92746f709b6-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2zwv9\" (UID: \"4ee1ae12-97e4-49f4-8f95-f92746f709b6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2zwv9" Oct 30 21:53:01.856892 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.856823 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ee1ae12-97e4-49f4-8f95-f92746f709b6-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2zwv9\" (UID: \"4ee1ae12-97e4-49f4-8f95-f92746f709b6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2zwv9" Oct 30 21:53:01.857104 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.856994 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01521af4-ecff-4b27-8a51-6d7ec3bb3efb-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7zbx\" (UID: \"01521af4-ecff-4b27-8a51-6d7ec3bb3efb\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7zbx" Oct 30 21:53:01.857140 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.857116 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01521af4-ecff-4b27-8a51-6d7ec3bb3efb-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7zbx\" (UID: \"01521af4-ecff-4b27-8a51-6d7ec3bb3efb\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7zbx" Oct 30 21:53:01.888196 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.888163 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2r27" Oct 30 21:53:01.921614 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.921572 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmrph\" (UniqueName: \"kubernetes.io/projected/01521af4-ecff-4b27-8a51-6d7ec3bb3efb-kube-api-access-cmrph\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7zbx\" (UID: \"01521af4-ecff-4b27-8a51-6d7ec3bb3efb\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7zbx" Oct 30 21:53:01.922383 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.921908 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7zbx" Oct 30 21:53:01.922383 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.922346 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv8vv\" (UniqueName: \"kubernetes.io/projected/4ee1ae12-97e4-49f4-8f95-f92746f709b6-kube-api-access-hv8vv\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2zwv9\" (UID: \"4ee1ae12-97e4-49f4-8f95-f92746f709b6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2zwv9" Oct 30 21:53:01.995349 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:01.995320 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2zwv9" Oct 30 21:53:02.317260 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:02.317228 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2r27"] Oct 30 21:53:02.320028 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:02.320004 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7zbx"] Oct 30 21:53:02.322247 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:53:02.322198 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3d268d9_feff_4875_bf43_d7a44f2b81f9.slice/crio-8af5e0f502c0965d90dc32aece14ec4993a2161bd3e6c667e0015863e1e0995a WatchSource:0}: Error finding container 8af5e0f502c0965d90dc32aece14ec4993a2161bd3e6c667e0015863e1e0995a: Status 404 returned error can't find the container with id 8af5e0f502c0965d90dc32aece14ec4993a2161bd3e6c667e0015863e1e0995a Oct 30 21:53:02.370072 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:53:02.370040 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01521af4_ecff_4b27_8a51_6d7ec3bb3efb.slice/crio-2eff324f439f694e514cc649f66c73ac3c7105ec1c7a82e5facdda0440bbf235 WatchSource:0}: Error finding container 2eff324f439f694e514cc649f66c73ac3c7105ec1c7a82e5facdda0440bbf235: Status 404 returned error can't find the container with id 2eff324f439f694e514cc649f66c73ac3c7105ec1c7a82e5facdda0440bbf235 Oct 30 21:53:02.433060 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:02.433031 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2zwv9"] Oct 30 21:53:02.436179 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:53:02.436137 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ee1ae12_97e4_49f4_8f95_f92746f709b6.slice/crio-7ffd5481a49c9253ebf77ba5d9e39fc269de32bc130e4500b1db199c1bf773c7 WatchSource:0}: Error finding container 7ffd5481a49c9253ebf77ba5d9e39fc269de32bc130e4500b1db199c1bf773c7: Status 404 returned error can't find the container with id 7ffd5481a49c9253ebf77ba5d9e39fc269de32bc130e4500b1db199c1bf773c7 Oct 30 21:53:02.467009 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:02.466963 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7zbx" event={"ID":"01521af4-ecff-4b27-8a51-6d7ec3bb3efb","Type":"ContainerStarted","Data":"2eff324f439f694e514cc649f66c73ac3c7105ec1c7a82e5facdda0440bbf235"} Oct 30 21:53:02.468385 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:02.468335 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2r27" event={"ID":"b3d268d9-feff-4875-bf43-d7a44f2b81f9","Type":"ContainerStarted","Data":"8af5e0f502c0965d90dc32aece14ec4993a2161bd3e6c667e0015863e1e0995a"} Oct 30 21:53:02.469680 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:02.469655 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2zwv9" event={"ID":"4ee1ae12-97e4-49f4-8f95-f92746f709b6","Type":"ContainerStarted","Data":"7ffd5481a49c9253ebf77ba5d9e39fc269de32bc130e4500b1db199c1bf773c7"} Oct 30 21:53:06.495988 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:06.495944 5918 generic.go:356] "Generic (PLEG): container finished" podID="41d8ed2e-6384-44a0-978f-5c463b551153" containerID="98d6c7f404a50d0127b4bed25e2f53bea0f5a8f19eb6463536a7f38e95848818" exitCode=0 Oct 30 21:53:06.496422 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:06.496087 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71msl55" event={"ID":"41d8ed2e-6384-44a0-978f-5c463b551153","Type":"ContainerDied","Data":"98d6c7f404a50d0127b4bed25e2f53bea0f5a8f19eb6463536a7f38e95848818"} Oct 30 21:53:08.507938 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:08.507910 5918 generic.go:356] "Generic (PLEG): container finished" podID="01521af4-ecff-4b27-8a51-6d7ec3bb3efb" containerID="732f225cc650d5b192d790e4d581ab562e2dde5e9aabe269e1d0cd905825feee" exitCode=0 Oct 30 21:53:08.508360 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:08.507994 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7zbx" event={"ID":"01521af4-ecff-4b27-8a51-6d7ec3bb3efb","Type":"ContainerDied","Data":"732f225cc650d5b192d790e4d581ab562e2dde5e9aabe269e1d0cd905825feee"} Oct 30 21:53:08.511144 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:08.511113 5918 generic.go:356] "Generic (PLEG): container finished" podID="4ee1ae12-97e4-49f4-8f95-f92746f709b6" containerID="16be9c52f9326a952333fd391d71eb800986f58ace2516239dc5c6b5a6067149" exitCode=0 Oct 30 21:53:08.511273 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:08.511235 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2zwv9" event={"ID":"4ee1ae12-97e4-49f4-8f95-f92746f709b6","Type":"ContainerDied","Data":"16be9c52f9326a952333fd391d71eb800986f58ace2516239dc5c6b5a6067149"} Oct 30 21:53:08.513429 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:08.513408 5918 generic.go:356] "Generic (PLEG): container finished" podID="41d8ed2e-6384-44a0-978f-5c463b551153" containerID="d3085a068e1b4391c28101da004a35f2c1d5e01c41aec81f38061d7ff2f85d6d" exitCode=0 Oct 30 21:53:08.513519 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:08.513459 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71msl55" event={"ID":"41d8ed2e-6384-44a0-978f-5c463b551153","Type":"ContainerDied","Data":"d3085a068e1b4391c28101da004a35f2c1d5e01c41aec81f38061d7ff2f85d6d"} Oct 30 21:53:09.519436 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:09.519401 5918 generic.go:356] "Generic (PLEG): container finished" podID="b3d268d9-feff-4875-bf43-d7a44f2b81f9" containerID="1e5bb1bc7f36608c8cdbf51981e7d65fb96ce38408b6b895ea0f109940ad623f" exitCode=0 Oct 30 21:53:09.519890 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:09.519471 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2r27" event={"ID":"b3d268d9-feff-4875-bf43-d7a44f2b81f9","Type":"ContainerDied","Data":"1e5bb1bc7f36608c8cdbf51981e7d65fb96ce38408b6b895ea0f109940ad623f"} Oct 30 21:53:10.525560 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:10.525534 5918 generic.go:356] "Generic (PLEG): container finished" podID="4ee1ae12-97e4-49f4-8f95-f92746f709b6" containerID="a74d2dfe55c76a3bb996507a03da2987543ec0da703567d950aa10661e6df1f1" exitCode=0 Oct 30 21:53:10.525958 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:10.525627 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2zwv9" event={"ID":"4ee1ae12-97e4-49f4-8f95-f92746f709b6","Type":"ContainerDied","Data":"a74d2dfe55c76a3bb996507a03da2987543ec0da703567d950aa10661e6df1f1"} Oct 30 21:53:11.535247 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:11.535159 5918 generic.go:356] "Generic (PLEG): container finished" podID="01521af4-ecff-4b27-8a51-6d7ec3bb3efb" containerID="13316b5b73846d9d4364dc42c7e94514775dba6e7ccf38e496488d04382631c3" exitCode=0 Oct 30 21:53:11.535971 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:11.535255 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7zbx" event={"ID":"01521af4-ecff-4b27-8a51-6d7ec3bb3efb","Type":"ContainerDied","Data":"13316b5b73846d9d4364dc42c7e94514775dba6e7ccf38e496488d04382631c3"} Oct 30 21:53:11.539359 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:11.539334 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2r27" event={"ID":"b3d268d9-feff-4875-bf43-d7a44f2b81f9","Type":"ContainerStarted","Data":"a91a342eed0c48fa9a9f03bd4ba80d65d61c22fdf26b1260a0b7ba4ce86889f1"} Oct 30 21:53:12.547933 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:12.547892 5918 generic.go:356] "Generic (PLEG): container finished" podID="b3d268d9-feff-4875-bf43-d7a44f2b81f9" containerID="a91a342eed0c48fa9a9f03bd4ba80d65d61c22fdf26b1260a0b7ba4ce86889f1" exitCode=0 Oct 30 21:53:12.548559 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:12.547997 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2r27" event={"ID":"b3d268d9-feff-4875-bf43-d7a44f2b81f9","Type":"ContainerDied","Data":"a91a342eed0c48fa9a9f03bd4ba80d65d61c22fdf26b1260a0b7ba4ce86889f1"} Oct 30 21:53:17.581819 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:17.581785 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71msl55" event={"ID":"41d8ed2e-6384-44a0-978f-5c463b551153","Type":"ContainerStarted","Data":"7b9b046ad1796048e6d17290675bc2b76468ff3943c921cc9f8248da3858ae37"} Oct 30 21:53:17.705316 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:17.705269 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71msl55" podStartSLOduration=3.206830524 podStartE2EDuration="19.705254708s" totalImagesPullingTime="16.498424184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:17.704583239 +0000 UTC m=+157.975281327" watchObservedRunningTime="2026-10-30 21:53:17.705254708 +0000 UTC m=+157.975952792" Oct 30 21:53:18.588980 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:18.588939 5918 generic.go:356] "Generic (PLEG): container finished" podID="01521af4-ecff-4b27-8a51-6d7ec3bb3efb" containerID="dd6eb17025dc4f1ae755b621b040457106e24b9ee5d5b85f5821bc53210a6e82" exitCode=0 Oct 30 21:53:18.589465 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:18.589037 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7zbx" event={"ID":"01521af4-ecff-4b27-8a51-6d7ec3bb3efb","Type":"ContainerDied","Data":"dd6eb17025dc4f1ae755b621b040457106e24b9ee5d5b85f5821bc53210a6e82"} Oct 30 21:53:18.591531 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:18.591391 5918 generic.go:356] "Generic (PLEG): container finished" podID="b3d268d9-feff-4875-bf43-d7a44f2b81f9" containerID="5aaf417bf52755877ce39e115294c58829c84bec18e0c64938cb688c280a829c" exitCode=0 Oct 30 21:53:18.591531 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:18.591472 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2r27" event={"ID":"b3d268d9-feff-4875-bf43-d7a44f2b81f9","Type":"ContainerDied","Data":"5aaf417bf52755877ce39e115294c58829c84bec18e0c64938cb688c280a829c"} Oct 30 21:53:18.593769 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:18.593744 5918 generic.go:356] "Generic (PLEG): container finished" podID="41d8ed2e-6384-44a0-978f-5c463b551153" containerID="7b9b046ad1796048e6d17290675bc2b76468ff3943c921cc9f8248da3858ae37" exitCode=0 Oct 30 21:53:18.593901 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:18.593772 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71msl55" event={"ID":"41d8ed2e-6384-44a0-978f-5c463b551153","Type":"ContainerDied","Data":"7b9b046ad1796048e6d17290675bc2b76468ff3943c921cc9f8248da3858ae37"} Oct 30 21:53:19.600034 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:19.599948 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2zwv9" event={"ID":"4ee1ae12-97e4-49f4-8f95-f92746f709b6","Type":"ContainerStarted","Data":"456184ca817bdaa3baf53f0e598c460ed0a5fffb90029d136d09e24941b83c6b"} Oct 30 21:53:19.723600 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:19.723539 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2zwv9" podStartSLOduration=2.673472822 podStartE2EDuration="18.723518661s" totalImagesPullingTime="16.050045839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:19.722680072 +0000 UTC m=+159.993378157" watchObservedRunningTime="2026-10-30 21:53:19.723518661 +0000 UTC m=+159.994216748" Oct 30 21:53:19.757129 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:19.757106 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2r27" Oct 30 21:53:19.761528 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:19.761506 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71msl55" Oct 30 21:53:19.855162 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:19.855094 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7zbx" Oct 30 21:53:19.935009 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:19.934974 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d8ed2e-6384-44a0-978f-5c463b551153-util\" (UniqueName: \"kubernetes.io/empty-dir/41d8ed2e-6384-44a0-978f-5c463b551153-util\") pod \"41d8ed2e-6384-44a0-978f-5c463b551153\" (UID: \"41d8ed2e-6384-44a0-978f-5c463b551153\") " Oct 30 21:53:19.935187 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:19.935031 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3d268d9-feff-4875-bf43-d7a44f2b81f9-kube-api-access-gv2v6\" (UniqueName: \"kubernetes.io/projected/b3d268d9-feff-4875-bf43-d7a44f2b81f9-kube-api-access-gv2v6\") pod \"b3d268d9-feff-4875-bf43-d7a44f2b81f9\" (UID: \"b3d268d9-feff-4875-bf43-d7a44f2b81f9\") " Oct 30 21:53:19.935187 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:19.935147 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3d268d9-feff-4875-bf43-d7a44f2b81f9-util\" (UniqueName: \"kubernetes.io/empty-dir/b3d268d9-feff-4875-bf43-d7a44f2b81f9-util\") pod \"b3d268d9-feff-4875-bf43-d7a44f2b81f9\" (UID: \"b3d268d9-feff-4875-bf43-d7a44f2b81f9\") " Oct 30 21:53:19.935331 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:19.935203 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41d8ed2e-6384-44a0-978f-5c463b551153-bundle\" (UniqueName: \"kubernetes.io/empty-dir/41d8ed2e-6384-44a0-978f-5c463b551153-bundle\") pod \"41d8ed2e-6384-44a0-978f-5c463b551153\" (UID: \"41d8ed2e-6384-44a0-978f-5c463b551153\") " Oct 30 21:53:19.935331 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:19.935267 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41d8ed2e-6384-44a0-978f-5c463b551153-kube-api-access-pb2mz\" (UniqueName: \"kubernetes.io/projected/41d8ed2e-6384-44a0-978f-5c463b551153-kube-api-access-pb2mz\") pod \"41d8ed2e-6384-44a0-978f-5c463b551153\" (UID: \"41d8ed2e-6384-44a0-978f-5c463b551153\") " Oct 30 21:53:19.935331 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:19.935302 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3d268d9-feff-4875-bf43-d7a44f2b81f9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3d268d9-feff-4875-bf43-d7a44f2b81f9-bundle\") pod \"b3d268d9-feff-4875-bf43-d7a44f2b81f9\" (UID: \"b3d268d9-feff-4875-bf43-d7a44f2b81f9\") " Oct 30 21:53:19.935947 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:19.935918 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d8ed2e-6384-44a0-978f-5c463b551153-bundle" pod "41d8ed2e-6384-44a0-978f-5c463b551153" (UID: "41d8ed2e-6384-44a0-978f-5c463b551153"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:19.935947 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:19.935921 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3d268d9-feff-4875-bf43-d7a44f2b81f9-bundle" pod "b3d268d9-feff-4875-bf43-d7a44f2b81f9" (UID: "b3d268d9-feff-4875-bf43-d7a44f2b81f9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:19.937545 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:19.937523 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3d268d9-feff-4875-bf43-d7a44f2b81f9-kube-api-access-gv2v6" pod "b3d268d9-feff-4875-bf43-d7a44f2b81f9" (UID: "b3d268d9-feff-4875-bf43-d7a44f2b81f9"). InnerVolumeSpecName "kube-api-access-gv2v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:19.937803 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:19.937787 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d8ed2e-6384-44a0-978f-5c463b551153-kube-api-access-pb2mz" pod "41d8ed2e-6384-44a0-978f-5c463b551153" (UID: "41d8ed2e-6384-44a0-978f-5c463b551153"). InnerVolumeSpecName "kube-api-access-pb2mz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:19.940530 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:19.940506 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3d268d9-feff-4875-bf43-d7a44f2b81f9-util" pod "b3d268d9-feff-4875-bf43-d7a44f2b81f9" (UID: "b3d268d9-feff-4875-bf43-d7a44f2b81f9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:19.940860 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:19.940841 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d8ed2e-6384-44a0-978f-5c463b551153-util" pod "41d8ed2e-6384-44a0-978f-5c463b551153" (UID: "41d8ed2e-6384-44a0-978f-5c463b551153"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:20.036374 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:20.036345 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01521af4-ecff-4b27-8a51-6d7ec3bb3efb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/01521af4-ecff-4b27-8a51-6d7ec3bb3efb-bundle\") pod \"01521af4-ecff-4b27-8a51-6d7ec3bb3efb\" (UID: \"01521af4-ecff-4b27-8a51-6d7ec3bb3efb\") " Oct 30 21:53:20.036374 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:20.036376 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01521af4-ecff-4b27-8a51-6d7ec3bb3efb-util\" (UniqueName: \"kubernetes.io/empty-dir/01521af4-ecff-4b27-8a51-6d7ec3bb3efb-util\") pod \"01521af4-ecff-4b27-8a51-6d7ec3bb3efb\" (UID: \"01521af4-ecff-4b27-8a51-6d7ec3bb3efb\") " Oct 30 21:53:20.036596 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:20.036464 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01521af4-ecff-4b27-8a51-6d7ec3bb3efb-kube-api-access-cmrph\" (UniqueName: \"kubernetes.io/projected/01521af4-ecff-4b27-8a51-6d7ec3bb3efb-kube-api-access-cmrph\") pod \"01521af4-ecff-4b27-8a51-6d7ec3bb3efb\" (UID: \"01521af4-ecff-4b27-8a51-6d7ec3bb3efb\") " Oct 30 21:53:20.036656 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:20.036629 5918 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3d268d9-feff-4875-bf43-d7a44f2b81f9-util\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:20.036656 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:20.036641 5918 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/41d8ed2e-6384-44a0-978f-5c463b551153-bundle\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:20.036656 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:20.036650 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb2mz\" (UniqueName: \"kubernetes.io/projected/41d8ed2e-6384-44a0-978f-5c463b551153-kube-api-access-pb2mz\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:20.036803 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:20.036658 5918 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3d268d9-feff-4875-bf43-d7a44f2b81f9-bundle\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:20.036803 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:20.036667 5918 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/41d8ed2e-6384-44a0-978f-5c463b551153-util\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:20.036803 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:20.036676 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv2v6\" (UniqueName: \"kubernetes.io/projected/b3d268d9-feff-4875-bf43-d7a44f2b81f9-kube-api-access-gv2v6\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:20.037107 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:20.037079 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01521af4-ecff-4b27-8a51-6d7ec3bb3efb-bundle" pod "01521af4-ecff-4b27-8a51-6d7ec3bb3efb" (UID: "01521af4-ecff-4b27-8a51-6d7ec3bb3efb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:20.038695 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:20.038672 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01521af4-ecff-4b27-8a51-6d7ec3bb3efb-kube-api-access-cmrph" pod "01521af4-ecff-4b27-8a51-6d7ec3bb3efb" (UID: "01521af4-ecff-4b27-8a51-6d7ec3bb3efb"). InnerVolumeSpecName "kube-api-access-cmrph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:20.041697 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:20.041675 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01521af4-ecff-4b27-8a51-6d7ec3bb3efb-util" pod "01521af4-ecff-4b27-8a51-6d7ec3bb3efb" (UID: "01521af4-ecff-4b27-8a51-6d7ec3bb3efb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:20.137116 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:20.137026 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmrph\" (UniqueName: \"kubernetes.io/projected/01521af4-ecff-4b27-8a51-6d7ec3bb3efb-kube-api-access-cmrph\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:20.137116 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:20.137056 5918 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01521af4-ecff-4b27-8a51-6d7ec3bb3efb-bundle\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:20.137116 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:20.137069 5918 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01521af4-ecff-4b27-8a51-6d7ec3bb3efb-util\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:20.605405 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:20.605367 5918 generic.go:356] "Generic (PLEG): container finished" podID="4ee1ae12-97e4-49f4-8f95-f92746f709b6" containerID="456184ca817bdaa3baf53f0e598c460ed0a5fffb90029d136d09e24941b83c6b" exitCode=0 Oct 30 21:53:20.605863 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:20.605489 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2zwv9" event={"ID":"4ee1ae12-97e4-49f4-8f95-f92746f709b6","Type":"ContainerDied","Data":"456184ca817bdaa3baf53f0e598c460ed0a5fffb90029d136d09e24941b83c6b"} Oct 30 21:53:20.607544 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:20.607520 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71msl55" Oct 30 21:53:20.607660 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:20.607550 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71msl55" event={"ID":"41d8ed2e-6384-44a0-978f-5c463b551153","Type":"ContainerDied","Data":"66328a95d11df149930084d3453381ac7b84c45244eb76a1493b18d419401433"} Oct 30 21:53:20.607660 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:20.607587 5918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66328a95d11df149930084d3453381ac7b84c45244eb76a1493b18d419401433" Oct 30 21:53:20.609550 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:20.609506 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7zbx" event={"ID":"01521af4-ecff-4b27-8a51-6d7ec3bb3efb","Type":"ContainerDied","Data":"2eff324f439f694e514cc649f66c73ac3c7105ec1c7a82e5facdda0440bbf235"} Oct 30 21:53:20.609550 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:20.609547 5918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2eff324f439f694e514cc649f66c73ac3c7105ec1c7a82e5facdda0440bbf235" Oct 30 21:53:20.609550 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:20.609516 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap7zbx" Oct 30 21:53:20.611517 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:20.611493 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2r27" event={"ID":"b3d268d9-feff-4875-bf43-d7a44f2b81f9","Type":"ContainerDied","Data":"8af5e0f502c0965d90dc32aece14ec4993a2161bd3e6c667e0015863e1e0995a"} Oct 30 21:53:20.611601 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:20.611533 5918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8af5e0f502c0965d90dc32aece14ec4993a2161bd3e6c667e0015863e1e0995a" Oct 30 21:53:20.611601 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:20.611543 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2r27" Oct 30 21:53:21.273628 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:53:21.273592 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 21:53:21.764061 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:21.764038 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2zwv9" Oct 30 21:53:21.852173 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:21.852145 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ee1ae12-97e4-49f4-8f95-f92746f709b6-util\" (UniqueName: \"kubernetes.io/empty-dir/4ee1ae12-97e4-49f4-8f95-f92746f709b6-util\") pod \"4ee1ae12-97e4-49f4-8f95-f92746f709b6\" (UID: \"4ee1ae12-97e4-49f4-8f95-f92746f709b6\") " Oct 30 21:53:21.852344 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:21.852187 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ee1ae12-97e4-49f4-8f95-f92746f709b6-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ee1ae12-97e4-49f4-8f95-f92746f709b6-bundle\") pod \"4ee1ae12-97e4-49f4-8f95-f92746f709b6\" (UID: \"4ee1ae12-97e4-49f4-8f95-f92746f709b6\") " Oct 30 21:53:21.852344 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:21.852250 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ee1ae12-97e4-49f4-8f95-f92746f709b6-kube-api-access-hv8vv\" (UniqueName: \"kubernetes.io/projected/4ee1ae12-97e4-49f4-8f95-f92746f709b6-kube-api-access-hv8vv\") pod \"4ee1ae12-97e4-49f4-8f95-f92746f709b6\" (UID: \"4ee1ae12-97e4-49f4-8f95-f92746f709b6\") " Oct 30 21:53:21.852661 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:21.852637 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ee1ae12-97e4-49f4-8f95-f92746f709b6-bundle" pod "4ee1ae12-97e4-49f4-8f95-f92746f709b6" (UID: "4ee1ae12-97e4-49f4-8f95-f92746f709b6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:21.854492 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:21.854471 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ee1ae12-97e4-49f4-8f95-f92746f709b6-kube-api-access-hv8vv" pod "4ee1ae12-97e4-49f4-8f95-f92746f709b6" (UID: "4ee1ae12-97e4-49f4-8f95-f92746f709b6"). InnerVolumeSpecName "kube-api-access-hv8vv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:21.857435 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:21.857411 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ee1ae12-97e4-49f4-8f95-f92746f709b6-util" pod "4ee1ae12-97e4-49f4-8f95-f92746f709b6" (UID: "4ee1ae12-97e4-49f4-8f95-f92746f709b6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:21.953167 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:21.953140 5918 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ee1ae12-97e4-49f4-8f95-f92746f709b6-util\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:21.953167 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:21.953164 5918 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ee1ae12-97e4-49f4-8f95-f92746f709b6-bundle\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:21.953342 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:21.953174 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv8vv\" (UniqueName: \"kubernetes.io/projected/4ee1ae12-97e4-49f4-8f95-f92746f709b6-kube-api-access-hv8vv\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:22.621107 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:22.621068 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2zwv9" event={"ID":"4ee1ae12-97e4-49f4-8f95-f92746f709b6","Type":"ContainerDied","Data":"7ffd5481a49c9253ebf77ba5d9e39fc269de32bc130e4500b1db199c1bf773c7"} Oct 30 21:53:22.621107 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:22.621096 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2zwv9" Oct 30 21:53:22.621324 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:22.621123 5918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ffd5481a49c9253ebf77ba5d9e39fc269de32bc130e4500b1db199c1bf773c7" Oct 30 21:53:26.779508 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.779472 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-22s6p"] Oct 30 21:53:26.779896 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.779820 5918 cpu_manager.go:414] "containerMap: removing container" podUID="41d8ed2e-6384-44a0-978f-5c463b551153" containerName="extract" Oct 30 21:53:26.779896 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.779832 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41d8ed2e-6384-44a0-978f-5c463b551153" containerName="extract" Oct 30 21:53:26.779896 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.779846 5918 cpu_manager.go:414] "containerMap: removing container" podUID="b3d268d9-feff-4875-bf43-d7a44f2b81f9" containerName="extract" Oct 30 21:53:26.779896 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.779850 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3d268d9-feff-4875-bf43-d7a44f2b81f9" containerName="extract" Oct 30 21:53:26.779896 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.779880 5918 cpu_manager.go:414] "containerMap: removing container" podUID="41d8ed2e-6384-44a0-978f-5c463b551153" containerName="util" Oct 30 21:53:26.779896 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.779887 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41d8ed2e-6384-44a0-978f-5c463b551153" containerName="util" Oct 30 21:53:26.779896 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.779897 5918 cpu_manager.go:414] "containerMap: removing container" podUID="4ee1ae12-97e4-49f4-8f95-f92746f709b6" containerName="util" Oct 30 21:53:26.780144 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.779903 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ee1ae12-97e4-49f4-8f95-f92746f709b6" containerName="util" Oct 30 21:53:26.780144 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.779911 5918 cpu_manager.go:414] "containerMap: removing container" podUID="01521af4-ecff-4b27-8a51-6d7ec3bb3efb" containerName="pull" Oct 30 21:53:26.780144 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.779915 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01521af4-ecff-4b27-8a51-6d7ec3bb3efb" containerName="pull" Oct 30 21:53:26.780144 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.779926 5918 cpu_manager.go:414] "containerMap: removing container" podUID="01521af4-ecff-4b27-8a51-6d7ec3bb3efb" containerName="util" Oct 30 21:53:26.780144 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.779930 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01521af4-ecff-4b27-8a51-6d7ec3bb3efb" containerName="util" Oct 30 21:53:26.780144 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.779935 5918 cpu_manager.go:414] "containerMap: removing container" podUID="b3d268d9-feff-4875-bf43-d7a44f2b81f9" containerName="util" Oct 30 21:53:26.780144 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.779939 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3d268d9-feff-4875-bf43-d7a44f2b81f9" containerName="util" Oct 30 21:53:26.780144 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.779948 5918 cpu_manager.go:414] "containerMap: removing container" podUID="b3d268d9-feff-4875-bf43-d7a44f2b81f9" containerName="pull" Oct 30 21:53:26.780144 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.779955 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3d268d9-feff-4875-bf43-d7a44f2b81f9" containerName="pull" Oct 30 21:53:26.780144 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.779965 5918 cpu_manager.go:414] "containerMap: removing container" podUID="01521af4-ecff-4b27-8a51-6d7ec3bb3efb" containerName="extract" Oct 30 21:53:26.780144 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.779970 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01521af4-ecff-4b27-8a51-6d7ec3bb3efb" containerName="extract" Oct 30 21:53:26.780144 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.779975 5918 cpu_manager.go:414] "containerMap: removing container" podUID="41d8ed2e-6384-44a0-978f-5c463b551153" containerName="pull" Oct 30 21:53:26.780144 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.779979 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41d8ed2e-6384-44a0-978f-5c463b551153" containerName="pull" Oct 30 21:53:26.780144 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.779985 5918 cpu_manager.go:414] "containerMap: removing container" podUID="4ee1ae12-97e4-49f4-8f95-f92746f709b6" containerName="pull" Oct 30 21:53:26.780144 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.779989 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ee1ae12-97e4-49f4-8f95-f92746f709b6" containerName="pull" Oct 30 21:53:26.780144 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.779995 5918 cpu_manager.go:414] "containerMap: removing container" podUID="4ee1ae12-97e4-49f4-8f95-f92746f709b6" containerName="extract" Oct 30 21:53:26.780144 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.779999 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ee1ae12-97e4-49f4-8f95-f92746f709b6" containerName="extract" Oct 30 21:53:26.780144 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.780053 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="01521af4-ecff-4b27-8a51-6d7ec3bb3efb" containerName="extract" Oct 30 21:53:26.780144 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.780060 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ee1ae12-97e4-49f4-8f95-f92746f709b6" containerName="extract" Oct 30 21:53:26.780144 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.780066 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="41d8ed2e-6384-44a0-978f-5c463b551153" containerName="extract" Oct 30 21:53:26.780144 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.780072 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3d268d9-feff-4875-bf43-d7a44f2b81f9" containerName="extract" Oct 30 21:53:26.782076 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.782060 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-22s6p" Oct 30 21:53:26.785119 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.785086 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljpjp\" (UniqueName: \"kubernetes.io/projected/a04dc7ee-afac-43c8-b345-4fd031162243-kube-api-access-ljpjp\") pod \"opendatahub-operator-controller-manager-77b67b5d96-22s6p\" (UID: \"a04dc7ee-afac-43c8-b345-4fd031162243\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-22s6p" Oct 30 21:53:26.785250 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.785135 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a04dc7ee-afac-43c8-b345-4fd031162243-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77b67b5d96-22s6p\" (UID: \"a04dc7ee-afac-43c8-b345-4fd031162243\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-22s6p" Oct 30 21:53:26.785338 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.785313 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a04dc7ee-afac-43c8-b345-4fd031162243-webhook-cert\") pod \"opendatahub-operator-controller-manager-77b67b5d96-22s6p\" (UID: \"a04dc7ee-afac-43c8-b345-4fd031162243\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-22s6p" Oct 30 21:53:26.785407 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.785379 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a04dc7ee-afac-43c8-b345-4fd031162243-metrics-cert\") pod \"opendatahub-operator-controller-manager-77b67b5d96-22s6p\" (UID: \"a04dc7ee-afac-43c8-b345-4fd031162243\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-22s6p" Oct 30 21:53:26.785464 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.785447 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a04dc7ee-afac-43c8-b345-4fd031162243-manifests\") pod \"opendatahub-operator-controller-manager-77b67b5d96-22s6p\" (UID: \"a04dc7ee-afac-43c8-b345-4fd031162243\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-22s6p" Oct 30 21:53:26.785523 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.785480 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a04dc7ee-afac-43c8-b345-4fd031162243-tmp\") pod \"opendatahub-operator-controller-manager-77b67b5d96-22s6p\" (UID: \"a04dc7ee-afac-43c8-b345-4fd031162243\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-22s6p" Oct 30 21:53:26.837517 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.837497 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 21:53:26.837660 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.837538 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 21:53:26.886169 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.886140 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a04dc7ee-afac-43c8-b345-4fd031162243-webhook-cert\") pod \"opendatahub-operator-controller-manager-77b67b5d96-22s6p\" (UID: \"a04dc7ee-afac-43c8-b345-4fd031162243\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-22s6p" Oct 30 21:53:26.886335 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.886185 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a04dc7ee-afac-43c8-b345-4fd031162243-metrics-cert\") pod \"opendatahub-operator-controller-manager-77b67b5d96-22s6p\" (UID: \"a04dc7ee-afac-43c8-b345-4fd031162243\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-22s6p" Oct 30 21:53:26.886335 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.886251 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a04dc7ee-afac-43c8-b345-4fd031162243-manifests\") pod \"opendatahub-operator-controller-manager-77b67b5d96-22s6p\" (UID: \"a04dc7ee-afac-43c8-b345-4fd031162243\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-22s6p" Oct 30 21:53:26.886335 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.886268 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a04dc7ee-afac-43c8-b345-4fd031162243-tmp\") pod \"opendatahub-operator-controller-manager-77b67b5d96-22s6p\" (UID: \"a04dc7ee-afac-43c8-b345-4fd031162243\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-22s6p" Oct 30 21:53:26.886335 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.886309 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljpjp\" (UniqueName: \"kubernetes.io/projected/a04dc7ee-afac-43c8-b345-4fd031162243-kube-api-access-ljpjp\") pod \"opendatahub-operator-controller-manager-77b67b5d96-22s6p\" (UID: \"a04dc7ee-afac-43c8-b345-4fd031162243\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-22s6p" Oct 30 21:53:26.886565 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.886371 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a04dc7ee-afac-43c8-b345-4fd031162243-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77b67b5d96-22s6p\" (UID: \"a04dc7ee-afac-43c8-b345-4fd031162243\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-22s6p" Oct 30 21:53:26.886679 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.886653 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a04dc7ee-afac-43c8-b345-4fd031162243-manifests\") pod \"opendatahub-operator-controller-manager-77b67b5d96-22s6p\" (UID: \"a04dc7ee-afac-43c8-b345-4fd031162243\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-22s6p" Oct 30 21:53:26.886679 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.886668 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 21:53:26.886765 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.886689 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 21:53:26.886765 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.886741 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-4xvvf\"" Oct 30 21:53:26.886852 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.886799 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 21:53:26.888701 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.888682 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a04dc7ee-afac-43c8-b345-4fd031162243-tmp\") pod \"opendatahub-operator-controller-manager-77b67b5d96-22s6p\" (UID: \"a04dc7ee-afac-43c8-b345-4fd031162243\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-22s6p" Oct 30 21:53:26.899357 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.899333 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a04dc7ee-afac-43c8-b345-4fd031162243-metrics-cert\") pod \"opendatahub-operator-controller-manager-77b67b5d96-22s6p\" (UID: \"a04dc7ee-afac-43c8-b345-4fd031162243\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-22s6p" Oct 30 21:53:26.899478 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.899450 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a04dc7ee-afac-43c8-b345-4fd031162243-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77b67b5d96-22s6p\" (UID: \"a04dc7ee-afac-43c8-b345-4fd031162243\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-22s6p" Oct 30 21:53:26.900039 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.900007 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a04dc7ee-afac-43c8-b345-4fd031162243-webhook-cert\") pod \"opendatahub-operator-controller-manager-77b67b5d96-22s6p\" (UID: \"a04dc7ee-afac-43c8-b345-4fd031162243\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-22s6p" Oct 30 21:53:26.900612 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.900595 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-22s6p"] Oct 30 21:53:26.956095 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:26.956064 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljpjp\" (UniqueName: \"kubernetes.io/projected/a04dc7ee-afac-43c8-b345-4fd031162243-kube-api-access-ljpjp\") pod \"opendatahub-operator-controller-manager-77b67b5d96-22s6p\" (UID: \"a04dc7ee-afac-43c8-b345-4fd031162243\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-22s6p" Oct 30 21:53:27.094009 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:27.093934 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-22s6p" Oct 30 21:53:27.581391 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:27.581366 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-22s6p"] Oct 30 21:53:27.583781 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:53:27.583745 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda04dc7ee_afac_43c8_b345_4fd031162243.slice/crio-69fbddc18966acc8c68a7d24c99b36bb02c9dcecdd3e8797befe87456c642b98 WatchSource:0}: Error finding container 69fbddc18966acc8c68a7d24c99b36bb02c9dcecdd3e8797befe87456c642b98: Status 404 returned error can't find the container with id 69fbddc18966acc8c68a7d24c99b36bb02c9dcecdd3e8797befe87456c642b98 Oct 30 21:53:27.643805 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:27.643768 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-22s6p" event={"ID":"a04dc7ee-afac-43c8-b345-4fd031162243","Type":"ContainerStarted","Data":"69fbddc18966acc8c68a7d24c99b36bb02c9dcecdd3e8797befe87456c642b98"} Oct 30 21:53:31.661994 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:31.661953 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-22s6p" event={"ID":"a04dc7ee-afac-43c8-b345-4fd031162243","Type":"ContainerStarted","Data":"7bbe343857f160b45d522e50089f487b13662b030810efe1aab2bfd484c4bc38"} Oct 30 21:53:32.667846 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:32.667813 5918 generic.go:356] "Generic (PLEG): container finished" podID="a04dc7ee-afac-43c8-b345-4fd031162243" containerID="7bbe343857f160b45d522e50089f487b13662b030810efe1aab2bfd484c4bc38" exitCode=0 Oct 30 21:53:32.668297 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:32.667889 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-22s6p" event={"ID":"a04dc7ee-afac-43c8-b345-4fd031162243","Type":"ContainerDied","Data":"7bbe343857f160b45d522e50089f487b13662b030810efe1aab2bfd484c4bc38"} Oct 30 21:53:33.675664 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:33.675631 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-22s6p" event={"ID":"a04dc7ee-afac-43c8-b345-4fd031162243","Type":"ContainerStarted","Data":"55ba5000eb1abd307768b1fd4e2e147b1005cce8cc9fd042a881571e1088e91a"} Oct 30 21:53:33.676024 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:33.675744 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-22s6p" Oct 30 21:53:33.932591 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:33.932544 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-22s6p" podStartSLOduration=3.062999305 podStartE2EDuration="7.932530142s" totalImagesPullingTime="4.396901676s" totalInitContainerRuntime="472.629161ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:33.881660758 +0000 UTC m=+174.152358865" watchObservedRunningTime="2026-10-30 21:53:33.932530142 +0000 UTC m=+174.203228225" Oct 30 21:53:38.403851 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:38.403817 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-wqlvd"] Oct 30 21:53:38.407352 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:38.407337 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wqlvd" Oct 30 21:53:38.475727 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:38.475700 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2wd8\" (UniqueName: \"kubernetes.io/projected/052814ae-fe16-48d7-851d-72e7e8553132-kube-api-access-d2wd8\") pod \"jobset-operator-674cd59dc5-wqlvd\" (UID: \"052814ae-fe16-48d7-851d-72e7e8553132\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wqlvd" Oct 30 21:53:38.475881 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:38.475739 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/052814ae-fe16-48d7-851d-72e7e8553132-tmp\") pod \"jobset-operator-674cd59dc5-wqlvd\" (UID: \"052814ae-fe16-48d7-851d-72e7e8553132\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wqlvd" Oct 30 21:53:38.511752 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:38.511721 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 21:53:38.511889 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:38.511752 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:53:38.511889 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:38.511873 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-cpxsv\"" Oct 30 21:53:38.512980 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:38.512959 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-wqlvd"] Oct 30 21:53:38.578175 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:38.577428 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2wd8\" (UniqueName: \"kubernetes.io/projected/052814ae-fe16-48d7-851d-72e7e8553132-kube-api-access-d2wd8\") pod \"jobset-operator-674cd59dc5-wqlvd\" (UID: \"052814ae-fe16-48d7-851d-72e7e8553132\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wqlvd" Oct 30 21:53:38.578175 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:38.577482 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/052814ae-fe16-48d7-851d-72e7e8553132-tmp\") pod \"jobset-operator-674cd59dc5-wqlvd\" (UID: \"052814ae-fe16-48d7-851d-72e7e8553132\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wqlvd" Oct 30 21:53:38.578175 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:38.577886 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/052814ae-fe16-48d7-851d-72e7e8553132-tmp\") pod \"jobset-operator-674cd59dc5-wqlvd\" (UID: \"052814ae-fe16-48d7-851d-72e7e8553132\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wqlvd" Oct 30 21:53:38.641197 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:38.641169 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2wd8\" (UniqueName: \"kubernetes.io/projected/052814ae-fe16-48d7-851d-72e7e8553132-kube-api-access-d2wd8\") pod \"jobset-operator-674cd59dc5-wqlvd\" (UID: \"052814ae-fe16-48d7-851d-72e7e8553132\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wqlvd" Oct 30 21:53:38.726669 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:38.726639 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wqlvd" Oct 30 21:53:39.099639 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:39.099560 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-wqlvd"] Oct 30 21:53:39.102661 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:53:39.102635 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod052814ae_fe16_48d7_851d_72e7e8553132.slice/crio-8ed205dd42a2825bc41b40c8e6dbe586bb42b396d23ab89ab4ee61f5193948dc WatchSource:0}: Error finding container 8ed205dd42a2825bc41b40c8e6dbe586bb42b396d23ab89ab4ee61f5193948dc: Status 404 returned error can't find the container with id 8ed205dd42a2825bc41b40c8e6dbe586bb42b396d23ab89ab4ee61f5193948dc Oct 30 21:53:39.705622 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:39.705579 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wqlvd" event={"ID":"052814ae-fe16-48d7-851d-72e7e8553132","Type":"ContainerStarted","Data":"8ed205dd42a2825bc41b40c8e6dbe586bb42b396d23ab89ab4ee61f5193948dc"} Oct 30 21:53:43.724847 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:43.724812 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wqlvd" event={"ID":"052814ae-fe16-48d7-851d-72e7e8553132","Type":"ContainerStarted","Data":"9af297ba00398de7accc2eb94c247246b592cc226e847a2103f6727569406e28"} Oct 30 21:53:43.849258 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:43.849169 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wqlvd" podStartSLOduration=1.9781239940000002 podStartE2EDuration="5.849151879s" totalImagesPullingTime="3.871027885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:43.846660076 +0000 UTC m=+184.117358161" watchObservedRunningTime="2026-10-30 21:53:43.849151879 +0000 UTC m=+184.119849963" Oct 30 21:53:44.681618 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:44.681588 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77b67b5d96-22s6p" Oct 30 21:53:51.273326 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:53:51.273235 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 21:53:59.204670 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:59.204642 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-744pk"] Oct 30 21:53:59.207679 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:59.207652 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-744pk" Oct 30 21:53:59.266964 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:59.266932 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-744pk"] Oct 30 21:53:59.316232 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:59.316173 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 21:53:59.316385 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:59.316206 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 21:53:59.316385 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:59.316301 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-qtj6x\"" Oct 30 21:53:59.360026 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:59.359993 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4zn5\" (UniqueName: \"kubernetes.io/projected/7016d340-efdf-4e80-8086-9d11a5baa6c6-kube-api-access-z4zn5\") pod \"cert-manager-webhook-5dbbbf75cc-744pk\" (UID: \"7016d340-efdf-4e80-8086-9d11a5baa6c6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-744pk" Oct 30 21:53:59.360204 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:59.360135 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7016d340-efdf-4e80-8086-9d11a5baa6c6-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-744pk\" (UID: \"7016d340-efdf-4e80-8086-9d11a5baa6c6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-744pk" Oct 30 21:53:59.461006 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:59.460928 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4zn5\" (UniqueName: \"kubernetes.io/projected/7016d340-efdf-4e80-8086-9d11a5baa6c6-kube-api-access-z4zn5\") pod \"cert-manager-webhook-5dbbbf75cc-744pk\" (UID: \"7016d340-efdf-4e80-8086-9d11a5baa6c6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-744pk" Oct 30 21:53:59.461006 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:59.460989 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7016d340-efdf-4e80-8086-9d11a5baa6c6-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-744pk\" (UID: \"7016d340-efdf-4e80-8086-9d11a5baa6c6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-744pk" Oct 30 21:53:59.526172 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:59.526136 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7016d340-efdf-4e80-8086-9d11a5baa6c6-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-744pk\" (UID: \"7016d340-efdf-4e80-8086-9d11a5baa6c6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-744pk" Oct 30 21:53:59.526407 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:59.526282 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4zn5\" (UniqueName: \"kubernetes.io/projected/7016d340-efdf-4e80-8086-9d11a5baa6c6-kube-api-access-z4zn5\") pod \"cert-manager-webhook-5dbbbf75cc-744pk\" (UID: \"7016d340-efdf-4e80-8086-9d11a5baa6c6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-744pk" Oct 30 21:53:59.820744 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:53:59.820658 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-744pk" Oct 30 21:54:00.198011 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:00.197986 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-744pk"] Oct 30 21:54:00.200523 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:54:00.200484 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7016d340_efdf_4e80_8086_9d11a5baa6c6.slice/crio-1fd344ce529fa9d6d48d0fb1b30ab7d7323a91420e067e032d40be169ef35f4f WatchSource:0}: Error finding container 1fd344ce529fa9d6d48d0fb1b30ab7d7323a91420e067e032d40be169ef35f4f: Status 404 returned error can't find the container with id 1fd344ce529fa9d6d48d0fb1b30ab7d7323a91420e067e032d40be169ef35f4f Oct 30 21:54:00.798301 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:00.798265 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-744pk" event={"ID":"7016d340-efdf-4e80-8086-9d11a5baa6c6","Type":"ContainerStarted","Data":"1fd344ce529fa9d6d48d0fb1b30ab7d7323a91420e067e032d40be169ef35f4f"} Oct 30 21:54:01.320969 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:01.320935 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-7j8xr"] Oct 30 21:54:01.323662 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:01.323646 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7j8xr" Oct 30 21:54:01.387699 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:01.387674 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-7j8xr"] Oct 30 21:54:01.478119 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:01.478091 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66q22\" (UniqueName: \"kubernetes.io/projected/a659ab08-f149-4f05-b64f-b35ca37877b0-kube-api-access-66q22\") pod \"cert-manager-cainjector-5c57bbd4fc-7j8xr\" (UID: \"a659ab08-f149-4f05-b64f-b35ca37877b0\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7j8xr" Oct 30 21:54:01.478313 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:01.478139 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a659ab08-f149-4f05-b64f-b35ca37877b0-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-7j8xr\" (UID: \"a659ab08-f149-4f05-b64f-b35ca37877b0\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7j8xr" Oct 30 21:54:01.493388 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:01.493365 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-j4dqx\"" Oct 30 21:54:01.579260 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:01.579143 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a659ab08-f149-4f05-b64f-b35ca37877b0-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-7j8xr\" (UID: \"a659ab08-f149-4f05-b64f-b35ca37877b0\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7j8xr" Oct 30 21:54:01.579446 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:01.579280 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66q22\" (UniqueName: \"kubernetes.io/projected/a659ab08-f149-4f05-b64f-b35ca37877b0-kube-api-access-66q22\") pod \"cert-manager-cainjector-5c57bbd4fc-7j8xr\" (UID: \"a659ab08-f149-4f05-b64f-b35ca37877b0\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7j8xr" Oct 30 21:54:01.653229 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:01.653188 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a659ab08-f149-4f05-b64f-b35ca37877b0-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-7j8xr\" (UID: \"a659ab08-f149-4f05-b64f-b35ca37877b0\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7j8xr" Oct 30 21:54:01.653384 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:01.653282 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66q22\" (UniqueName: \"kubernetes.io/projected/a659ab08-f149-4f05-b64f-b35ca37877b0-kube-api-access-66q22\") pod \"cert-manager-cainjector-5c57bbd4fc-7j8xr\" (UID: \"a659ab08-f149-4f05-b64f-b35ca37877b0\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7j8xr" Oct 30 21:54:01.933609 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:01.933576 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7j8xr" Oct 30 21:54:02.369343 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:02.369267 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-7j8xr"] Oct 30 21:54:02.372874 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:54:02.372848 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda659ab08_f149_4f05_b64f_b35ca37877b0.slice/crio-e6c2d2ce5b78e629bec1de484305de643c897fbea69f451744e40da15a04af61 WatchSource:0}: Error finding container e6c2d2ce5b78e629bec1de484305de643c897fbea69f451744e40da15a04af61: Status 404 returned error can't find the container with id e6c2d2ce5b78e629bec1de484305de643c897fbea69f451744e40da15a04af61 Oct 30 21:54:02.809012 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:02.808974 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7j8xr" event={"ID":"a659ab08-f149-4f05-b64f-b35ca37877b0","Type":"ContainerStarted","Data":"e6c2d2ce5b78e629bec1de484305de643c897fbea69f451744e40da15a04af61"} Oct 30 21:54:05.825019 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:05.824974 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-744pk" event={"ID":"7016d340-efdf-4e80-8086-9d11a5baa6c6","Type":"ContainerStarted","Data":"94a9b97f429cad89f15f8529235eca864e89cac130ca123d061aabfaa0ba4cc4"} Oct 30 21:54:05.825495 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:05.825109 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-744pk" Oct 30 21:54:05.826456 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:05.826432 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7j8xr" event={"ID":"a659ab08-f149-4f05-b64f-b35ca37877b0","Type":"ContainerStarted","Data":"ce36a299a2ccc97a4d6f1b9fb1e6a995be5678d9a90d85bdcf4e02f8ddb7df87"} Oct 30 21:54:05.863749 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:05.863720 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-b76569bf4-jtdtb"] Oct 30 21:54:05.869591 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:05.869572 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b76569bf4-jtdtb" Oct 30 21:54:05.927665 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:05.927636 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b76569bf4-jtdtb"] Oct 30 21:54:05.937616 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:05.937588 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 21:54:05.937766 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:05.937655 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 21:54:05.937766 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:05.937747 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 21:54:05.963416 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:05.963386 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 21:54:05.963565 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:05.963443 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-s4xz8\"" Oct 30 21:54:05.988277 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:05.988207 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-744pk" podStartSLOduration=2.27821822 podStartE2EDuration="6.988193069s" totalImagesPullingTime="4.709974849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:05.978301598 +0000 UTC m=+206.248999681" watchObservedRunningTime="2026-10-30 21:54:05.988193069 +0000 UTC m=+206.258891153" Oct 30 21:54:06.020538 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:06.020484 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/669ab9c8-6c2a-4689-9e4d-e59f77e78fbb-tmp\") pod \"kube-auth-proxy-b76569bf4-jtdtb\" (UID: \"669ab9c8-6c2a-4689-9e4d-e59f77e78fbb\") " pod="openshift-ingress/kube-auth-proxy-b76569bf4-jtdtb" Oct 30 21:54:06.020538 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:06.020531 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-628qt\" (UniqueName: \"kubernetes.io/projected/669ab9c8-6c2a-4689-9e4d-e59f77e78fbb-kube-api-access-628qt\") pod \"kube-auth-proxy-b76569bf4-jtdtb\" (UID: \"669ab9c8-6c2a-4689-9e4d-e59f77e78fbb\") " pod="openshift-ingress/kube-auth-proxy-b76569bf4-jtdtb" Oct 30 21:54:06.020840 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:06.020718 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/669ab9c8-6c2a-4689-9e4d-e59f77e78fbb-tls-certs\") pod \"kube-auth-proxy-b76569bf4-jtdtb\" (UID: \"669ab9c8-6c2a-4689-9e4d-e59f77e78fbb\") " pod="openshift-ingress/kube-auth-proxy-b76569bf4-jtdtb" Oct 30 21:54:06.101917 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:06.101817 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7j8xr" podStartSLOduration=2.279614258 podStartE2EDuration="5.101807217s" totalImagesPullingTime="2.822192959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:06.101318271 +0000 UTC m=+206.372016355" watchObservedRunningTime="2026-10-30 21:54:06.101807217 +0000 UTC m=+206.372505295" Oct 30 21:54:06.121593 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:06.121563 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/669ab9c8-6c2a-4689-9e4d-e59f77e78fbb-tls-certs\") pod \"kube-auth-proxy-b76569bf4-jtdtb\" (UID: \"669ab9c8-6c2a-4689-9e4d-e59f77e78fbb\") " pod="openshift-ingress/kube-auth-proxy-b76569bf4-jtdtb" Oct 30 21:54:06.121766 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:06.121608 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/669ab9c8-6c2a-4689-9e4d-e59f77e78fbb-tmp\") pod \"kube-auth-proxy-b76569bf4-jtdtb\" (UID: \"669ab9c8-6c2a-4689-9e4d-e59f77e78fbb\") " pod="openshift-ingress/kube-auth-proxy-b76569bf4-jtdtb" Oct 30 21:54:06.121766 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:06.121630 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-628qt\" (UniqueName: \"kubernetes.io/projected/669ab9c8-6c2a-4689-9e4d-e59f77e78fbb-kube-api-access-628qt\") pod \"kube-auth-proxy-b76569bf4-jtdtb\" (UID: \"669ab9c8-6c2a-4689-9e4d-e59f77e78fbb\") " pod="openshift-ingress/kube-auth-proxy-b76569bf4-jtdtb" Oct 30 21:54:06.123990 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:06.123959 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/669ab9c8-6c2a-4689-9e4d-e59f77e78fbb-tmp\") pod \"kube-auth-proxy-b76569bf4-jtdtb\" (UID: \"669ab9c8-6c2a-4689-9e4d-e59f77e78fbb\") " pod="openshift-ingress/kube-auth-proxy-b76569bf4-jtdtb" Oct 30 21:54:06.124516 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:06.124492 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/669ab9c8-6c2a-4689-9e4d-e59f77e78fbb-tls-certs\") pod \"kube-auth-proxy-b76569bf4-jtdtb\" (UID: \"669ab9c8-6c2a-4689-9e4d-e59f77e78fbb\") " pod="openshift-ingress/kube-auth-proxy-b76569bf4-jtdtb" Oct 30 21:54:06.187067 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:06.187028 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-628qt\" (UniqueName: \"kubernetes.io/projected/669ab9c8-6c2a-4689-9e4d-e59f77e78fbb-kube-api-access-628qt\") pod \"kube-auth-proxy-b76569bf4-jtdtb\" (UID: \"669ab9c8-6c2a-4689-9e4d-e59f77e78fbb\") " pod="openshift-ingress/kube-auth-proxy-b76569bf4-jtdtb" Oct 30 21:54:06.480739 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:06.480705 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b76569bf4-jtdtb" Oct 30 21:54:06.859096 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:06.859073 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b76569bf4-jtdtb"] Oct 30 21:54:06.861011 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:54:06.860984 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod669ab9c8_6c2a_4689_9e4d_e59f77e78fbb.slice/crio-a552f2c9fd85e4fc83764a2db43452582844c2a5acf90f82367ae41a1f0a231c WatchSource:0}: Error finding container a552f2c9fd85e4fc83764a2db43452582844c2a5acf90f82367ae41a1f0a231c: Status 404 returned error can't find the container with id a552f2c9fd85e4fc83764a2db43452582844c2a5acf90f82367ae41a1f0a231c Oct 30 21:54:07.838959 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:07.838922 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b76569bf4-jtdtb" event={"ID":"669ab9c8-6c2a-4689-9e4d-e59f77e78fbb","Type":"ContainerStarted","Data":"a552f2c9fd85e4fc83764a2db43452582844c2a5acf90f82367ae41a1f0a231c"} Oct 30 21:54:09.858419 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:09.858386 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b76569bf4-jtdtb" event={"ID":"669ab9c8-6c2a-4689-9e4d-e59f77e78fbb","Type":"ContainerStarted","Data":"26732507c3da122cc37b7752c8d18eccea59c76b11adc741fefd8c5c56e57392"} Oct 30 21:54:09.978053 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:09.978005 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-b76569bf4-jtdtb" podStartSLOduration=2.225324982 podStartE2EDuration="4.97799237s" totalImagesPullingTime="2.752667388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:09.977653224 +0000 UTC m=+210.248351308" watchObservedRunningTime="2026-10-30 21:54:09.97799237 +0000 UTC m=+210.248690454" Oct 30 21:54:10.270248 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:10.270161 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vrjss"] Oct 30 21:54:10.273774 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:10.273758 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-vrjss" Oct 30 21:54:10.308145 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:10.308120 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ld2vq"] Oct 30 21:54:10.311349 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:10.311334 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-ld2vq" Oct 30 21:54:10.328810 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:10.328794 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-z7pdc\"" Oct 30 21:54:10.357710 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:10.357690 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 21:54:10.358778 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:10.358755 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vrjss"] Oct 30 21:54:10.379256 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:10.379233 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 21:54:10.379332 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:10.379273 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 21:54:10.394856 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:10.394834 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ld2vq"] Oct 30 21:54:10.429336 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:10.429316 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-86hk6\"" Oct 30 21:54:10.460431 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:10.460411 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b6dd786f-1bf3-4afc-9d6e-250aecd415bc-config-file\") pod \"limitador-limitador-74dfcfb894-vrjss\" (UID: \"b6dd786f-1bf3-4afc-9d6e-250aecd415bc\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vrjss" Oct 30 21:54:10.460544 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:10.460441 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rznbc\" (UniqueName: \"kubernetes.io/projected/766c56e9-71be-48d2-ae53-3b8cad19ec03-kube-api-access-rznbc\") pod \"authorino-57f7f8659f-ld2vq\" (UID: \"766c56e9-71be-48d2-ae53-3b8cad19ec03\") " pod="kuadrant-system/authorino-57f7f8659f-ld2vq" Oct 30 21:54:10.460602 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:10.460582 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kglpg\" (UniqueName: \"kubernetes.io/projected/b6dd786f-1bf3-4afc-9d6e-250aecd415bc-kube-api-access-kglpg\") pod \"limitador-limitador-74dfcfb894-vrjss\" (UID: \"b6dd786f-1bf3-4afc-9d6e-250aecd415bc\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vrjss" Oct 30 21:54:10.520467 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:10.520392 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vrjss"] Oct 30 21:54:10.561190 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:10.561162 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kglpg\" (UniqueName: \"kubernetes.io/projected/b6dd786f-1bf3-4afc-9d6e-250aecd415bc-kube-api-access-kglpg\") pod \"limitador-limitador-74dfcfb894-vrjss\" (UID: \"b6dd786f-1bf3-4afc-9d6e-250aecd415bc\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vrjss" Oct 30 21:54:10.561324 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:10.561201 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b6dd786f-1bf3-4afc-9d6e-250aecd415bc-config-file\") pod \"limitador-limitador-74dfcfb894-vrjss\" (UID: \"b6dd786f-1bf3-4afc-9d6e-250aecd415bc\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vrjss" Oct 30 21:54:10.561324 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:10.561251 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rznbc\" (UniqueName: \"kubernetes.io/projected/766c56e9-71be-48d2-ae53-3b8cad19ec03-kube-api-access-rznbc\") pod \"authorino-57f7f8659f-ld2vq\" (UID: \"766c56e9-71be-48d2-ae53-3b8cad19ec03\") " pod="kuadrant-system/authorino-57f7f8659f-ld2vq" Oct 30 21:54:10.561736 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:10.561719 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b6dd786f-1bf3-4afc-9d6e-250aecd415bc-config-file\") pod \"limitador-limitador-74dfcfb894-vrjss\" (UID: \"b6dd786f-1bf3-4afc-9d6e-250aecd415bc\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vrjss" Oct 30 21:54:10.624414 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:10.624385 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rznbc\" (UniqueName: \"kubernetes.io/projected/766c56e9-71be-48d2-ae53-3b8cad19ec03-kube-api-access-rznbc\") pod \"authorino-57f7f8659f-ld2vq\" (UID: \"766c56e9-71be-48d2-ae53-3b8cad19ec03\") " pod="kuadrant-system/authorino-57f7f8659f-ld2vq" Oct 30 21:54:10.624850 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:10.624832 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kglpg\" (UniqueName: \"kubernetes.io/projected/b6dd786f-1bf3-4afc-9d6e-250aecd415bc-kube-api-access-kglpg\") pod \"limitador-limitador-74dfcfb894-vrjss\" (UID: \"b6dd786f-1bf3-4afc-9d6e-250aecd415bc\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vrjss" Oct 30 21:54:10.886139 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:10.886058 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-vrjss" Oct 30 21:54:10.921610 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:10.921577 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-ld2vq" Oct 30 21:54:11.264717 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:11.264672 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vrjss"] Oct 30 21:54:11.266794 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:54:11.266770 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6dd786f_1bf3_4afc_9d6e_250aecd415bc.slice/crio-3aae9aba401edc97b62b08b08701df28d5aa654505cc7a84d36392352aad8a4f WatchSource:0}: Error finding container 3aae9aba401edc97b62b08b08701df28d5aa654505cc7a84d36392352aad8a4f: Status 404 returned error can't find the container with id 3aae9aba401edc97b62b08b08701df28d5aa654505cc7a84d36392352aad8a4f Oct 30 21:54:11.320121 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:11.320096 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ld2vq"] Oct 30 21:54:11.322557 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:54:11.322521 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod766c56e9_71be_48d2_ae53_3b8cad19ec03.slice/crio-114ed44559861d10d2cdea6fac52afc12570f209ce1f21240ffe922834b87e1a WatchSource:0}: Error finding container 114ed44559861d10d2cdea6fac52afc12570f209ce1f21240ffe922834b87e1a: Status 404 returned error can't find the container with id 114ed44559861d10d2cdea6fac52afc12570f209ce1f21240ffe922834b87e1a Oct 30 21:54:11.834035 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:11.834010 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-744pk" Oct 30 21:54:11.867969 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:11.867940 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-vrjss" event={"ID":"b6dd786f-1bf3-4afc-9d6e-250aecd415bc","Type":"ContainerStarted","Data":"3aae9aba401edc97b62b08b08701df28d5aa654505cc7a84d36392352aad8a4f"} Oct 30 21:54:11.869029 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:11.869007 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-ld2vq" event={"ID":"766c56e9-71be-48d2-ae53-3b8cad19ec03","Type":"ContainerStarted","Data":"114ed44559861d10d2cdea6fac52afc12570f209ce1f21240ffe922834b87e1a"} Oct 30 21:54:13.305118 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:13.305085 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-xm656"] Oct 30 21:54:13.324636 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:13.324608 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xm656" Oct 30 21:54:13.367695 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:13.367660 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-xm656"] Oct 30 21:54:13.415931 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:13.415898 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 30 21:54:13.416109 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:13.415992 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 30 21:54:13.488138 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:13.488105 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/aea8b7e0-9ce6-46b9-8164-314dc39f9114-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-xm656\" (UID: \"aea8b7e0-9ce6-46b9-8164-314dc39f9114\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xm656" Oct 30 21:54:13.488342 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:13.488235 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/aea8b7e0-9ce6-46b9-8164-314dc39f9114-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-xm656\" (UID: \"aea8b7e0-9ce6-46b9-8164-314dc39f9114\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xm656" Oct 30 21:54:13.488342 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:13.488313 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsdj2\" (UniqueName: \"kubernetes.io/projected/aea8b7e0-9ce6-46b9-8164-314dc39f9114-kube-api-access-jsdj2\") pod \"kuadrant-console-plugin-66c4fb575d-xm656\" (UID: \"aea8b7e0-9ce6-46b9-8164-314dc39f9114\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xm656" Oct 30 21:54:13.590666 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:13.589657 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/aea8b7e0-9ce6-46b9-8164-314dc39f9114-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-xm656\" (UID: \"aea8b7e0-9ce6-46b9-8164-314dc39f9114\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xm656" Oct 30 21:54:13.590666 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:13.589744 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsdj2\" (UniqueName: \"kubernetes.io/projected/aea8b7e0-9ce6-46b9-8164-314dc39f9114-kube-api-access-jsdj2\") pod \"kuadrant-console-plugin-66c4fb575d-xm656\" (UID: \"aea8b7e0-9ce6-46b9-8164-314dc39f9114\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xm656" Oct 30 21:54:13.590666 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:13.589783 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/aea8b7e0-9ce6-46b9-8164-314dc39f9114-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-xm656\" (UID: \"aea8b7e0-9ce6-46b9-8164-314dc39f9114\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xm656" Oct 30 21:54:13.592434 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:13.591167 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/aea8b7e0-9ce6-46b9-8164-314dc39f9114-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-xm656\" (UID: \"aea8b7e0-9ce6-46b9-8164-314dc39f9114\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xm656" Oct 30 21:54:13.592987 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:13.592962 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/aea8b7e0-9ce6-46b9-8164-314dc39f9114-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-xm656\" (UID: \"aea8b7e0-9ce6-46b9-8164-314dc39f9114\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xm656" Oct 30 21:54:13.666408 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:13.666377 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsdj2\" (UniqueName: \"kubernetes.io/projected/aea8b7e0-9ce6-46b9-8164-314dc39f9114-kube-api-access-jsdj2\") pod \"kuadrant-console-plugin-66c4fb575d-xm656\" (UID: \"aea8b7e0-9ce6-46b9-8164-314dc39f9114\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xm656" Oct 30 21:54:13.962096 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:13.962061 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xm656" Oct 30 21:54:14.350024 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:14.349990 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-xm656"] Oct 30 21:54:14.352669 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:54:14.352639 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaea8b7e0_9ce6_46b9_8164_314dc39f9114.slice/crio-cd7bf21f36849f7f6f821f16ff4ec87f4a15eb52afbacab669528f20f916be93 WatchSource:0}: Error finding container cd7bf21f36849f7f6f821f16ff4ec87f4a15eb52afbacab669528f20f916be93: Status 404 returned error can't find the container with id cd7bf21f36849f7f6f821f16ff4ec87f4a15eb52afbacab669528f20f916be93 Oct 30 21:54:14.895078 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:14.895046 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xm656" event={"ID":"aea8b7e0-9ce6-46b9-8164-314dc39f9114","Type":"ContainerStarted","Data":"cd7bf21f36849f7f6f821f16ff4ec87f4a15eb52afbacab669528f20f916be93"} Oct 30 21:54:15.901060 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:15.900981 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-ld2vq" event={"ID":"766c56e9-71be-48d2-ae53-3b8cad19ec03","Type":"ContainerStarted","Data":"2dad93f88bdd862e557725f58a646ee93f3adc176fd92edfb1111469de9dd5e6"} Oct 30 21:54:15.902690 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:15.902665 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-vrjss" event={"ID":"b6dd786f-1bf3-4afc-9d6e-250aecd415bc","Type":"ContainerStarted","Data":"b909e74ba6b77f01e53198ae58a9a0ef487abc888fd80164e413af39f626541a"} Oct 30 21:54:15.902835 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:15.902720 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-vrjss" Oct 30 21:54:16.019269 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:16.019207 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-ld2vq" podStartSLOduration=1.851519385 podStartE2EDuration="6.019193036s" totalImagesPullingTime="4.167673651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:16.01885345 +0000 UTC m=+216.289551529" watchObservedRunningTime="2026-10-30 21:54:16.019193036 +0000 UTC m=+216.289891120" Oct 30 21:54:16.140407 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:16.140350 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-vrjss" podStartSLOduration=2.564109868 podStartE2EDuration="6.140337231s" totalImagesPullingTime="3.576227363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:16.138952654 +0000 UTC m=+216.409650740" watchObservedRunningTime="2026-10-30 21:54:16.140337231 +0000 UTC m=+216.411035315" Oct 30 21:54:17.533607 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:17.533565 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ld2vq"] Oct 30 21:54:17.912984 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:17.912905 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-ld2vq" podUID="766c56e9-71be-48d2-ae53-3b8cad19ec03" containerName="authorino" containerID="cri-o://2dad93f88bdd862e557725f58a646ee93f3adc176fd92edfb1111469de9dd5e6" gracePeriod=30 Oct 30 21:54:18.187572 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:18.187546 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-ld2vq" Oct 30 21:54:18.239681 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:18.239652 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/766c56e9-71be-48d2-ae53-3b8cad19ec03-kube-api-access-rznbc\" (UniqueName: \"kubernetes.io/projected/766c56e9-71be-48d2-ae53-3b8cad19ec03-kube-api-access-rznbc\") pod \"766c56e9-71be-48d2-ae53-3b8cad19ec03\" (UID: \"766c56e9-71be-48d2-ae53-3b8cad19ec03\") " Oct 30 21:54:18.242341 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:18.242305 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/766c56e9-71be-48d2-ae53-3b8cad19ec03-kube-api-access-rznbc" pod "766c56e9-71be-48d2-ae53-3b8cad19ec03" (UID: "766c56e9-71be-48d2-ae53-3b8cad19ec03"). InnerVolumeSpecName "kube-api-access-rznbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:18.340901 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:18.340869 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rznbc\" (UniqueName: \"kubernetes.io/projected/766c56e9-71be-48d2-ae53-3b8cad19ec03-kube-api-access-rznbc\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:18.919748 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:18.919716 5918 generic.go:356] "Generic (PLEG): container finished" podID="766c56e9-71be-48d2-ae53-3b8cad19ec03" containerID="2dad93f88bdd862e557725f58a646ee93f3adc176fd92edfb1111469de9dd5e6" exitCode=0 Oct 30 21:54:18.920261 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:18.919781 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-ld2vq" Oct 30 21:54:18.920261 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:18.919795 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-ld2vq" event={"ID":"766c56e9-71be-48d2-ae53-3b8cad19ec03","Type":"ContainerDied","Data":"2dad93f88bdd862e557725f58a646ee93f3adc176fd92edfb1111469de9dd5e6"} Oct 30 21:54:18.920261 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:18.919835 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-ld2vq" event={"ID":"766c56e9-71be-48d2-ae53-3b8cad19ec03","Type":"ContainerDied","Data":"114ed44559861d10d2cdea6fac52afc12570f209ce1f21240ffe922834b87e1a"} Oct 30 21:54:18.920261 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:18.919859 5918 scope.go:122] "RemoveContainer" containerID="2dad93f88bdd862e557725f58a646ee93f3adc176fd92edfb1111469de9dd5e6" Oct 30 21:54:18.930860 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:18.930833 5918 scope.go:122] "RemoveContainer" containerID="2dad93f88bdd862e557725f58a646ee93f3adc176fd92edfb1111469de9dd5e6" Oct 30 21:54:18.931144 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:54:18.931123 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dad93f88bdd862e557725f58a646ee93f3adc176fd92edfb1111469de9dd5e6\": container with ID starting with 2dad93f88bdd862e557725f58a646ee93f3adc176fd92edfb1111469de9dd5e6 not found: ID does not exist" containerID="2dad93f88bdd862e557725f58a646ee93f3adc176fd92edfb1111469de9dd5e6" Oct 30 21:54:18.931254 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:18.931150 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dad93f88bdd862e557725f58a646ee93f3adc176fd92edfb1111469de9dd5e6"} err="failed to get container status \"2dad93f88bdd862e557725f58a646ee93f3adc176fd92edfb1111469de9dd5e6\": rpc error: code = NotFound desc = could not find container \"2dad93f88bdd862e557725f58a646ee93f3adc176fd92edfb1111469de9dd5e6\": container with ID starting with 2dad93f88bdd862e557725f58a646ee93f3adc176fd92edfb1111469de9dd5e6 not found: ID does not exist" Oct 30 21:54:19.095458 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:19.095425 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ld2vq"] Oct 30 21:54:19.101042 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:19.101014 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ld2vq"] Oct 30 21:54:19.146055 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:19.146026 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-fvsvn"] Oct 30 21:54:19.146583 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:19.146560 5918 cpu_manager.go:414] "containerMap: removing container" podUID="766c56e9-71be-48d2-ae53-3b8cad19ec03" containerName="authorino" Oct 30 21:54:19.146583 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:19.146583 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="766c56e9-71be-48d2-ae53-3b8cad19ec03" containerName="authorino" Oct 30 21:54:19.146806 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:19.146695 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="766c56e9-71be-48d2-ae53-3b8cad19ec03" containerName="authorino" Oct 30 21:54:19.163528 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:19.163499 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-fvsvn" Oct 30 21:54:19.208903 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:19.208876 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-fvsvn"] Oct 30 21:54:19.217729 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:19.217706 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-sd5gt\"" Oct 30 21:54:19.248913 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:19.248879 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqlrc\" (UniqueName: \"kubernetes.io/projected/653b4478-e68e-49af-af12-4516e065567d-kube-api-access-lqlrc\") pod \"cert-manager-597b95cb84-fvsvn\" (UID: \"653b4478-e68e-49af-af12-4516e065567d\") " pod="cert-manager/cert-manager-597b95cb84-fvsvn" Oct 30 21:54:19.249093 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:19.249032 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/653b4478-e68e-49af-af12-4516e065567d-bound-sa-token\") pod \"cert-manager-597b95cb84-fvsvn\" (UID: \"653b4478-e68e-49af-af12-4516e065567d\") " pod="cert-manager/cert-manager-597b95cb84-fvsvn" Oct 30 21:54:19.350183 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:19.350146 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/653b4478-e68e-49af-af12-4516e065567d-bound-sa-token\") pod \"cert-manager-597b95cb84-fvsvn\" (UID: \"653b4478-e68e-49af-af12-4516e065567d\") " pod="cert-manager/cert-manager-597b95cb84-fvsvn" Oct 30 21:54:19.350364 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:19.350210 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqlrc\" (UniqueName: \"kubernetes.io/projected/653b4478-e68e-49af-af12-4516e065567d-kube-api-access-lqlrc\") pod \"cert-manager-597b95cb84-fvsvn\" (UID: \"653b4478-e68e-49af-af12-4516e065567d\") " pod="cert-manager/cert-manager-597b95cb84-fvsvn" Oct 30 21:54:19.414503 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:19.414472 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/653b4478-e68e-49af-af12-4516e065567d-bound-sa-token\") pod \"cert-manager-597b95cb84-fvsvn\" (UID: \"653b4478-e68e-49af-af12-4516e065567d\") " pod="cert-manager/cert-manager-597b95cb84-fvsvn" Oct 30 21:54:19.414683 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:19.414620 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqlrc\" (UniqueName: \"kubernetes.io/projected/653b4478-e68e-49af-af12-4516e065567d-kube-api-access-lqlrc\") pod \"cert-manager-597b95cb84-fvsvn\" (UID: \"653b4478-e68e-49af-af12-4516e065567d\") " pod="cert-manager/cert-manager-597b95cb84-fvsvn" Oct 30 21:54:19.478900 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:19.478864 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-fvsvn" Oct 30 21:54:19.851275 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:19.851180 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="766c56e9-71be-48d2-ae53-3b8cad19ec03" path="/var/lib/kubelet/pods/766c56e9-71be-48d2-ae53-3b8cad19ec03/volumes" Oct 30 21:54:19.863423 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:19.863399 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-fvsvn"] Oct 30 21:54:19.866299 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:54:19.866261 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod653b4478_e68e_49af_af12_4516e065567d.slice/crio-6935099bcda322002d9cc71865237d3419b4051c6f86cc658f71360b21689256 WatchSource:0}: Error finding container 6935099bcda322002d9cc71865237d3419b4051c6f86cc658f71360b21689256: Status 404 returned error can't find the container with id 6935099bcda322002d9cc71865237d3419b4051c6f86cc658f71360b21689256 Oct 30 21:54:19.927485 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:19.927444 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-fvsvn" event={"ID":"653b4478-e68e-49af-af12-4516e065567d","Type":"ContainerStarted","Data":"6935099bcda322002d9cc71865237d3419b4051c6f86cc658f71360b21689256"} Oct 30 21:54:20.933790 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:20.933749 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-fvsvn" event={"ID":"653b4478-e68e-49af-af12-4516e065567d","Type":"ContainerStarted","Data":"8aae7c6f73d138db58e78aadd5d790162ef50be74c3d71fd01031ac377b3c703"} Oct 30 21:54:20.935375 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:20.935350 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xm656" event={"ID":"aea8b7e0-9ce6-46b9-8164-314dc39f9114","Type":"ContainerStarted","Data":"cb50a5fd5f0884ad54023f1a2cb0d6c3f6547ea485388f4d62e7e3f15439f3c8"} Oct 30 21:54:21.053081 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:21.053025 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-fvsvn" podStartSLOduration=2.053011452 podStartE2EDuration="2.053011452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:21.052094731 +0000 UTC m=+221.322792827" watchObservedRunningTime="2026-10-30 21:54:21.053011452 +0000 UTC m=+221.323709530" Oct 30 21:54:21.172082 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:21.172034 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xm656" podStartSLOduration=2.240663959 podStartE2EDuration="8.172021723s" totalImagesPullingTime="5.931357764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:21.171581876 +0000 UTC m=+221.442279959" watchObservedRunningTime="2026-10-30 21:54:21.172021723 +0000 UTC m=+221.442719807" Oct 30 21:54:21.273964 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:54:21.273880 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 21:54:26.908374 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:26.908344 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-vrjss" Oct 30 21:54:27.306178 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:27.306143 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vrjss"] Oct 30 21:54:27.306385 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:27.306349 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-vrjss" podUID="b6dd786f-1bf3-4afc-9d6e-250aecd415bc" containerName="limitador" containerID="cri-o://b909e74ba6b77f01e53198ae58a9a0ef487abc888fd80164e413af39f626541a" gracePeriod=30 Oct 30 21:54:27.853686 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:27.853661 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-vrjss" Oct 30 21:54:27.936077 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:27.936057 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6dd786f-1bf3-4afc-9d6e-250aecd415bc-config-file\" (UniqueName: \"kubernetes.io/configmap/b6dd786f-1bf3-4afc-9d6e-250aecd415bc-config-file\") pod \"b6dd786f-1bf3-4afc-9d6e-250aecd415bc\" (UID: \"b6dd786f-1bf3-4afc-9d6e-250aecd415bc\") " Oct 30 21:54:27.936459 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:27.936127 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6dd786f-1bf3-4afc-9d6e-250aecd415bc-kube-api-access-kglpg\" (UniqueName: \"kubernetes.io/projected/b6dd786f-1bf3-4afc-9d6e-250aecd415bc-kube-api-access-kglpg\") pod \"b6dd786f-1bf3-4afc-9d6e-250aecd415bc\" (UID: \"b6dd786f-1bf3-4afc-9d6e-250aecd415bc\") " Oct 30 21:54:27.936459 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:27.936423 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6dd786f-1bf3-4afc-9d6e-250aecd415bc-config-file" pod "b6dd786f-1bf3-4afc-9d6e-250aecd415bc" (UID: "b6dd786f-1bf3-4afc-9d6e-250aecd415bc"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:27.938399 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:27.938376 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6dd786f-1bf3-4afc-9d6e-250aecd415bc-kube-api-access-kglpg" pod "b6dd786f-1bf3-4afc-9d6e-250aecd415bc" (UID: "b6dd786f-1bf3-4afc-9d6e-250aecd415bc"). InnerVolumeSpecName "kube-api-access-kglpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:27.967393 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:27.967370 5918 generic.go:356] "Generic (PLEG): container finished" podID="b6dd786f-1bf3-4afc-9d6e-250aecd415bc" containerID="b909e74ba6b77f01e53198ae58a9a0ef487abc888fd80164e413af39f626541a" exitCode=0 Oct 30 21:54:27.967479 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:27.967418 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-vrjss" Oct 30 21:54:27.967516 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:27.967417 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-vrjss" event={"ID":"b6dd786f-1bf3-4afc-9d6e-250aecd415bc","Type":"ContainerDied","Data":"b909e74ba6b77f01e53198ae58a9a0ef487abc888fd80164e413af39f626541a"} Oct 30 21:54:27.967570 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:27.967555 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-vrjss" event={"ID":"b6dd786f-1bf3-4afc-9d6e-250aecd415bc","Type":"ContainerDied","Data":"3aae9aba401edc97b62b08b08701df28d5aa654505cc7a84d36392352aad8a4f"} Oct 30 21:54:27.967607 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:27.967585 5918 scope.go:122] "RemoveContainer" containerID="b909e74ba6b77f01e53198ae58a9a0ef487abc888fd80164e413af39f626541a" Oct 30 21:54:27.977116 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:27.977100 5918 scope.go:122] "RemoveContainer" containerID="b909e74ba6b77f01e53198ae58a9a0ef487abc888fd80164e413af39f626541a" Oct 30 21:54:27.977425 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:54:27.977407 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b909e74ba6b77f01e53198ae58a9a0ef487abc888fd80164e413af39f626541a\": container with ID starting with b909e74ba6b77f01e53198ae58a9a0ef487abc888fd80164e413af39f626541a not found: ID does not exist" containerID="b909e74ba6b77f01e53198ae58a9a0ef487abc888fd80164e413af39f626541a" Oct 30 21:54:27.977488 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:27.977434 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b909e74ba6b77f01e53198ae58a9a0ef487abc888fd80164e413af39f626541a"} err="failed to get container status \"b909e74ba6b77f01e53198ae58a9a0ef487abc888fd80164e413af39f626541a\": rpc error: code = NotFound desc = could not find container \"b909e74ba6b77f01e53198ae58a9a0ef487abc888fd80164e413af39f626541a\": container with ID starting with b909e74ba6b77f01e53198ae58a9a0ef487abc888fd80164e413af39f626541a not found: ID does not exist" Oct 30 21:54:28.037378 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:28.037355 5918 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b6dd786f-1bf3-4afc-9d6e-250aecd415bc-config-file\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:28.037378 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:28.037377 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kglpg\" (UniqueName: \"kubernetes.io/projected/b6dd786f-1bf3-4afc-9d6e-250aecd415bc-kube-api-access-kglpg\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:28.140855 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:28.140825 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vrjss"] Oct 30 21:54:28.151787 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:28.151759 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vrjss"] Oct 30 21:54:29.849401 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:29.849372 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6dd786f-1bf3-4afc-9d6e-250aecd415bc" path="/var/lib/kubelet/pods/b6dd786f-1bf3-4afc-9d6e-250aecd415bc/volumes" Oct 30 21:54:35.949667 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:35.949634 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-b2lk7"] Oct 30 21:54:35.950060 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:35.949988 5918 cpu_manager.go:414] "containerMap: removing container" podUID="b6dd786f-1bf3-4afc-9d6e-250aecd415bc" containerName="limitador" Oct 30 21:54:35.950060 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:35.949998 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6dd786f-1bf3-4afc-9d6e-250aecd415bc" containerName="limitador" Oct 30 21:54:35.950130 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:35.950092 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6dd786f-1bf3-4afc-9d6e-250aecd415bc" containerName="limitador" Oct 30 21:54:35.953940 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:35.953919 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-b2lk7" Oct 30 21:54:36.008717 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:36.008687 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 21:54:36.008849 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:36.008724 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 21:54:36.015242 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:36.015201 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbpq4\" (UniqueName: \"kubernetes.io/projected/aef35927-4a0c-445c-8579-61c28d79e02b-kube-api-access-hbpq4\") pod \"kserve-module-controller-manager-7566cf5fc8-b2lk7\" (UID: \"aef35927-4a0c-445c-8579-61c28d79e02b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-b2lk7" Oct 30 21:54:36.015304 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:36.015263 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aef35927-4a0c-445c-8579-61c28d79e02b-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-b2lk7\" (UID: \"aef35927-4a0c-445c-8579-61c28d79e02b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-b2lk7" Oct 30 21:54:36.015352 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:36.015335 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aef35927-4a0c-445c-8579-61c28d79e02b-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-b2lk7\" (UID: \"aef35927-4a0c-445c-8579-61c28d79e02b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-b2lk7" Oct 30 21:54:36.015395 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:36.015365 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/aef35927-4a0c-445c-8579-61c28d79e02b-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-b2lk7\" (UID: \"aef35927-4a0c-445c-8579-61c28d79e02b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-b2lk7" Oct 30 21:54:36.116870 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:36.116839 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbpq4\" (UniqueName: \"kubernetes.io/projected/aef35927-4a0c-445c-8579-61c28d79e02b-kube-api-access-hbpq4\") pod \"kserve-module-controller-manager-7566cf5fc8-b2lk7\" (UID: \"aef35927-4a0c-445c-8579-61c28d79e02b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-b2lk7" Oct 30 21:54:36.117032 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:36.116883 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aef35927-4a0c-445c-8579-61c28d79e02b-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-b2lk7\" (UID: \"aef35927-4a0c-445c-8579-61c28d79e02b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-b2lk7" Oct 30 21:54:36.117032 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:36.116909 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aef35927-4a0c-445c-8579-61c28d79e02b-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-b2lk7\" (UID: \"aef35927-4a0c-445c-8579-61c28d79e02b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-b2lk7" Oct 30 21:54:36.117032 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:36.116930 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/aef35927-4a0c-445c-8579-61c28d79e02b-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-b2lk7\" (UID: \"aef35927-4a0c-445c-8579-61c28d79e02b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-b2lk7" Oct 30 21:54:36.117310 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:36.117293 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aef35927-4a0c-445c-8579-61c28d79e02b-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-b2lk7\" (UID: \"aef35927-4a0c-445c-8579-61c28d79e02b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-b2lk7" Oct 30 21:54:36.117358 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:36.117325 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/aef35927-4a0c-445c-8579-61c28d79e02b-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-b2lk7\" (UID: \"aef35927-4a0c-445c-8579-61c28d79e02b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-b2lk7" Oct 30 21:54:36.160416 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:36.160388 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-nqbr9\"" Oct 30 21:54:36.160760 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:36.160742 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 21:54:36.162151 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:36.162116 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-b2lk7"] Oct 30 21:54:36.168801 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:36.168783 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aef35927-4a0c-445c-8579-61c28d79e02b-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-b2lk7\" (UID: \"aef35927-4a0c-445c-8579-61c28d79e02b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-b2lk7" Oct 30 21:54:36.182204 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:36.182173 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbpq4\" (UniqueName: \"kubernetes.io/projected/aef35927-4a0c-445c-8579-61c28d79e02b-kube-api-access-hbpq4\") pod \"kserve-module-controller-manager-7566cf5fc8-b2lk7\" (UID: \"aef35927-4a0c-445c-8579-61c28d79e02b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-b2lk7" Oct 30 21:54:36.265199 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:36.265109 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-b2lk7" Oct 30 21:54:36.699911 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:36.699887 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-b2lk7"] Oct 30 21:54:36.702791 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:54:36.702767 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaef35927_4a0c_445c_8579_61c28d79e02b.slice/crio-f54b66ee15fce5a91469be8c189911f57f24571efb2762574ad93c1ddbd7ce6c WatchSource:0}: Error finding container f54b66ee15fce5a91469be8c189911f57f24571efb2762574ad93c1ddbd7ce6c: Status 404 returned error can't find the container with id f54b66ee15fce5a91469be8c189911f57f24571efb2762574ad93c1ddbd7ce6c Oct 30 21:54:37.010403 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:37.010316 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-b2lk7" event={"ID":"aef35927-4a0c-445c-8579-61c28d79e02b","Type":"ContainerStarted","Data":"f54b66ee15fce5a91469be8c189911f57f24571efb2762574ad93c1ddbd7ce6c"} Oct 30 21:54:40.026323 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:40.026285 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-b2lk7" event={"ID":"aef35927-4a0c-445c-8579-61c28d79e02b","Type":"ContainerStarted","Data":"d88efed1d0c8832f716bd3c9142d0d9aee5f74f60db6ac64fe986d9bfafb840f"} Oct 30 21:54:40.026717 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:40.026442 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-b2lk7" Oct 30 21:54:40.199793 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:54:40.199747 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-b2lk7" podStartSLOduration=2.93820238 podStartE2EDuration="5.199731569s" totalImagesPullingTime="2.261529189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:40.198498265 +0000 UTC m=+240.469196352" watchObservedRunningTime="2026-10-30 21:54:40.199731569 +0000 UTC m=+240.470429650" Oct 30 21:54:51.273853 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:54:51.273820 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 21:55:01.032669 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:01.032638 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-b2lk7" Oct 30 21:55:21.274168 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:55:21.274091 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 21:55:42.414016 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:42.413986 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-rqjfj"] Oct 30 21:55:42.418642 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:42.418624 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-rqjfj" Oct 30 21:55:42.481796 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:42.481771 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-nvdbh\"" Oct 30 21:55:42.481796 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:42.481795 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 30 21:55:42.521987 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:42.521954 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4rr6\" (UniqueName: \"kubernetes.io/projected/7e8c5776-206b-4be6-8f5b-4b3c26c6da52-kube-api-access-g4rr6\") pod \"kserve-controller-manager-776cc66d47-rqjfj\" (UID: \"7e8c5776-206b-4be6-8f5b-4b3c26c6da52\") " pod="opendatahub/kserve-controller-manager-776cc66d47-rqjfj" Oct 30 21:55:42.522130 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:42.522066 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7e8c5776-206b-4be6-8f5b-4b3c26c6da52-cert\") pod \"kserve-controller-manager-776cc66d47-rqjfj\" (UID: \"7e8c5776-206b-4be6-8f5b-4b3c26c6da52\") " pod="opendatahub/kserve-controller-manager-776cc66d47-rqjfj" Oct 30 21:55:42.524308 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:42.524286 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-rqjfj"] Oct 30 21:55:42.525306 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:42.525288 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-dq7zs"] Oct 30 21:55:42.527804 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:42.527790 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dq7zs" Oct 30 21:55:42.583720 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:42.583687 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 21:55:42.583861 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:42.583755 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-2gfsw\"" Oct 30 21:55:42.584813 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:42.584793 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-dq7zs"] Oct 30 21:55:42.622604 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:42.622576 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7e8c5776-206b-4be6-8f5b-4b3c26c6da52-cert\") pod \"kserve-controller-manager-776cc66d47-rqjfj\" (UID: \"7e8c5776-206b-4be6-8f5b-4b3c26c6da52\") " pod="opendatahub/kserve-controller-manager-776cc66d47-rqjfj" Oct 30 21:55:42.622728 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:42.622633 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/728bdf4f-f9d5-420b-bb7b-7f7297a67ff7-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-dq7zs\" (UID: \"728bdf4f-f9d5-420b-bb7b-7f7297a67ff7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dq7zs" Oct 30 21:55:42.622728 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:42.622661 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzcbz\" (UniqueName: \"kubernetes.io/projected/728bdf4f-f9d5-420b-bb7b-7f7297a67ff7-kube-api-access-kzcbz\") pod \"llmisvc-controller-manager-869d5dcc5d-dq7zs\" (UID: \"728bdf4f-f9d5-420b-bb7b-7f7297a67ff7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dq7zs" Oct 30 21:55:42.622834 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:42.622815 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4rr6\" (UniqueName: \"kubernetes.io/projected/7e8c5776-206b-4be6-8f5b-4b3c26c6da52-kube-api-access-g4rr6\") pod \"kserve-controller-manager-776cc66d47-rqjfj\" (UID: \"7e8c5776-206b-4be6-8f5b-4b3c26c6da52\") " pod="opendatahub/kserve-controller-manager-776cc66d47-rqjfj" Oct 30 21:55:42.625345 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:42.625323 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7e8c5776-206b-4be6-8f5b-4b3c26c6da52-cert\") pod \"kserve-controller-manager-776cc66d47-rqjfj\" (UID: \"7e8c5776-206b-4be6-8f5b-4b3c26c6da52\") " pod="opendatahub/kserve-controller-manager-776cc66d47-rqjfj" Oct 30 21:55:42.685618 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:42.685590 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4rr6\" (UniqueName: \"kubernetes.io/projected/7e8c5776-206b-4be6-8f5b-4b3c26c6da52-kube-api-access-g4rr6\") pod \"kserve-controller-manager-776cc66d47-rqjfj\" (UID: \"7e8c5776-206b-4be6-8f5b-4b3c26c6da52\") " pod="opendatahub/kserve-controller-manager-776cc66d47-rqjfj" Oct 30 21:55:42.723789 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:42.723764 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/728bdf4f-f9d5-420b-bb7b-7f7297a67ff7-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-dq7zs\" (UID: \"728bdf4f-f9d5-420b-bb7b-7f7297a67ff7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dq7zs" Oct 30 21:55:42.723931 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:42.723795 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzcbz\" (UniqueName: \"kubernetes.io/projected/728bdf4f-f9d5-420b-bb7b-7f7297a67ff7-kube-api-access-kzcbz\") pod \"llmisvc-controller-manager-869d5dcc5d-dq7zs\" (UID: \"728bdf4f-f9d5-420b-bb7b-7f7297a67ff7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dq7zs" Oct 30 21:55:42.726181 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:42.726158 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/728bdf4f-f9d5-420b-bb7b-7f7297a67ff7-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-dq7zs\" (UID: \"728bdf4f-f9d5-420b-bb7b-7f7297a67ff7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dq7zs" Oct 30 21:55:42.730069 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:42.730051 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-rqjfj" Oct 30 21:55:42.790128 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:42.790098 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzcbz\" (UniqueName: \"kubernetes.io/projected/728bdf4f-f9d5-420b-bb7b-7f7297a67ff7-kube-api-access-kzcbz\") pod \"llmisvc-controller-manager-869d5dcc5d-dq7zs\" (UID: \"728bdf4f-f9d5-420b-bb7b-7f7297a67ff7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dq7zs" Oct 30 21:55:42.839154 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:42.839121 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dq7zs" Oct 30 21:55:43.110998 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:43.110954 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-rqjfj"] Oct 30 21:55:43.113330 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:55:43.113300 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e8c5776_206b_4be6_8f5b_4b3c26c6da52.slice/crio-2c5bf95e02bef6b7d5363b7d82daa1ed7bdf43c332aaaa1e63d6804cfba00c8f WatchSource:0}: Error finding container 2c5bf95e02bef6b7d5363b7d82daa1ed7bdf43c332aaaa1e63d6804cfba00c8f: Status 404 returned error can't find the container with id 2c5bf95e02bef6b7d5363b7d82daa1ed7bdf43c332aaaa1e63d6804cfba00c8f Oct 30 21:55:43.211899 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:43.211875 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-dq7zs"] Oct 30 21:55:43.213334 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:55:43.213299 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod728bdf4f_f9d5_420b_bb7b_7f7297a67ff7.slice/crio-251c1c19b282d4cc9b16f766b90e9c939b0033a585c1caa10793e04f5d832148 WatchSource:0}: Error finding container 251c1c19b282d4cc9b16f766b90e9c939b0033a585c1caa10793e04f5d832148: Status 404 returned error can't find the container with id 251c1c19b282d4cc9b16f766b90e9c939b0033a585c1caa10793e04f5d832148 Oct 30 21:55:43.318162 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:43.318124 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-rqjfj" event={"ID":"7e8c5776-206b-4be6-8f5b-4b3c26c6da52","Type":"ContainerStarted","Data":"2c5bf95e02bef6b7d5363b7d82daa1ed7bdf43c332aaaa1e63d6804cfba00c8f"} Oct 30 21:55:43.319299 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:43.319257 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dq7zs" event={"ID":"728bdf4f-f9d5-420b-bb7b-7f7297a67ff7","Type":"ContainerStarted","Data":"251c1c19b282d4cc9b16f766b90e9c939b0033a585c1caa10793e04f5d832148"} Oct 30 21:55:45.872771 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:45.872732 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-57fcf7d945-4fsw9"] Oct 30 21:55:45.876611 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:45.876587 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57fcf7d945-4fsw9" Oct 30 21:55:45.956765 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:45.956730 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96d23985-308b-44bd-928f-0563b9d19ff9-metrics-certs\") pod \"maas-api-57fcf7d945-4fsw9\" (UID: \"96d23985-308b-44bd-928f-0563b9d19ff9\") " pod="odh-ai-gateway-infra/maas-api-57fcf7d945-4fsw9" Oct 30 21:55:45.956934 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:45.956846 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d23985-308b-44bd-928f-0563b9d19ff9-openshift-service-ca\") pod \"maas-api-57fcf7d945-4fsw9\" (UID: \"96d23985-308b-44bd-928f-0563b9d19ff9\") " pod="odh-ai-gateway-infra/maas-api-57fcf7d945-4fsw9" Oct 30 21:55:45.956994 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:45.956956 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t95bf\" (UniqueName: \"kubernetes.io/projected/96d23985-308b-44bd-928f-0563b9d19ff9-kube-api-access-t95bf\") pod \"maas-api-57fcf7d945-4fsw9\" (UID: \"96d23985-308b-44bd-928f-0563b9d19ff9\") " pod="odh-ai-gateway-infra/maas-api-57fcf7d945-4fsw9" Oct 30 21:55:45.957039 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:45.957005 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96d23985-308b-44bd-928f-0563b9d19ff9-maas-api-tls\") pod \"maas-api-57fcf7d945-4fsw9\" (UID: \"96d23985-308b-44bd-928f-0563b9d19ff9\") " pod="odh-ai-gateway-infra/maas-api-57fcf7d945-4fsw9" Oct 30 21:55:45.967310 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:45.967283 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:55:45.981405 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:45.981321 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:55:45.981550 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:45.981331 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 21:55:45.981550 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:45.981546 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lxv7t\"" Oct 30 21:55:45.981652 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:45.981563 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 21:55:45.983112 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:45.983091 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-57fcf7d945-4fsw9"] Oct 30 21:55:46.058364 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:46.058294 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t95bf\" (UniqueName: \"kubernetes.io/projected/96d23985-308b-44bd-928f-0563b9d19ff9-kube-api-access-t95bf\") pod \"maas-api-57fcf7d945-4fsw9\" (UID: \"96d23985-308b-44bd-928f-0563b9d19ff9\") " pod="odh-ai-gateway-infra/maas-api-57fcf7d945-4fsw9" Oct 30 21:55:46.058514 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:46.058362 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96d23985-308b-44bd-928f-0563b9d19ff9-maas-api-tls\") pod \"maas-api-57fcf7d945-4fsw9\" (UID: \"96d23985-308b-44bd-928f-0563b9d19ff9\") " pod="odh-ai-gateway-infra/maas-api-57fcf7d945-4fsw9" Oct 30 21:55:46.058514 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:46.058461 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96d23985-308b-44bd-928f-0563b9d19ff9-metrics-certs\") pod \"maas-api-57fcf7d945-4fsw9\" (UID: \"96d23985-308b-44bd-928f-0563b9d19ff9\") " pod="odh-ai-gateway-infra/maas-api-57fcf7d945-4fsw9" Oct 30 21:55:46.058514 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:46.058498 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d23985-308b-44bd-928f-0563b9d19ff9-openshift-service-ca\") pod \"maas-api-57fcf7d945-4fsw9\" (UID: \"96d23985-308b-44bd-928f-0563b9d19ff9\") " pod="odh-ai-gateway-infra/maas-api-57fcf7d945-4fsw9" Oct 30 21:55:46.059049 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:46.059026 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d23985-308b-44bd-928f-0563b9d19ff9-openshift-service-ca\") pod \"maas-api-57fcf7d945-4fsw9\" (UID: \"96d23985-308b-44bd-928f-0563b9d19ff9\") " pod="odh-ai-gateway-infra/maas-api-57fcf7d945-4fsw9" Oct 30 21:55:46.060841 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:46.060817 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96d23985-308b-44bd-928f-0563b9d19ff9-maas-api-tls\") pod \"maas-api-57fcf7d945-4fsw9\" (UID: \"96d23985-308b-44bd-928f-0563b9d19ff9\") " pod="odh-ai-gateway-infra/maas-api-57fcf7d945-4fsw9" Oct 30 21:55:46.061139 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:46.061113 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96d23985-308b-44bd-928f-0563b9d19ff9-metrics-certs\") pod \"maas-api-57fcf7d945-4fsw9\" (UID: \"96d23985-308b-44bd-928f-0563b9d19ff9\") " pod="odh-ai-gateway-infra/maas-api-57fcf7d945-4fsw9" Oct 30 21:55:46.121749 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:46.121722 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t95bf\" (UniqueName: \"kubernetes.io/projected/96d23985-308b-44bd-928f-0563b9d19ff9-kube-api-access-t95bf\") pod \"maas-api-57fcf7d945-4fsw9\" (UID: \"96d23985-308b-44bd-928f-0563b9d19ff9\") " pod="odh-ai-gateway-infra/maas-api-57fcf7d945-4fsw9" Oct 30 21:55:46.193012 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:46.192974 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57fcf7d945-4fsw9" Oct 30 21:55:46.337134 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:46.337032 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-rqjfj" event={"ID":"7e8c5776-206b-4be6-8f5b-4b3c26c6da52","Type":"ContainerStarted","Data":"9b13ac04ce8938312916ad484a232ca948854929638d00060b6c4c708ae47694"} Oct 30 21:55:46.337134 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:46.337096 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-rqjfj" Oct 30 21:55:46.338738 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:46.338716 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dq7zs" event={"ID":"728bdf4f-f9d5-420b-bb7b-7f7297a67ff7","Type":"ContainerStarted","Data":"56febc75b0f46e4de774a1e08bad1f8bbb1f11c0e8c04c7c4e094218f48cd991"} Oct 30 21:55:46.338865 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:46.338850 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dq7zs" Oct 30 21:55:46.456660 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:46.456616 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-rqjfj" podStartSLOduration=1.497343052 podStartE2EDuration="4.456602692s" totalImagesPullingTime="2.95925964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:46.456407611 +0000 UTC m=+306.727105696" watchObservedRunningTime="2026-10-30 21:55:46.456602692 +0000 UTC m=+306.727300775" Oct 30 21:55:46.544003 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:46.543975 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-65cf7699d5-s4lgf"] Oct 30 21:55:46.548536 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:46.548513 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65cf7699d5-s4lgf" Oct 30 21:55:46.576839 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:46.576792 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dq7zs" podStartSLOduration=2.044998798 podStartE2EDuration="4.57677957s" totalImagesPullingTime="2.531780772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:46.575348161 +0000 UTC m=+306.846046247" watchObservedRunningTime="2026-10-30 21:55:46.57677957 +0000 UTC m=+306.847477654" Oct 30 21:55:46.605736 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:46.605664 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 21:55:46.605736 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:46.605713 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-86hk6\"" Oct 30 21:55:46.607298 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:46.607273 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65cf7699d5-s4lgf"] Oct 30 21:55:46.613574 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:46.613551 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-57fcf7d945-4fsw9"] Oct 30 21:55:46.615745 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:55:46.615716 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d23985_308b_44bd_928f_0563b9d19ff9.slice/crio-ac650c0d135b44bc3b94dd42f59b624744bacf79262d40992a9702b0f297e9d3 WatchSource:0}: Error finding container ac650c0d135b44bc3b94dd42f59b624744bacf79262d40992a9702b0f297e9d3: Status 404 returned error can't find the container with id ac650c0d135b44bc3b94dd42f59b624744bacf79262d40992a9702b0f297e9d3 Oct 30 21:55:46.663363 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:46.663334 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e797ed5b-bb45-4228-915c-20e4377a5911-tls-cert\") pod \"authorino-65cf7699d5-s4lgf\" (UID: \"e797ed5b-bb45-4228-915c-20e4377a5911\") " pod="kuadrant-system/authorino-65cf7699d5-s4lgf" Oct 30 21:55:46.663481 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:46.663385 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtjg5\" (UniqueName: \"kubernetes.io/projected/e797ed5b-bb45-4228-915c-20e4377a5911-kube-api-access-mtjg5\") pod \"authorino-65cf7699d5-s4lgf\" (UID: \"e797ed5b-bb45-4228-915c-20e4377a5911\") " pod="kuadrant-system/authorino-65cf7699d5-s4lgf" Oct 30 21:55:46.764633 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:46.764603 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e797ed5b-bb45-4228-915c-20e4377a5911-tls-cert\") pod \"authorino-65cf7699d5-s4lgf\" (UID: \"e797ed5b-bb45-4228-915c-20e4377a5911\") " pod="kuadrant-system/authorino-65cf7699d5-s4lgf" Oct 30 21:55:46.764778 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:46.764687 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtjg5\" (UniqueName: \"kubernetes.io/projected/e797ed5b-bb45-4228-915c-20e4377a5911-kube-api-access-mtjg5\") pod \"authorino-65cf7699d5-s4lgf\" (UID: \"e797ed5b-bb45-4228-915c-20e4377a5911\") " pod="kuadrant-system/authorino-65cf7699d5-s4lgf" Oct 30 21:55:46.767096 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:46.767077 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e797ed5b-bb45-4228-915c-20e4377a5911-tls-cert\") pod \"authorino-65cf7699d5-s4lgf\" (UID: \"e797ed5b-bb45-4228-915c-20e4377a5911\") " pod="kuadrant-system/authorino-65cf7699d5-s4lgf" Oct 30 21:55:46.828509 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:46.828485 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtjg5\" (UniqueName: \"kubernetes.io/projected/e797ed5b-bb45-4228-915c-20e4377a5911-kube-api-access-mtjg5\") pod \"authorino-65cf7699d5-s4lgf\" (UID: \"e797ed5b-bb45-4228-915c-20e4377a5911\") " pod="kuadrant-system/authorino-65cf7699d5-s4lgf" Oct 30 21:55:46.860393 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:46.860287 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65cf7699d5-s4lgf" Oct 30 21:55:47.233907 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:47.233884 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65cf7699d5-s4lgf"] Oct 30 21:55:47.236019 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:55:47.235991 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode797ed5b_bb45_4228_915c_20e4377a5911.slice/crio-b72aac6dee790e2d3e884f0730ed47a94e87f748be18f91b826bac94b39c9a4c WatchSource:0}: Error finding container b72aac6dee790e2d3e884f0730ed47a94e87f748be18f91b826bac94b39c9a4c: Status 404 returned error can't find the container with id b72aac6dee790e2d3e884f0730ed47a94e87f748be18f91b826bac94b39c9a4c Oct 30 21:55:47.345474 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:47.345437 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57fcf7d945-4fsw9" event={"ID":"96d23985-308b-44bd-928f-0563b9d19ff9","Type":"ContainerStarted","Data":"ac650c0d135b44bc3b94dd42f59b624744bacf79262d40992a9702b0f297e9d3"} Oct 30 21:55:47.346769 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:47.346743 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65cf7699d5-s4lgf" event={"ID":"e797ed5b-bb45-4228-915c-20e4377a5911","Type":"ContainerStarted","Data":"b72aac6dee790e2d3e884f0730ed47a94e87f748be18f91b826bac94b39c9a4c"} Oct 30 21:55:49.360552 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:49.360496 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65cf7699d5-s4lgf" event={"ID":"e797ed5b-bb45-4228-915c-20e4377a5911","Type":"ContainerStarted","Data":"0fc18d98f8d821047cb9fd67455432b476ef4aa5aef055d903f9a0ef329e7da8"} Oct 30 21:55:49.484303 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:49.484245 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-65cf7699d5-s4lgf" podStartSLOduration=2.274661519 podStartE2EDuration="3.484208275s" totalImagesPullingTime="1.209546756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:49.482342211 +0000 UTC m=+309.753040307" watchObservedRunningTime="2026-10-30 21:55:49.484208275 +0000 UTC m=+309.754906358" Oct 30 21:55:50.366982 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:50.366949 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57fcf7d945-4fsw9" event={"ID":"96d23985-308b-44bd-928f-0563b9d19ff9","Type":"ContainerStarted","Data":"888776de194d5e4b481f34f750f0396538cd734770bfb627cb06ab9e062c9489"} Oct 30 21:55:50.367415 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:50.367031 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-57fcf7d945-4fsw9" Oct 30 21:55:50.522095 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:50.522045 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-57fcf7d945-4fsw9" podStartSLOduration=2.491533088 podStartE2EDuration="5.522032019s" totalImagesPullingTime="3.030498931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:50.521471779 +0000 UTC m=+310.792169863" watchObservedRunningTime="2026-10-30 21:55:50.522032019 +0000 UTC m=+310.792730103" Oct 30 21:55:51.274398 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:55:51.274362 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 21:55:56.377722 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:55:56.377691 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-57fcf7d945-4fsw9" Oct 30 21:56:17.349637 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:56:17.349609 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dq7zs" Oct 30 21:56:17.352413 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:56:17.352394 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-rqjfj" Oct 30 21:56:21.273960 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:56:21.273912 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 21:56:51.035739 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:56:51.035650 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-6v8v2"] Oct 30 21:56:51.038031 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:56:51.038015 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-6v8v2" Oct 30 21:56:51.099912 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:56:51.099884 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-6v8v2"] Oct 30 21:56:51.165911 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:56:51.165877 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwcz9\" (UniqueName: \"kubernetes.io/projected/e70b1e47-1244-4502-92f3-7a2288cc28ab-kube-api-access-dwcz9\") pod \"llmisvc-controller-manager-6466c554d8-6v8v2\" (UID: \"e70b1e47-1244-4502-92f3-7a2288cc28ab\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-6v8v2" Oct 30 21:56:51.166101 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:56:51.165922 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e70b1e47-1244-4502-92f3-7a2288cc28ab-cert\") pod \"llmisvc-controller-manager-6466c554d8-6v8v2\" (UID: \"e70b1e47-1244-4502-92f3-7a2288cc28ab\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-6v8v2" Oct 30 21:56:51.267409 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:56:51.267368 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwcz9\" (UniqueName: \"kubernetes.io/projected/e70b1e47-1244-4502-92f3-7a2288cc28ab-kube-api-access-dwcz9\") pod \"llmisvc-controller-manager-6466c554d8-6v8v2\" (UID: \"e70b1e47-1244-4502-92f3-7a2288cc28ab\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-6v8v2" Oct 30 21:56:51.267589 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:56:51.267448 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e70b1e47-1244-4502-92f3-7a2288cc28ab-cert\") pod \"llmisvc-controller-manager-6466c554d8-6v8v2\" (UID: \"e70b1e47-1244-4502-92f3-7a2288cc28ab\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-6v8v2" Oct 30 21:56:51.269997 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:56:51.269971 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e70b1e47-1244-4502-92f3-7a2288cc28ab-cert\") pod \"llmisvc-controller-manager-6466c554d8-6v8v2\" (UID: \"e70b1e47-1244-4502-92f3-7a2288cc28ab\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-6v8v2" Oct 30 21:56:51.274980 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:56:51.274946 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 21:56:51.330956 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:56:51.330898 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwcz9\" (UniqueName: \"kubernetes.io/projected/e70b1e47-1244-4502-92f3-7a2288cc28ab-kube-api-access-dwcz9\") pod \"llmisvc-controller-manager-6466c554d8-6v8v2\" (UID: \"e70b1e47-1244-4502-92f3-7a2288cc28ab\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-6v8v2" Oct 30 21:56:51.349790 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:56:51.349769 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-6v8v2" Oct 30 21:56:51.725386 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:56:51.725361 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-6v8v2"] Oct 30 21:56:51.727464 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:56:51.727435 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode70b1e47_1244_4502_92f3_7a2288cc28ab.slice/crio-851f1ef4199f63837101e23fb311a30d7c7ea3d8dfcdbb0c124a6656fe6f3b04 WatchSource:0}: Error finding container 851f1ef4199f63837101e23fb311a30d7c7ea3d8dfcdbb0c124a6656fe6f3b04: Status 404 returned error can't find the container with id 851f1ef4199f63837101e23fb311a30d7c7ea3d8dfcdbb0c124a6656fe6f3b04 Oct 30 21:56:52.674108 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:56:52.674074 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-6v8v2" event={"ID":"e70b1e47-1244-4502-92f3-7a2288cc28ab","Type":"ContainerStarted","Data":"77f9b8fa14f31486e829aee489ebef8435d7ac57076d48e017fe0e81675bd562"} Oct 30 21:56:52.674651 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:56:52.674131 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-6v8v2" event={"ID":"e70b1e47-1244-4502-92f3-7a2288cc28ab","Type":"ContainerStarted","Data":"851f1ef4199f63837101e23fb311a30d7c7ea3d8dfcdbb0c124a6656fe6f3b04"} Oct 30 21:56:52.674651 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:56:52.674171 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-6v8v2" Oct 30 21:56:52.795964 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:56:52.795857 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-6v8v2" podStartSLOduration=2.013956105 podStartE2EDuration="2.795842169s" totalImagesPullingTime="781.886064ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:52.795447976 +0000 UTC m=+373.066146061" watchObservedRunningTime="2026-10-30 21:56:52.795842169 +0000 UTC m=+373.066540252" Oct 30 21:57:21.273729 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:57:21.273693 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 21:57:23.680757 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:57:23.680726 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-6v8v2" Oct 30 21:57:23.853585 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:57:23.853553 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-dq7zs"] Oct 30 21:57:23.853747 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:57:23.853718 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dq7zs" podUID="728bdf4f-f9d5-420b-bb7b-7f7297a67ff7" containerName="manager" containerID="cri-o://56febc75b0f46e4de774a1e08bad1f8bbb1f11c0e8c04c7c4e094218f48cd991" gracePeriod=30 Oct 30 21:57:24.100779 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:57:24.100755 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dq7zs" Oct 30 21:57:24.174663 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:57:24.174640 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/728bdf4f-f9d5-420b-bb7b-7f7297a67ff7-cert\" (UniqueName: \"kubernetes.io/secret/728bdf4f-f9d5-420b-bb7b-7f7297a67ff7-cert\") pod \"728bdf4f-f9d5-420b-bb7b-7f7297a67ff7\" (UID: \"728bdf4f-f9d5-420b-bb7b-7f7297a67ff7\") " Oct 30 21:57:24.174803 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:57:24.174692 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/728bdf4f-f9d5-420b-bb7b-7f7297a67ff7-kube-api-access-kzcbz\" (UniqueName: \"kubernetes.io/projected/728bdf4f-f9d5-420b-bb7b-7f7297a67ff7-kube-api-access-kzcbz\") pod \"728bdf4f-f9d5-420b-bb7b-7f7297a67ff7\" (UID: \"728bdf4f-f9d5-420b-bb7b-7f7297a67ff7\") " Oct 30 21:57:24.176902 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:57:24.176876 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/728bdf4f-f9d5-420b-bb7b-7f7297a67ff7-cert" pod "728bdf4f-f9d5-420b-bb7b-7f7297a67ff7" (UID: "728bdf4f-f9d5-420b-bb7b-7f7297a67ff7"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:24.177015 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:57:24.176913 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/728bdf4f-f9d5-420b-bb7b-7f7297a67ff7-kube-api-access-kzcbz" pod "728bdf4f-f9d5-420b-bb7b-7f7297a67ff7" (UID: "728bdf4f-f9d5-420b-bb7b-7f7297a67ff7"). InnerVolumeSpecName "kube-api-access-kzcbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:24.276250 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:57:24.276150 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzcbz\" (UniqueName: \"kubernetes.io/projected/728bdf4f-f9d5-420b-bb7b-7f7297a67ff7-kube-api-access-kzcbz\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:24.276250 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:57:24.276177 5918 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/728bdf4f-f9d5-420b-bb7b-7f7297a67ff7-cert\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:24.832248 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:57:24.832190 5918 generic.go:356] "Generic (PLEG): container finished" podID="728bdf4f-f9d5-420b-bb7b-7f7297a67ff7" containerID="56febc75b0f46e4de774a1e08bad1f8bbb1f11c0e8c04c7c4e094218f48cd991" exitCode=0 Oct 30 21:57:24.832729 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:57:24.832261 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dq7zs" Oct 30 21:57:24.832729 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:57:24.832285 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dq7zs" event={"ID":"728bdf4f-f9d5-420b-bb7b-7f7297a67ff7","Type":"ContainerDied","Data":"56febc75b0f46e4de774a1e08bad1f8bbb1f11c0e8c04c7c4e094218f48cd991"} Oct 30 21:57:24.832729 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:57:24.832358 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dq7zs" event={"ID":"728bdf4f-f9d5-420b-bb7b-7f7297a67ff7","Type":"ContainerDied","Data":"251c1c19b282d4cc9b16f766b90e9c939b0033a585c1caa10793e04f5d832148"} Oct 30 21:57:24.832729 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:57:24.832393 5918 scope.go:122] "RemoveContainer" containerID="56febc75b0f46e4de774a1e08bad1f8bbb1f11c0e8c04c7c4e094218f48cd991" Oct 30 21:57:24.841583 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:57:24.841568 5918 scope.go:122] "RemoveContainer" containerID="56febc75b0f46e4de774a1e08bad1f8bbb1f11c0e8c04c7c4e094218f48cd991" Oct 30 21:57:24.841829 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:57:24.841807 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56febc75b0f46e4de774a1e08bad1f8bbb1f11c0e8c04c7c4e094218f48cd991\": container with ID starting with 56febc75b0f46e4de774a1e08bad1f8bbb1f11c0e8c04c7c4e094218f48cd991 not found: ID does not exist" containerID="56febc75b0f46e4de774a1e08bad1f8bbb1f11c0e8c04c7c4e094218f48cd991" Oct 30 21:57:24.841896 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:57:24.841841 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56febc75b0f46e4de774a1e08bad1f8bbb1f11c0e8c04c7c4e094218f48cd991"} err="failed to get container status \"56febc75b0f46e4de774a1e08bad1f8bbb1f11c0e8c04c7c4e094218f48cd991\": rpc error: code = NotFound desc = could not find container \"56febc75b0f46e4de774a1e08bad1f8bbb1f11c0e8c04c7c4e094218f48cd991\": container with ID starting with 56febc75b0f46e4de774a1e08bad1f8bbb1f11c0e8c04c7c4e094218f48cd991 not found: ID does not exist" Oct 30 21:57:25.001030 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:57:25.000999 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-dq7zs"] Oct 30 21:57:25.051466 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:57:25.051430 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-dq7zs"] Oct 30 21:57:25.850344 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:57:25.850318 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="728bdf4f-f9d5-420b-bb7b-7f7297a67ff7" path="/var/lib/kubelet/pods/728bdf4f-f9d5-420b-bb7b-7f7297a67ff7/volumes" Oct 30 21:57:51.274150 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:57:51.274112 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 21:58:17.506648 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:17.506617 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb"] Oct 30 21:58:17.507091 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:17.507049 5918 cpu_manager.go:414] "containerMap: removing container" podUID="728bdf4f-f9d5-420b-bb7b-7f7297a67ff7" containerName="manager" Oct 30 21:58:17.507091 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:17.507061 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="728bdf4f-f9d5-420b-bb7b-7f7297a67ff7" containerName="manager" Oct 30 21:58:17.507165 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:17.507149 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="728bdf4f-f9d5-420b-bb7b-7f7297a67ff7" containerName="manager" Oct 30 21:58:17.510344 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:17.510323 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" Oct 30 21:58:17.571238 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:17.568121 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-gbj8s\"" Oct 30 21:58:17.571238 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:17.568912 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 21:58:17.622054 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:17.622018 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb"] Oct 30 21:58:17.674313 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:17.674273 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7707ff63-b08b-4233-afd0-4d71f7a19835-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-ckkdb\" (UID: \"7707ff63-b08b-4233-afd0-4d71f7a19835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" Oct 30 21:58:17.674313 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:17.674312 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7707ff63-b08b-4233-afd0-4d71f7a19835-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-ckkdb\" (UID: \"7707ff63-b08b-4233-afd0-4d71f7a19835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" Oct 30 21:58:17.674566 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:17.674337 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7707ff63-b08b-4233-afd0-4d71f7a19835-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-ckkdb\" (UID: \"7707ff63-b08b-4233-afd0-4d71f7a19835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" Oct 30 21:58:17.674566 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:17.674507 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7707ff63-b08b-4233-afd0-4d71f7a19835-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-ckkdb\" (UID: \"7707ff63-b08b-4233-afd0-4d71f7a19835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" Oct 30 21:58:17.674566 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:17.674555 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbs77\" (UniqueName: \"kubernetes.io/projected/7707ff63-b08b-4233-afd0-4d71f7a19835-kube-api-access-mbs77\") pod \"maas-default-gateway-openshift-default-67bf986f99-ckkdb\" (UID: \"7707ff63-b08b-4233-afd0-4d71f7a19835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" Oct 30 21:58:17.674726 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:17.674616 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7707ff63-b08b-4233-afd0-4d71f7a19835-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-ckkdb\" (UID: \"7707ff63-b08b-4233-afd0-4d71f7a19835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" Oct 30 21:58:17.674726 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:17.674650 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7707ff63-b08b-4233-afd0-4d71f7a19835-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-ckkdb\" (UID: \"7707ff63-b08b-4233-afd0-4d71f7a19835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" Oct 30 21:58:17.674841 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:17.674752 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7707ff63-b08b-4233-afd0-4d71f7a19835-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-ckkdb\" (UID: \"7707ff63-b08b-4233-afd0-4d71f7a19835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" Oct 30 21:58:17.674841 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:17.674773 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7707ff63-b08b-4233-afd0-4d71f7a19835-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-ckkdb\" (UID: \"7707ff63-b08b-4233-afd0-4d71f7a19835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" Oct 30 21:58:17.775661 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:17.775568 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7707ff63-b08b-4233-afd0-4d71f7a19835-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-ckkdb\" (UID: \"7707ff63-b08b-4233-afd0-4d71f7a19835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" Oct 30 21:58:17.775661 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:17.775603 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7707ff63-b08b-4233-afd0-4d71f7a19835-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-ckkdb\" (UID: \"7707ff63-b08b-4233-afd0-4d71f7a19835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" Oct 30 21:58:17.775895 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:17.775725 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7707ff63-b08b-4233-afd0-4d71f7a19835-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-ckkdb\" (UID: \"7707ff63-b08b-4233-afd0-4d71f7a19835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" Oct 30 21:58:17.775895 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:17.775763 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7707ff63-b08b-4233-afd0-4d71f7a19835-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-ckkdb\" (UID: \"7707ff63-b08b-4233-afd0-4d71f7a19835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" Oct 30 21:58:17.775895 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:17.775798 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7707ff63-b08b-4233-afd0-4d71f7a19835-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-ckkdb\" (UID: \"7707ff63-b08b-4233-afd0-4d71f7a19835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" Oct 30 21:58:17.775895 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:17.775871 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7707ff63-b08b-4233-afd0-4d71f7a19835-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-ckkdb\" (UID: \"7707ff63-b08b-4233-afd0-4d71f7a19835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" Oct 30 21:58:17.776124 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:17.775904 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbs77\" (UniqueName: \"kubernetes.io/projected/7707ff63-b08b-4233-afd0-4d71f7a19835-kube-api-access-mbs77\") pod \"maas-default-gateway-openshift-default-67bf986f99-ckkdb\" (UID: \"7707ff63-b08b-4233-afd0-4d71f7a19835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" Oct 30 21:58:17.776124 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:17.775934 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7707ff63-b08b-4233-afd0-4d71f7a19835-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-ckkdb\" (UID: \"7707ff63-b08b-4233-afd0-4d71f7a19835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" Oct 30 21:58:17.776124 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:17.775965 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7707ff63-b08b-4233-afd0-4d71f7a19835-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-ckkdb\" (UID: \"7707ff63-b08b-4233-afd0-4d71f7a19835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" Oct 30 21:58:17.776322 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:17.776120 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7707ff63-b08b-4233-afd0-4d71f7a19835-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-ckkdb\" (UID: \"7707ff63-b08b-4233-afd0-4d71f7a19835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" Oct 30 21:58:17.776322 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:17.776156 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7707ff63-b08b-4233-afd0-4d71f7a19835-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-ckkdb\" (UID: \"7707ff63-b08b-4233-afd0-4d71f7a19835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" Oct 30 21:58:17.776496 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:17.776472 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7707ff63-b08b-4233-afd0-4d71f7a19835-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-ckkdb\" (UID: \"7707ff63-b08b-4233-afd0-4d71f7a19835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" Oct 30 21:58:17.776550 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:17.776506 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7707ff63-b08b-4233-afd0-4d71f7a19835-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-ckkdb\" (UID: \"7707ff63-b08b-4233-afd0-4d71f7a19835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" Oct 30 21:58:17.776607 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:17.776507 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7707ff63-b08b-4233-afd0-4d71f7a19835-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-ckkdb\" (UID: \"7707ff63-b08b-4233-afd0-4d71f7a19835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" Oct 30 21:58:17.778413 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:17.778389 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7707ff63-b08b-4233-afd0-4d71f7a19835-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-ckkdb\" (UID: \"7707ff63-b08b-4233-afd0-4d71f7a19835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" Oct 30 21:58:17.778638 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:17.778619 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7707ff63-b08b-4233-afd0-4d71f7a19835-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-ckkdb\" (UID: \"7707ff63-b08b-4233-afd0-4d71f7a19835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" Oct 30 21:58:17.839248 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:17.839165 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7707ff63-b08b-4233-afd0-4d71f7a19835-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-ckkdb\" (UID: \"7707ff63-b08b-4233-afd0-4d71f7a19835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" Oct 30 21:58:17.839451 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:17.839403 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbs77\" (UniqueName: \"kubernetes.io/projected/7707ff63-b08b-4233-afd0-4d71f7a19835-kube-api-access-mbs77\") pod \"maas-default-gateway-openshift-default-67bf986f99-ckkdb\" (UID: \"7707ff63-b08b-4233-afd0-4d71f7a19835\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" Oct 30 21:58:18.123265 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:18.123157 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" Oct 30 21:58:18.578427 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:18.578394 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb"] Oct 30 21:58:18.580134 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:58:18.580106 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7707ff63_b08b_4233_afd0_4d71f7a19835.slice/crio-e28b3c5b0aa1bb83d9b688757ff93b0ebe2091f2ae27c0d8e68de98e5277cf8c WatchSource:0}: Error finding container e28b3c5b0aa1bb83d9b688757ff93b0ebe2091f2ae27c0d8e68de98e5277cf8c: Status 404 returned error can't find the container with id e28b3c5b0aa1bb83d9b688757ff93b0ebe2091f2ae27c0d8e68de98e5277cf8c Oct 30 21:58:18.582672 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:18.582651 5918 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:58:19.104757 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:19.104721 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" event={"ID":"7707ff63-b08b-4233-afd0-4d71f7a19835","Type":"ContainerStarted","Data":"e28b3c5b0aa1bb83d9b688757ff93b0ebe2091f2ae27c0d8e68de98e5277cf8c"} Oct 30 21:58:21.277712 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:58:21.277628 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 21:58:22.507321 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:22.507287 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:22.507578 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:22.507373 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:22.507578 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:22.507416 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:23.127809 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:23.127771 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" event={"ID":"7707ff63-b08b-4233-afd0-4d71f7a19835","Type":"ContainerStarted","Data":"46314bbf9833f7a9727665fb4e0686ed50195676b49c1f14ebcc3442efcb65b5"} Oct 30 21:58:24.123938 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:24.123904 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" Oct 30 21:58:24.125395 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:24.125370 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-ckkdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:58:24.125480 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:24.125432 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" podUID="7707ff63-b08b-4233-afd0-4d71f7a19835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 21:58:25.123923 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:25.123881 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-ckkdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:58:25.124110 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:25.123945 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" podUID="7707ff63-b08b-4233-afd0-4d71f7a19835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 21:58:26.124295 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:26.124200 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-ckkdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:58:26.124710 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:26.124324 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" podUID="7707ff63-b08b-4233-afd0-4d71f7a19835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 21:58:27.123995 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:27.123954 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-ckkdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:58:27.124197 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:27.124020 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" podUID="7707ff63-b08b-4233-afd0-4d71f7a19835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 21:58:28.124099 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:28.124061 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-ckkdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:58:28.124542 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:28.124119 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" podUID="7707ff63-b08b-4233-afd0-4d71f7a19835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 21:58:29.124225 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:29.124178 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-ckkdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:58:29.124599 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:29.124246 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" podUID="7707ff63-b08b-4233-afd0-4d71f7a19835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 21:58:30.707157 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:30.707098 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" Oct 30 21:58:30.707711 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:30.707692 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" Oct 30 21:58:30.708579 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:30.708554 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" Oct 30 21:58:30.873187 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:30.873133 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" podStartSLOduration=9.949649838 podStartE2EDuration="13.873119188s" totalImagesPullingTime="3.92346935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:23.306269106 +0000 UTC m=+463.576967190" watchObservedRunningTime="2026-10-30 21:58:30.873119188 +0000 UTC m=+471.143817272" Oct 30 21:58:32.633248 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:32.633200 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws"] Oct 30 21:58:32.636950 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:32.636932 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" Oct 30 21:58:32.716088 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:32.716057 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac8d2c79-d9b9-4650-8ffa-39022153bff9-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-zwvws\" (UID: \"ac8d2c79-d9b9-4650-8ffa-39022153bff9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" Oct 30 21:58:32.716236 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:32.716092 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klrwj\" (UniqueName: \"kubernetes.io/projected/ac8d2c79-d9b9-4650-8ffa-39022153bff9-kube-api-access-klrwj\") pod \"maas-default-gateway-openshift-default-67bf986f99-zwvws\" (UID: \"ac8d2c79-d9b9-4650-8ffa-39022153bff9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" Oct 30 21:58:32.716236 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:32.716112 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac8d2c79-d9b9-4650-8ffa-39022153bff9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-zwvws\" (UID: \"ac8d2c79-d9b9-4650-8ffa-39022153bff9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" Oct 30 21:58:32.716319 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:32.716259 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8d2c79-d9b9-4650-8ffa-39022153bff9-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-zwvws\" (UID: \"ac8d2c79-d9b9-4650-8ffa-39022153bff9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" Oct 30 21:58:32.716319 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:32.716300 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac8d2c79-d9b9-4650-8ffa-39022153bff9-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-zwvws\" (UID: \"ac8d2c79-d9b9-4650-8ffa-39022153bff9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" Oct 30 21:58:32.716390 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:32.716319 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac8d2c79-d9b9-4650-8ffa-39022153bff9-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-zwvws\" (UID: \"ac8d2c79-d9b9-4650-8ffa-39022153bff9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" Oct 30 21:58:32.716603 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:32.716413 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8d2c79-d9b9-4650-8ffa-39022153bff9-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-zwvws\" (UID: \"ac8d2c79-d9b9-4650-8ffa-39022153bff9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" Oct 30 21:58:32.716603 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:32.716435 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac8d2c79-d9b9-4650-8ffa-39022153bff9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-zwvws\" (UID: \"ac8d2c79-d9b9-4650-8ffa-39022153bff9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" Oct 30 21:58:32.716603 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:32.716482 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac8d2c79-d9b9-4650-8ffa-39022153bff9-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-zwvws\" (UID: \"ac8d2c79-d9b9-4650-8ffa-39022153bff9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" Oct 30 21:58:32.817891 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:32.817861 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8d2c79-d9b9-4650-8ffa-39022153bff9-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-zwvws\" (UID: \"ac8d2c79-d9b9-4650-8ffa-39022153bff9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" Oct 30 21:58:32.818092 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:32.817900 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac8d2c79-d9b9-4650-8ffa-39022153bff9-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-zwvws\" (UID: \"ac8d2c79-d9b9-4650-8ffa-39022153bff9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" Oct 30 21:58:32.818092 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:32.817916 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac8d2c79-d9b9-4650-8ffa-39022153bff9-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-zwvws\" (UID: \"ac8d2c79-d9b9-4650-8ffa-39022153bff9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" Oct 30 21:58:32.818092 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:32.817942 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8d2c79-d9b9-4650-8ffa-39022153bff9-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-zwvws\" (UID: \"ac8d2c79-d9b9-4650-8ffa-39022153bff9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" Oct 30 21:58:32.818092 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:32.817958 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac8d2c79-d9b9-4650-8ffa-39022153bff9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-zwvws\" (UID: \"ac8d2c79-d9b9-4650-8ffa-39022153bff9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" Oct 30 21:58:32.818092 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:32.818009 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac8d2c79-d9b9-4650-8ffa-39022153bff9-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-zwvws\" (UID: \"ac8d2c79-d9b9-4650-8ffa-39022153bff9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" Oct 30 21:58:32.818092 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:32.818048 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac8d2c79-d9b9-4650-8ffa-39022153bff9-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-zwvws\" (UID: \"ac8d2c79-d9b9-4650-8ffa-39022153bff9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" Oct 30 21:58:32.818414 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:32.818342 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klrwj\" (UniqueName: \"kubernetes.io/projected/ac8d2c79-d9b9-4650-8ffa-39022153bff9-kube-api-access-klrwj\") pod \"maas-default-gateway-openshift-default-67bf986f99-zwvws\" (UID: \"ac8d2c79-d9b9-4650-8ffa-39022153bff9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" Oct 30 21:58:32.818414 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:32.818377 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac8d2c79-d9b9-4650-8ffa-39022153bff9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-zwvws\" (UID: \"ac8d2c79-d9b9-4650-8ffa-39022153bff9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" Oct 30 21:58:32.818527 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:32.818421 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8d2c79-d9b9-4650-8ffa-39022153bff9-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-zwvws\" (UID: \"ac8d2c79-d9b9-4650-8ffa-39022153bff9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" Oct 30 21:58:32.818527 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:32.818491 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8d2c79-d9b9-4650-8ffa-39022153bff9-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-zwvws\" (UID: \"ac8d2c79-d9b9-4650-8ffa-39022153bff9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" Oct 30 21:58:32.818619 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:32.818573 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac8d2c79-d9b9-4650-8ffa-39022153bff9-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-zwvws\" (UID: \"ac8d2c79-d9b9-4650-8ffa-39022153bff9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" Oct 30 21:58:32.818808 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:32.818785 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac8d2c79-d9b9-4650-8ffa-39022153bff9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-zwvws\" (UID: \"ac8d2c79-d9b9-4650-8ffa-39022153bff9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" Oct 30 21:58:32.818903 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:32.818887 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac8d2c79-d9b9-4650-8ffa-39022153bff9-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-zwvws\" (UID: \"ac8d2c79-d9b9-4650-8ffa-39022153bff9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" Oct 30 21:58:32.820388 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:32.820373 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac8d2c79-d9b9-4650-8ffa-39022153bff9-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-zwvws\" (UID: \"ac8d2c79-d9b9-4650-8ffa-39022153bff9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" Oct 30 21:58:32.820866 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:32.820851 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac8d2c79-d9b9-4650-8ffa-39022153bff9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-zwvws\" (UID: \"ac8d2c79-d9b9-4650-8ffa-39022153bff9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" Oct 30 21:58:32.852615 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:32.852581 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws"] Oct 30 21:58:32.881572 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:32.881539 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac8d2c79-d9b9-4650-8ffa-39022153bff9-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-zwvws\" (UID: \"ac8d2c79-d9b9-4650-8ffa-39022153bff9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" Oct 30 21:58:32.881719 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:32.881602 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klrwj\" (UniqueName: \"kubernetes.io/projected/ac8d2c79-d9b9-4650-8ffa-39022153bff9-kube-api-access-klrwj\") pod \"maas-default-gateway-openshift-default-67bf986f99-zwvws\" (UID: \"ac8d2c79-d9b9-4650-8ffa-39022153bff9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" Oct 30 21:58:32.949766 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:32.949734 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" Oct 30 21:58:33.378517 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:33.378491 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws"] Oct 30 21:58:33.380612 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:58:33.380587 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac8d2c79_d9b9_4650_8ffa_39022153bff9.slice/crio-744f9e0b8dc8204e510e31c0fa75f6999d89fcb67cfd454b5469bf1dc396d388 WatchSource:0}: Error finding container 744f9e0b8dc8204e510e31c0fa75f6999d89fcb67cfd454b5469bf1dc396d388: Status 404 returned error can't find the container with id 744f9e0b8dc8204e510e31c0fa75f6999d89fcb67cfd454b5469bf1dc396d388 Oct 30 21:58:33.383812 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:33.383777 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:33.383901 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:33.383847 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:33.383901 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:33.383878 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:34.184072 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:34.184029 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" event={"ID":"ac8d2c79-d9b9-4650-8ffa-39022153bff9","Type":"ContainerStarted","Data":"76cefb2b521e9d2194138e8c790131b4b2bf4e0bcfc3847777ed6e6c14295ed2"} Oct 30 21:58:34.184480 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:34.184091 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" event={"ID":"ac8d2c79-d9b9-4650-8ffa-39022153bff9","Type":"ContainerStarted","Data":"744f9e0b8dc8204e510e31c0fa75f6999d89fcb67cfd454b5469bf1dc396d388"} Oct 30 21:58:34.949903 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:34.949872 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" Oct 30 21:58:34.951148 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:34.951121 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-zwvws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 21:58:34.951299 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:34.951179 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" podUID="ac8d2c79-d9b9-4650-8ffa-39022153bff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 21:58:35.950589 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:35.950543 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-zwvws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 21:58:35.950983 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:35.950619 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" podUID="ac8d2c79-d9b9-4650-8ffa-39022153bff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 21:58:36.950840 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:36.950803 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-zwvws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 21:58:36.951265 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:36.950860 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" podUID="ac8d2c79-d9b9-4650-8ffa-39022153bff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 21:58:37.950693 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:37.950652 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-zwvws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 21:58:37.950869 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:37.950718 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" podUID="ac8d2c79-d9b9-4650-8ffa-39022153bff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 21:58:38.950423 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:38.950385 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-zwvws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 21:58:38.950640 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:38.950444 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" podUID="ac8d2c79-d9b9-4650-8ffa-39022153bff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 21:58:39.950488 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:39.950452 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-zwvws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 21:58:39.950986 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:39.950525 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" podUID="ac8d2c79-d9b9-4650-8ffa-39022153bff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 21:58:40.833182 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:40.833130 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" podStartSLOduration=8.83311546 podStartE2EDuration="8.83311546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:34.409260498 +0000 UTC m=+474.679958584" watchObservedRunningTime="2026-10-30 21:58:40.83311546 +0000 UTC m=+481.103813543" Oct 30 21:58:40.834729 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:40.834699 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-65cf7699d5-s4lgf"] Oct 30 21:58:40.834937 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:40.834910 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-65cf7699d5-s4lgf" podUID="e797ed5b-bb45-4228-915c-20e4377a5911" containerName="authorino" containerID="cri-o://0fc18d98f8d821047cb9fd67455432b476ef4aa5aef055d903f9a0ef329e7da8" gracePeriod=30 Oct 30 21:58:41.100138 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:41.100077 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65cf7699d5-s4lgf" Oct 30 21:58:41.204354 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:41.204323 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e797ed5b-bb45-4228-915c-20e4377a5911-kube-api-access-mtjg5\" (UniqueName: \"kubernetes.io/projected/e797ed5b-bb45-4228-915c-20e4377a5911-kube-api-access-mtjg5\") pod \"e797ed5b-bb45-4228-915c-20e4377a5911\" (UID: \"e797ed5b-bb45-4228-915c-20e4377a5911\") " Oct 30 21:58:41.204528 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:41.204424 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e797ed5b-bb45-4228-915c-20e4377a5911-tls-cert\" (UniqueName: \"kubernetes.io/secret/e797ed5b-bb45-4228-915c-20e4377a5911-tls-cert\") pod \"e797ed5b-bb45-4228-915c-20e4377a5911\" (UID: \"e797ed5b-bb45-4228-915c-20e4377a5911\") " Oct 30 21:58:41.206721 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:41.206684 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e797ed5b-bb45-4228-915c-20e4377a5911-kube-api-access-mtjg5" pod "e797ed5b-bb45-4228-915c-20e4377a5911" (UID: "e797ed5b-bb45-4228-915c-20e4377a5911"). InnerVolumeSpecName "kube-api-access-mtjg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:41.215404 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:41.215375 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e797ed5b-bb45-4228-915c-20e4377a5911-tls-cert" pod "e797ed5b-bb45-4228-915c-20e4377a5911" (UID: "e797ed5b-bb45-4228-915c-20e4377a5911"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:41.242747 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:41.242708 5918 generic.go:356] "Generic (PLEG): container finished" podID="e797ed5b-bb45-4228-915c-20e4377a5911" containerID="0fc18d98f8d821047cb9fd67455432b476ef4aa5aef055d903f9a0ef329e7da8" exitCode=0 Oct 30 21:58:41.242876 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:41.242766 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65cf7699d5-s4lgf" Oct 30 21:58:41.242876 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:41.242768 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65cf7699d5-s4lgf" event={"ID":"e797ed5b-bb45-4228-915c-20e4377a5911","Type":"ContainerDied","Data":"0fc18d98f8d821047cb9fd67455432b476ef4aa5aef055d903f9a0ef329e7da8"} Oct 30 21:58:41.242947 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:41.242890 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65cf7699d5-s4lgf" event={"ID":"e797ed5b-bb45-4228-915c-20e4377a5911","Type":"ContainerDied","Data":"b72aac6dee790e2d3e884f0730ed47a94e87f748be18f91b826bac94b39c9a4c"} Oct 30 21:58:41.242947 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:41.242917 5918 scope.go:122] "RemoveContainer" containerID="0fc18d98f8d821047cb9fd67455432b476ef4aa5aef055d903f9a0ef329e7da8" Oct 30 21:58:41.254718 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:41.254700 5918 scope.go:122] "RemoveContainer" containerID="0fc18d98f8d821047cb9fd67455432b476ef4aa5aef055d903f9a0ef329e7da8" Oct 30 21:58:41.255084 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:58:41.255062 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fc18d98f8d821047cb9fd67455432b476ef4aa5aef055d903f9a0ef329e7da8\": container with ID starting with 0fc18d98f8d821047cb9fd67455432b476ef4aa5aef055d903f9a0ef329e7da8 not found: ID does not exist" containerID="0fc18d98f8d821047cb9fd67455432b476ef4aa5aef055d903f9a0ef329e7da8" Oct 30 21:58:41.255162 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:41.255093 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fc18d98f8d821047cb9fd67455432b476ef4aa5aef055d903f9a0ef329e7da8"} err="failed to get container status \"0fc18d98f8d821047cb9fd67455432b476ef4aa5aef055d903f9a0ef329e7da8\": rpc error: code = NotFound desc = could not find container \"0fc18d98f8d821047cb9fd67455432b476ef4aa5aef055d903f9a0ef329e7da8\": container with ID starting with 0fc18d98f8d821047cb9fd67455432b476ef4aa5aef055d903f9a0ef329e7da8 not found: ID does not exist" Oct 30 21:58:41.305804 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:41.305780 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtjg5\" (UniqueName: \"kubernetes.io/projected/e797ed5b-bb45-4228-915c-20e4377a5911-kube-api-access-mtjg5\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:41.305804 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:41.305806 5918 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e797ed5b-bb45-4228-915c-20e4377a5911-tls-cert\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:41.411935 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:41.411856 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-65cf7699d5-s4lgf"] Oct 30 21:58:41.418915 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:41.418889 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-65cf7699d5-s4lgf"] Oct 30 21:58:41.562748 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:41.562721 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" Oct 30 21:58:41.563250 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:41.563208 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" Oct 30 21:58:41.564095 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:41.564076 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" Oct 30 21:58:41.850708 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:58:41.850672 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e797ed5b-bb45-4228-915c-20e4377a5911" path="/var/lib/kubelet/pods/e797ed5b-bb45-4228-915c-20e4377a5911/volumes" Oct 30 21:58:51.274614 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:58:51.274576 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 21:59:21.273886 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:59:21.273849 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 21:59:43.889065 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:43.889030 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2"] Oct 30 21:59:43.889606 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:43.889459 5918 cpu_manager.go:414] "containerMap: removing container" podUID="e797ed5b-bb45-4228-915c-20e4377a5911" containerName="authorino" Oct 30 21:59:43.889606 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:43.889471 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e797ed5b-bb45-4228-915c-20e4377a5911" containerName="authorino" Oct 30 21:59:43.889606 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:43.889559 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="e797ed5b-bb45-4228-915c-20e4377a5911" containerName="authorino" Oct 30 21:59:43.894061 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:43.894040 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 21:59:43.956100 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:43.956069 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2"] Oct 30 21:59:44.003785 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:44.003754 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bg6f\" (UniqueName: \"kubernetes.io/projected/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-kube-api-access-8bg6f\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2\" (UID: \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 21:59:44.003978 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:44.003818 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-workload-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2\" (UID: \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 21:59:44.003978 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:44.003861 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-istio-token\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2\" (UID: \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 21:59:44.003978 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:44.003885 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-workload-certs\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2\" (UID: \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 21:59:44.003978 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:44.003929 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-credential-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2\" (UID: \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 21:59:44.004185 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:44.003989 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-istio-data\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2\" (UID: \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 21:59:44.004185 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:44.004012 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-istio-podinfo\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2\" (UID: \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 21:59:44.004550 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:44.004520 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-istiod-ca-cert\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2\" (UID: \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 21:59:44.004628 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:44.004569 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-istio-envoy\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2\" (UID: \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 21:59:44.031709 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:44.031682 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-9cff60-openshift-default-dockercfg-z696x\"" Oct 30 21:59:44.106160 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:44.106126 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-credential-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2\" (UID: \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 21:59:44.106340 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:44.106189 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-istio-data\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2\" (UID: \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 21:59:44.106340 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:44.106229 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-istio-podinfo\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2\" (UID: \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 21:59:44.106340 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:44.106297 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-istiod-ca-cert\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2\" (UID: \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 21:59:44.106340 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:44.106325 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-istio-envoy\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2\" (UID: \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 21:59:44.106554 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:44.106391 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bg6f\" (UniqueName: \"kubernetes.io/projected/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-kube-api-access-8bg6f\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2\" (UID: \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 21:59:44.106554 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:44.106426 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-workload-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2\" (UID: \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 21:59:44.106554 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:44.106470 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-istio-token\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2\" (UID: \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 21:59:44.106554 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:44.106498 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-workload-certs\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2\" (UID: \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 21:59:44.106770 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:44.106611 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-credential-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2\" (UID: \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 21:59:44.106770 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:44.106692 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-istio-data\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2\" (UID: \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 21:59:44.106880 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:44.106858 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-workload-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2\" (UID: \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 21:59:44.106974 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:44.106948 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-workload-certs\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2\" (UID: \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 21:59:44.107261 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:44.107207 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-istiod-ca-cert\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2\" (UID: \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 21:59:44.108946 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:44.108923 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-istio-envoy\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2\" (UID: \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 21:59:44.109036 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:44.108983 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-istio-podinfo\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2\" (UID: \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 21:59:44.169799 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:44.169735 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-istio-token\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2\" (UID: \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 21:59:44.169934 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:44.169910 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bg6f\" (UniqueName: \"kubernetes.io/projected/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-kube-api-access-8bg6f\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2\" (UID: \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 21:59:44.206811 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:44.206777 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 21:59:44.635377 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:44.635357 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2"] Oct 30 21:59:44.637626 ip-10-0-137-81 kubenswrapper[5918]: W1030 21:59:44.637597 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6f37484_37b2_4fab_8b6b_cdc1c171ff93.slice/crio-a135c2cd5dc55856c09c588c4e38c224992f548503d768c10fdb31c49b36be56 WatchSource:0}: Error finding container a135c2cd5dc55856c09c588c4e38c224992f548503d768c10fdb31c49b36be56: Status 404 returned error can't find the container with id a135c2cd5dc55856c09c588c4e38c224992f548503d768c10fdb31c49b36be56 Oct 30 21:59:44.639794 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:44.639764 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:44.639865 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:44.639825 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:44.639865 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:44.639854 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:45.560369 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:45.560335 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" event={"ID":"f6f37484-37b2-4fab-8b6b-cdc1c171ff93","Type":"ContainerStarted","Data":"b3d14947d3b89637bb8d139bdff16d91ff5cd35a461de9324822e4b4fbbfc0d7"} Oct 30 21:59:45.560749 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:45.560411 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" event={"ID":"f6f37484-37b2-4fab-8b6b-cdc1c171ff93","Type":"ContainerStarted","Data":"a135c2cd5dc55856c09c588c4e38c224992f548503d768c10fdb31c49b36be56"} Oct 30 21:59:45.735741 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:45.735684 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" podStartSLOduration=2.735670349 podStartE2EDuration="2.735670349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:45.734180996 +0000 UTC m=+546.004879081" watchObservedRunningTime="2026-10-30 21:59:45.735670349 +0000 UTC m=+546.006368432" Oct 30 21:59:46.208062 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:46.208033 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 21:59:46.213042 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:46.213018 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 21:59:46.565667 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:46.565592 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 21:59:46.566478 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:46.566460 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 21:59:51.274769 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:59:51.274681 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 21:59:58.864195 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:58.864161 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx"] Oct 30 21:59:58.870182 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:58.870157 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx" Oct 30 21:59:58.971152 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:58.971114 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/599ae550-c43b-4094-be3d-d9a256a25a63-metrics-certs\") pod \"maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx\" (UID: \"599ae550-c43b-4094-be3d-d9a256a25a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx" Oct 30 21:59:58.971321 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:58.971161 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/599ae550-c43b-4094-be3d-d9a256a25a63-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx\" (UID: \"599ae550-c43b-4094-be3d-d9a256a25a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx" Oct 30 21:59:58.971321 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:58.971284 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/599ae550-c43b-4094-be3d-d9a256a25a63-maas-api-tls\") pod \"maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx\" (UID: \"599ae550-c43b-4094-be3d-d9a256a25a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx" Oct 30 21:59:58.971321 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:58.971313 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6lcs\" (UniqueName: \"kubernetes.io/projected/599ae550-c43b-4094-be3d-d9a256a25a63-kube-api-access-c6lcs\") pod \"maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx\" (UID: \"599ae550-c43b-4094-be3d-d9a256a25a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx" Oct 30 21:59:58.976393 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:58.976375 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-8ca52a-serving-cert\"" Oct 30 21:59:58.978160 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:58.978139 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx"] Oct 30 21:59:59.072714 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:59.072680 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/599ae550-c43b-4094-be3d-d9a256a25a63-maas-api-tls\") pod \"maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx\" (UID: \"599ae550-c43b-4094-be3d-d9a256a25a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx" Oct 30 21:59:59.072887 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:59.072743 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6lcs\" (UniqueName: \"kubernetes.io/projected/599ae550-c43b-4094-be3d-d9a256a25a63-kube-api-access-c6lcs\") pod \"maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx\" (UID: \"599ae550-c43b-4094-be3d-d9a256a25a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx" Oct 30 21:59:59.072887 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:59:59.072846 5918 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-8ca52a-serving-cert: secret "maas-api-e2e-worker-w3-8ca52a-serving-cert" not found Oct 30 21:59:59.073001 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:59.072887 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/599ae550-c43b-4094-be3d-d9a256a25a63-metrics-certs\") pod \"maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx\" (UID: \"599ae550-c43b-4094-be3d-d9a256a25a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx" Oct 30 21:59:59.073001 ip-10-0-137-81 kubenswrapper[5918]: E1030 21:59:59.072919 5918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/599ae550-c43b-4094-be3d-d9a256a25a63-maas-api-tls podName:599ae550-c43b-4094-be3d-d9a256a25a63 nodeName:}" failed. No retries permitted until 2026-10-30 21:59:59.572900459 +0000 UTC m=+559.843598521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/599ae550-c43b-4094-be3d-d9a256a25a63-maas-api-tls") pod "maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx" (UID: "599ae550-c43b-4094-be3d-d9a256a25a63") : secret "maas-api-e2e-worker-w3-8ca52a-serving-cert" not found Oct 30 21:59:59.073001 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:59.072969 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/599ae550-c43b-4094-be3d-d9a256a25a63-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx\" (UID: \"599ae550-c43b-4094-be3d-d9a256a25a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx" Oct 30 21:59:59.073625 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:59.073600 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/599ae550-c43b-4094-be3d-d9a256a25a63-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx\" (UID: \"599ae550-c43b-4094-be3d-d9a256a25a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx" Oct 30 21:59:59.075504 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:59.075483 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/599ae550-c43b-4094-be3d-d9a256a25a63-metrics-certs\") pod \"maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx\" (UID: \"599ae550-c43b-4094-be3d-d9a256a25a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx" Oct 30 21:59:59.136143 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:59.136063 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6lcs\" (UniqueName: \"kubernetes.io/projected/599ae550-c43b-4094-be3d-d9a256a25a63-kube-api-access-c6lcs\") pod \"maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx\" (UID: \"599ae550-c43b-4094-be3d-d9a256a25a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx" Oct 30 21:59:59.578844 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:59.578806 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/599ae550-c43b-4094-be3d-d9a256a25a63-maas-api-tls\") pod \"maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx\" (UID: \"599ae550-c43b-4094-be3d-d9a256a25a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx" Oct 30 21:59:59.581457 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:59.581434 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/599ae550-c43b-4094-be3d-d9a256a25a63-maas-api-tls\") pod \"maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx\" (UID: \"599ae550-c43b-4094-be3d-d9a256a25a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx" Oct 30 21:59:59.589154 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:59.589117 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg"] Oct 30 21:59:59.593092 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:59.593076 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg" Oct 30 21:59:59.648776 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:59.648749 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-3783c2\"" Oct 30 21:59:59.652672 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:59.652650 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg"] Oct 30 21:59:59.680257 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:59.680233 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc656e82-591f-4997-9576-9297fd4dde35-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg\" (UID: \"bc656e82-591f-4997-9576-9297fd4dde35\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg" Oct 30 21:59:59.680396 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:59.680376 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prgdk\" (UniqueName: \"kubernetes.io/projected/bc656e82-591f-4997-9576-9297fd4dde35-kube-api-access-prgdk\") pod \"payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg\" (UID: \"bc656e82-591f-4997-9576-9297fd4dde35\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg" Oct 30 21:59:59.781320 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:59.781283 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prgdk\" (UniqueName: \"kubernetes.io/projected/bc656e82-591f-4997-9576-9297fd4dde35-kube-api-access-prgdk\") pod \"payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg\" (UID: \"bc656e82-591f-4997-9576-9297fd4dde35\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg" Oct 30 21:59:59.781493 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:59.781362 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc656e82-591f-4997-9576-9297fd4dde35-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg\" (UID: \"bc656e82-591f-4997-9576-9297fd4dde35\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg" Oct 30 21:59:59.782088 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:59.782068 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc656e82-591f-4997-9576-9297fd4dde35-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg\" (UID: \"bc656e82-591f-4997-9576-9297fd4dde35\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg" Oct 30 21:59:59.784112 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:59.784092 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx" Oct 30 21:59:59.847527 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:59.847496 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prgdk\" (UniqueName: \"kubernetes.io/projected/bc656e82-591f-4997-9576-9297fd4dde35-kube-api-access-prgdk\") pod \"payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg\" (UID: \"bc656e82-591f-4997-9576-9297fd4dde35\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg" Oct 30 21:59:59.905436 ip-10-0-137-81 kubenswrapper[5918]: I1030 21:59:59.905405 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg" Oct 30 22:00:00.223393 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:00.223355 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx"] Oct 30 22:00:00.261413 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:00:00.261382 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod599ae550_c43b_4094_be3d_d9a256a25a63.slice/crio-604e413602f974af43b4dd361c77bd1ce816a0a16f722d77185ef78b4835b295 WatchSource:0}: Error finding container 604e413602f974af43b4dd361c77bd1ce816a0a16f722d77185ef78b4835b295: Status 404 returned error can't find the container with id 604e413602f974af43b4dd361c77bd1ce816a0a16f722d77185ef78b4835b295 Oct 30 22:00:00.283925 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:00.283900 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg"] Oct 30 22:00:00.286122 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:00:00.286088 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc656e82_591f_4997_9576_9297fd4dde35.slice/crio-2c98168cc5a42d3c5bae63825c30efa6e7a8d9f016539b04884cee00e38267b5 WatchSource:0}: Error finding container 2c98168cc5a42d3c5bae63825c30efa6e7a8d9f016539b04884cee00e38267b5: Status 404 returned error can't find the container with id 2c98168cc5a42d3c5bae63825c30efa6e7a8d9f016539b04884cee00e38267b5 Oct 30 22:00:00.638346 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:00.638263 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx" event={"ID":"599ae550-c43b-4094-be3d-d9a256a25a63","Type":"ContainerStarted","Data":"604e413602f974af43b4dd361c77bd1ce816a0a16f722d77185ef78b4835b295"} Oct 30 22:00:00.639563 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:00.639535 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg" event={"ID":"bc656e82-591f-4997-9576-9297fd4dde35","Type":"ContainerStarted","Data":"2c98168cc5a42d3c5bae63825c30efa6e7a8d9f016539b04884cee00e38267b5"} Oct 30 22:00:01.647096 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:01.647054 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx" event={"ID":"599ae550-c43b-4094-be3d-d9a256a25a63","Type":"ContainerStarted","Data":"33372e9fcc5a4e7e13fc055fb2a5ee3771280ee3d2620fbec4e4e7c491e767c8"} Oct 30 22:00:01.647601 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:01.647233 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx" Oct 30 22:00:01.780917 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:01.780864 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx" podStartSLOduration=2.93409731 podStartE2EDuration="3.780847273s" totalImagesPullingTime="846.749963ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:01.778641658 +0000 UTC m=+562.049339738" watchObservedRunningTime="2026-10-30 22:00:01.780847273 +0000 UTC m=+562.051545420" Oct 30 22:00:03.663563 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:03.663475 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg" event={"ID":"bc656e82-591f-4997-9576-9297fd4dde35","Type":"ContainerStarted","Data":"998e6a31afecae3cd39b7f2613c46e68f16862f72489fe7b3298b7b9ef9babd1"} Oct 30 22:00:03.663563 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:03.663555 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg" Oct 30 22:00:03.784708 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:03.784655 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg" podStartSLOduration=1.841551615 podStartE2EDuration="4.784639639s" totalImagesPullingTime="2.943088024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:03.783266461 +0000 UTC m=+564.053964598" watchObservedRunningTime="2026-10-30 22:00:03.784639639 +0000 UTC m=+564.055337723" Oct 30 22:00:04.589669 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:04.589634 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q"] Oct 30 22:00:04.593426 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:04.593407 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q" Oct 30 22:00:04.648484 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:04.648457 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-3783c2-serving-cert\"" Oct 30 22:00:04.699256 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:04.699204 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q"] Oct 30 22:00:04.730114 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:04.730082 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/035da59c-1f1a-4384-9c42-43bc5d184bea-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q\" (UID: \"035da59c-1f1a-4384-9c42-43bc5d184bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q" Oct 30 22:00:04.730299 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:04.730271 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/035da59c-1f1a-4384-9c42-43bc5d184bea-maas-api-tls\") pod \"maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q\" (UID: \"035da59c-1f1a-4384-9c42-43bc5d184bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q" Oct 30 22:00:04.730448 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:04.730430 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/035da59c-1f1a-4384-9c42-43bc5d184bea-metrics-certs\") pod \"maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q\" (UID: \"035da59c-1f1a-4384-9c42-43bc5d184bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q" Oct 30 22:00:04.730520 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:04.730493 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms22t\" (UniqueName: \"kubernetes.io/projected/035da59c-1f1a-4384-9c42-43bc5d184bea-kube-api-access-ms22t\") pod \"maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q\" (UID: \"035da59c-1f1a-4384-9c42-43bc5d184bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q" Oct 30 22:00:04.831958 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:04.831922 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/035da59c-1f1a-4384-9c42-43bc5d184bea-maas-api-tls\") pod \"maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q\" (UID: \"035da59c-1f1a-4384-9c42-43bc5d184bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q" Oct 30 22:00:04.832126 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:04.832013 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/035da59c-1f1a-4384-9c42-43bc5d184bea-metrics-certs\") pod \"maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q\" (UID: \"035da59c-1f1a-4384-9c42-43bc5d184bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q" Oct 30 22:00:04.832126 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:00:04.832028 5918 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-3783c2-serving-cert: secret "maas-api-e2e-worker-w5-3783c2-serving-cert" not found Oct 30 22:00:04.832126 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:00:04.832091 5918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/035da59c-1f1a-4384-9c42-43bc5d184bea-maas-api-tls podName:035da59c-1f1a-4384-9c42-43bc5d184bea nodeName:}" failed. No retries permitted until 2026-10-30 22:00:05.332074632 +0000 UTC m=+565.602772694 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/035da59c-1f1a-4384-9c42-43bc5d184bea-maas-api-tls") pod "maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q" (UID: "035da59c-1f1a-4384-9c42-43bc5d184bea") : secret "maas-api-e2e-worker-w5-3783c2-serving-cert" not found Oct 30 22:00:04.832126 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:04.832034 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms22t\" (UniqueName: \"kubernetes.io/projected/035da59c-1f1a-4384-9c42-43bc5d184bea-kube-api-access-ms22t\") pod \"maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q\" (UID: \"035da59c-1f1a-4384-9c42-43bc5d184bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q" Oct 30 22:00:04.832362 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:04.832164 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/035da59c-1f1a-4384-9c42-43bc5d184bea-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q\" (UID: \"035da59c-1f1a-4384-9c42-43bc5d184bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q" Oct 30 22:00:04.832856 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:04.832830 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/035da59c-1f1a-4384-9c42-43bc5d184bea-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q\" (UID: \"035da59c-1f1a-4384-9c42-43bc5d184bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q" Oct 30 22:00:04.834539 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:04.834519 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/035da59c-1f1a-4384-9c42-43bc5d184bea-metrics-certs\") pod \"maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q\" (UID: \"035da59c-1f1a-4384-9c42-43bc5d184bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q" Oct 30 22:00:04.900514 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:04.900442 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms22t\" (UniqueName: \"kubernetes.io/projected/035da59c-1f1a-4384-9c42-43bc5d184bea-kube-api-access-ms22t\") pod \"maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q\" (UID: \"035da59c-1f1a-4384-9c42-43bc5d184bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q" Oct 30 22:00:05.337011 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:05.336977 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/035da59c-1f1a-4384-9c42-43bc5d184bea-maas-api-tls\") pod \"maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q\" (UID: \"035da59c-1f1a-4384-9c42-43bc5d184bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q" Oct 30 22:00:05.339692 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:05.339664 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/035da59c-1f1a-4384-9c42-43bc5d184bea-maas-api-tls\") pod \"maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q\" (UID: \"035da59c-1f1a-4384-9c42-43bc5d184bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q" Oct 30 22:00:05.505956 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:05.505918 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q" Oct 30 22:00:05.887158 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:05.887128 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q"] Oct 30 22:00:05.896409 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:00:05.896385 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod035da59c_1f1a_4384_9c42_43bc5d184bea.slice/crio-6bfb9a9c2d463d191b0e5cf2f3b8d206849dfe3a040f9a02eff7eddae25a7e99 WatchSource:0}: Error finding container 6bfb9a9c2d463d191b0e5cf2f3b8d206849dfe3a040f9a02eff7eddae25a7e99: Status 404 returned error can't find the container with id 6bfb9a9c2d463d191b0e5cf2f3b8d206849dfe3a040f9a02eff7eddae25a7e99 Oct 30 22:00:06.681472 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:06.681434 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q" event={"ID":"035da59c-1f1a-4384-9c42-43bc5d184bea","Type":"ContainerStarted","Data":"6bfb9a9c2d463d191b0e5cf2f3b8d206849dfe3a040f9a02eff7eddae25a7e99"} Oct 30 22:00:07.663596 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:07.663570 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx" Oct 30 22:00:07.688438 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:07.688408 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q" event={"ID":"035da59c-1f1a-4384-9c42-43bc5d184bea","Type":"ContainerStarted","Data":"4c1d5282cee64cf459ee1bca580fe2f1890e55d5edef498bb743ddeef686382b"} Oct 30 22:00:07.688578 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:07.688463 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q" Oct 30 22:00:07.907891 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:07.907837 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q" podStartSLOduration=3.036345602 podStartE2EDuration="3.907818231s" totalImagesPullingTime="871.472629ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:07.906531858 +0000 UTC m=+568.177229965" watchObservedRunningTime="2026-10-30 22:00:07.907818231 +0000 UTC m=+568.178516324" Oct 30 22:00:13.698953 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:13.698885 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q" Oct 30 22:00:14.672761 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:14.672732 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg" Oct 30 22:00:17.053078 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:17.053036 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs"] Oct 30 22:00:17.057066 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:17.057043 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs" Oct 30 22:00:17.115176 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:17.115150 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-6a0a09-serving-cert\"" Oct 30 22:00:17.156896 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:17.156864 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9a3c6a5-841a-49c8-afb7-7627ab28963c-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs\" (UID: \"d9a3c6a5-841a-49c8-afb7-7627ab28963c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs" Oct 30 22:00:17.157042 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:17.156903 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9a3c6a5-841a-49c8-afb7-7627ab28963c-metrics-certs\") pod \"maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs\" (UID: \"d9a3c6a5-841a-49c8-afb7-7627ab28963c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs" Oct 30 22:00:17.157089 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:17.157056 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jspdd\" (UniqueName: \"kubernetes.io/projected/d9a3c6a5-841a-49c8-afb7-7627ab28963c-kube-api-access-jspdd\") pod \"maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs\" (UID: \"d9a3c6a5-841a-49c8-afb7-7627ab28963c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs" Oct 30 22:00:17.157170 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:17.157155 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9a3c6a5-841a-49c8-afb7-7627ab28963c-maas-api-tls\") pod \"maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs\" (UID: \"d9a3c6a5-841a-49c8-afb7-7627ab28963c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs" Oct 30 22:00:17.166412 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:17.166384 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs"] Oct 30 22:00:17.257956 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:17.257927 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jspdd\" (UniqueName: \"kubernetes.io/projected/d9a3c6a5-841a-49c8-afb7-7627ab28963c-kube-api-access-jspdd\") pod \"maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs\" (UID: \"d9a3c6a5-841a-49c8-afb7-7627ab28963c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs" Oct 30 22:00:17.258109 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:17.258003 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9a3c6a5-841a-49c8-afb7-7627ab28963c-maas-api-tls\") pod \"maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs\" (UID: \"d9a3c6a5-841a-49c8-afb7-7627ab28963c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs" Oct 30 22:00:17.258109 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:17.258039 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9a3c6a5-841a-49c8-afb7-7627ab28963c-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs\" (UID: \"d9a3c6a5-841a-49c8-afb7-7627ab28963c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs" Oct 30 22:00:17.258109 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:17.258056 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9a3c6a5-841a-49c8-afb7-7627ab28963c-metrics-certs\") pod \"maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs\" (UID: \"d9a3c6a5-841a-49c8-afb7-7627ab28963c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs" Oct 30 22:00:17.258321 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:00:17.258170 5918 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-6a0a09-serving-cert: secret "maas-api-e2e-worker-w0-6a0a09-serving-cert" not found Oct 30 22:00:17.258321 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:00:17.258281 5918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9a3c6a5-841a-49c8-afb7-7627ab28963c-maas-api-tls podName:d9a3c6a5-841a-49c8-afb7-7627ab28963c nodeName:}" failed. No retries permitted until 2026-10-30 22:00:17.758257075 +0000 UTC m=+578.028955153 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d9a3c6a5-841a-49c8-afb7-7627ab28963c-maas-api-tls") pod "maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs" (UID: "d9a3c6a5-841a-49c8-afb7-7627ab28963c") : secret "maas-api-e2e-worker-w0-6a0a09-serving-cert" not found Oct 30 22:00:17.258710 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:17.258686 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9a3c6a5-841a-49c8-afb7-7627ab28963c-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs\" (UID: \"d9a3c6a5-841a-49c8-afb7-7627ab28963c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs" Oct 30 22:00:17.260546 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:17.260525 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9a3c6a5-841a-49c8-afb7-7627ab28963c-metrics-certs\") pod \"maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs\" (UID: \"d9a3c6a5-841a-49c8-afb7-7627ab28963c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs" Oct 30 22:00:17.323806 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:17.323738 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jspdd\" (UniqueName: \"kubernetes.io/projected/d9a3c6a5-841a-49c8-afb7-7627ab28963c-kube-api-access-jspdd\") pod \"maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs\" (UID: \"d9a3c6a5-841a-49c8-afb7-7627ab28963c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs" Oct 30 22:00:17.763635 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:17.763603 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9a3c6a5-841a-49c8-afb7-7627ab28963c-maas-api-tls\") pod \"maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs\" (UID: \"d9a3c6a5-841a-49c8-afb7-7627ab28963c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs" Oct 30 22:00:17.766277 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:17.766246 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9a3c6a5-841a-49c8-afb7-7627ab28963c-maas-api-tls\") pod \"maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs\" (UID: \"d9a3c6a5-841a-49c8-afb7-7627ab28963c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs" Oct 30 22:00:17.970152 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:17.970121 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs" Oct 30 22:00:18.367750 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:00:18.367716 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9a3c6a5_841a_49c8_afb7_7627ab28963c.slice/crio-e76c532c2f2157322fc362f5f2fc6e1fb06d0b1fb20f65036ba0c79332cd5fcd WatchSource:0}: Error finding container e76c532c2f2157322fc362f5f2fc6e1fb06d0b1fb20f65036ba0c79332cd5fcd: Status 404 returned error can't find the container with id e76c532c2f2157322fc362f5f2fc6e1fb06d0b1fb20f65036ba0c79332cd5fcd Oct 30 22:00:18.393571 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:18.393545 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs"] Oct 30 22:00:18.744635 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:18.744601 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs" event={"ID":"d9a3c6a5-841a-49c8-afb7-7627ab28963c","Type":"ContainerStarted","Data":"e76c532c2f2157322fc362f5f2fc6e1fb06d0b1fb20f65036ba0c79332cd5fcd"} Oct 30 22:00:19.750742 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:19.750704 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs" event={"ID":"d9a3c6a5-841a-49c8-afb7-7627ab28963c","Type":"ContainerStarted","Data":"b77251d5d864e70ea5539f8de73502a1db04cca2a1f6f63d41e8558781ea21dd"} Oct 30 22:00:19.751160 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:19.750816 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs" Oct 30 22:00:19.887528 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:19.887460 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs" podStartSLOduration=3.049143931 podStartE2EDuration="3.887440675s" totalImagesPullingTime="838.296744ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:19.885861965 +0000 UTC m=+580.156560072" watchObservedRunningTime="2026-10-30 22:00:19.887440675 +0000 UTC m=+580.158138761" Oct 30 22:00:21.274596 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:00:21.274556 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:00:25.762271 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:25.762245 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs" Oct 30 22:00:34.395137 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:34.395106 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln"] Oct 30 22:00:34.399154 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:34.399134 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln" Oct 30 22:00:34.428279 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:34.428245 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq2gs\" (UniqueName: \"kubernetes.io/projected/47da8d4b-2ae1-459b-8d7b-522745a5fd56-kube-api-access-wq2gs\") pod \"maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln\" (UID: \"47da8d4b-2ae1-459b-8d7b-522745a5fd56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln" Oct 30 22:00:34.428404 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:34.428314 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47da8d4b-2ae1-459b-8d7b-522745a5fd56-maas-api-tls\") pod \"maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln\" (UID: \"47da8d4b-2ae1-459b-8d7b-522745a5fd56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln" Oct 30 22:00:34.428404 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:34.428355 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47da8d4b-2ae1-459b-8d7b-522745a5fd56-metrics-certs\") pod \"maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln\" (UID: \"47da8d4b-2ae1-459b-8d7b-522745a5fd56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln" Oct 30 22:00:34.428404 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:34.428382 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47da8d4b-2ae1-459b-8d7b-522745a5fd56-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln\" (UID: \"47da8d4b-2ae1-459b-8d7b-522745a5fd56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln" Oct 30 22:00:34.453565 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:34.453543 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-9cff60-serving-cert\"" Oct 30 22:00:34.505038 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:34.504997 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln"] Oct 30 22:00:34.530056 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:34.530020 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq2gs\" (UniqueName: \"kubernetes.io/projected/47da8d4b-2ae1-459b-8d7b-522745a5fd56-kube-api-access-wq2gs\") pod \"maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln\" (UID: \"47da8d4b-2ae1-459b-8d7b-522745a5fd56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln" Oct 30 22:00:34.530250 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:34.530105 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47da8d4b-2ae1-459b-8d7b-522745a5fd56-maas-api-tls\") pod \"maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln\" (UID: \"47da8d4b-2ae1-459b-8d7b-522745a5fd56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln" Oct 30 22:00:34.530250 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:34.530173 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47da8d4b-2ae1-459b-8d7b-522745a5fd56-metrics-certs\") pod \"maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln\" (UID: \"47da8d4b-2ae1-459b-8d7b-522745a5fd56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln" Oct 30 22:00:34.530250 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:34.530201 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47da8d4b-2ae1-459b-8d7b-522745a5fd56-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln\" (UID: \"47da8d4b-2ae1-459b-8d7b-522745a5fd56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln" Oct 30 22:00:34.530431 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:00:34.530264 5918 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-9cff60-serving-cert: secret "maas-api-e2e-worker-w1-9cff60-serving-cert" not found Oct 30 22:00:34.530431 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:00:34.530359 5918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47da8d4b-2ae1-459b-8d7b-522745a5fd56-maas-api-tls podName:47da8d4b-2ae1-459b-8d7b-522745a5fd56 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:35.030335538 +0000 UTC m=+595.301033616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/47da8d4b-2ae1-459b-8d7b-522745a5fd56-maas-api-tls") pod "maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln" (UID: "47da8d4b-2ae1-459b-8d7b-522745a5fd56") : secret "maas-api-e2e-worker-w1-9cff60-serving-cert" not found Oct 30 22:00:34.531078 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:34.530996 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47da8d4b-2ae1-459b-8d7b-522745a5fd56-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln\" (UID: \"47da8d4b-2ae1-459b-8d7b-522745a5fd56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln" Oct 30 22:00:34.532755 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:34.532735 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47da8d4b-2ae1-459b-8d7b-522745a5fd56-metrics-certs\") pod \"maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln\" (UID: \"47da8d4b-2ae1-459b-8d7b-522745a5fd56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln" Oct 30 22:00:34.595495 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:34.595460 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq2gs\" (UniqueName: \"kubernetes.io/projected/47da8d4b-2ae1-459b-8d7b-522745a5fd56-kube-api-access-wq2gs\") pod \"maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln\" (UID: \"47da8d4b-2ae1-459b-8d7b-522745a5fd56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln" Oct 30 22:00:35.037393 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:35.037360 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47da8d4b-2ae1-459b-8d7b-522745a5fd56-maas-api-tls\") pod \"maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln\" (UID: \"47da8d4b-2ae1-459b-8d7b-522745a5fd56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln" Oct 30 22:00:35.039891 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:35.039870 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47da8d4b-2ae1-459b-8d7b-522745a5fd56-maas-api-tls\") pod \"maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln\" (UID: \"47da8d4b-2ae1-459b-8d7b-522745a5fd56\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln" Oct 30 22:00:35.312232 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:35.312131 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln" Oct 30 22:00:35.708867 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:35.708829 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln"] Oct 30 22:00:35.710034 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:00:35.710010 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47da8d4b_2ae1_459b_8d7b_522745a5fd56.slice/crio-b0c95bb4b102502284e0890767e54a152d24fe620d282713795f637a3609e0b2 WatchSource:0}: Error finding container b0c95bb4b102502284e0890767e54a152d24fe620d282713795f637a3609e0b2: Status 404 returned error can't find the container with id b0c95bb4b102502284e0890767e54a152d24fe620d282713795f637a3609e0b2 Oct 30 22:00:35.833172 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:35.833140 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln" event={"ID":"47da8d4b-2ae1-459b-8d7b-522745a5fd56","Type":"ContainerStarted","Data":"b0c95bb4b102502284e0890767e54a152d24fe620d282713795f637a3609e0b2"} Oct 30 22:00:36.840073 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:36.839979 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln" event={"ID":"47da8d4b-2ae1-459b-8d7b-522745a5fd56","Type":"ContainerStarted","Data":"2b5954cbfad37b2dab0001550fe20c77108fe62ed85acbd08c631b7a81b8eb7f"} Oct 30 22:00:36.840588 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:36.840136 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln" Oct 30 22:00:36.982267 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:36.982195 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln" podStartSLOduration=2.107300716 podStartE2EDuration="2.982178795s" totalImagesPullingTime="874.878079ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:36.982042237 +0000 UTC m=+597.252740323" watchObservedRunningTime="2026-10-30 22:00:36.982178795 +0000 UTC m=+597.252876877" Oct 30 22:00:42.852550 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:42.852523 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln" Oct 30 22:00:51.275530 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:00:51.275483 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:00:53.951422 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:53.951335 5918 generic.go:356] "Generic (PLEG): container finished" podID="f6f37484-37b2-4fab-8b6b-cdc1c171ff93" containerID="b3d14947d3b89637bb8d139bdff16d91ff5cd35a461de9324822e4b4fbbfc0d7" exitCode=0 Oct 30 22:00:53.951422 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:53.951408 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" event={"ID":"f6f37484-37b2-4fab-8b6b-cdc1c171ff93","Type":"ContainerDied","Data":"b3d14947d3b89637bb8d139bdff16d91ff5cd35a461de9324822e4b4fbbfc0d7"} Oct 30 22:00:53.951869 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:53.951778 5918 scope.go:122] "RemoveContainer" containerID="b3d14947d3b89637bb8d139bdff16d91ff5cd35a461de9324822e4b4fbbfc0d7" Oct 30 22:00:53.953239 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:53.953186 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:53.953353 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:53.953313 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:53.953417 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:53.953363 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:54.959718 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:54.959687 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" event={"ID":"f6f37484-37b2-4fab-8b6b-cdc1c171ff93","Type":"ContainerStarted","Data":"cadd378cbd16596b10517a87e400a134074a3a4b3959ccd5c5b27a27bbe420e5"} Oct 30 22:00:55.207661 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:55.207624 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 22:00:55.208019 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:55.207938 5918 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 22:00:55.208164 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:55.208141 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" podUID="f6f37484-37b2-4fab-8b6b-cdc1c171ff93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 22:00:56.211617 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:56.211578 5918 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 22:00:56.212046 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:56.211656 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" podUID="f6f37484-37b2-4fab-8b6b-cdc1c171ff93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 22:00:57.207629 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:57.207591 5918 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 22:00:57.207919 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:57.207889 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" podUID="f6f37484-37b2-4fab-8b6b-cdc1c171ff93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 22:00:58.211242 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:58.209619 5918 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 22:00:58.211242 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:58.209681 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" podUID="f6f37484-37b2-4fab-8b6b-cdc1c171ff93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 22:00:59.208072 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:59.207591 5918 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 22:00:59.208072 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:00:59.207694 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" podUID="f6f37484-37b2-4fab-8b6b-cdc1c171ff93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 22:01:00.211691 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:00.211657 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 22:01:00.212097 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:00.211935 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 22:01:00.212735 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:00.212713 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 22:01:07.311368 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:07.311325 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d"] Oct 30 22:01:07.314267 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:07.314245 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" Oct 30 22:01:07.419460 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:07.419422 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e63fad41-0e42-41ee-ae20-919f15fffe55-dshm\") pod \"simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d\" (UID: \"e63fad41-0e42-41ee-ae20-919f15fffe55\") " pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" Oct 30 22:01:07.419736 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:07.419490 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e63fad41-0e42-41ee-ae20-919f15fffe55-model-cache\") pod \"simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d\" (UID: \"e63fad41-0e42-41ee-ae20-919f15fffe55\") " pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" Oct 30 22:01:07.419736 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:07.419687 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e63fad41-0e42-41ee-ae20-919f15fffe55-tls-certs\") pod \"simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d\" (UID: \"e63fad41-0e42-41ee-ae20-919f15fffe55\") " pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" Oct 30 22:01:07.419736 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:07.419724 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8ca52a\"/\"default-dockercfg-7cw79\"" Oct 30 22:01:07.419897 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:07.419751 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8ca52a\"/\"kube-root-ca.crt\"" Oct 30 22:01:07.419897 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:07.419759 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8ca52a\"/\"simulator-w3-8ca52a-kserve-self-signed-certs\"" Oct 30 22:01:07.419897 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:07.419789 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8ca52a\"/\"openshift-service-ca.crt\"" Oct 30 22:01:07.419897 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:07.419796 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e63fad41-0e42-41ee-ae20-919f15fffe55-tmp-dir\") pod \"simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d\" (UID: \"e63fad41-0e42-41ee-ae20-919f15fffe55\") " pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" Oct 30 22:01:07.419897 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:07.419840 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjk5d\" (UniqueName: \"kubernetes.io/projected/e63fad41-0e42-41ee-ae20-919f15fffe55-kube-api-access-mjk5d\") pod \"simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d\" (UID: \"e63fad41-0e42-41ee-ae20-919f15fffe55\") " pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" Oct 30 22:01:07.420100 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:07.419898 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e63fad41-0e42-41ee-ae20-919f15fffe55-home\") pod \"simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d\" (UID: \"e63fad41-0e42-41ee-ae20-919f15fffe55\") " pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" Oct 30 22:01:07.422017 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:07.421996 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d"] Oct 30 22:01:07.520764 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:07.520733 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e63fad41-0e42-41ee-ae20-919f15fffe55-dshm\") pod \"simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d\" (UID: \"e63fad41-0e42-41ee-ae20-919f15fffe55\") " pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" Oct 30 22:01:07.520925 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:07.520784 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e63fad41-0e42-41ee-ae20-919f15fffe55-model-cache\") pod \"simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d\" (UID: \"e63fad41-0e42-41ee-ae20-919f15fffe55\") " pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" Oct 30 22:01:07.520925 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:07.520860 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e63fad41-0e42-41ee-ae20-919f15fffe55-tls-certs\") pod \"simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d\" (UID: \"e63fad41-0e42-41ee-ae20-919f15fffe55\") " pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" Oct 30 22:01:07.520925 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:07.520901 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e63fad41-0e42-41ee-ae20-919f15fffe55-tmp-dir\") pod \"simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d\" (UID: \"e63fad41-0e42-41ee-ae20-919f15fffe55\") " pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" Oct 30 22:01:07.521089 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:07.520938 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjk5d\" (UniqueName: \"kubernetes.io/projected/e63fad41-0e42-41ee-ae20-919f15fffe55-kube-api-access-mjk5d\") pod \"simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d\" (UID: \"e63fad41-0e42-41ee-ae20-919f15fffe55\") " pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" Oct 30 22:01:07.521089 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:07.520967 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e63fad41-0e42-41ee-ae20-919f15fffe55-home\") pod \"simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d\" (UID: \"e63fad41-0e42-41ee-ae20-919f15fffe55\") " pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" Oct 30 22:01:07.521290 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:07.521266 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e63fad41-0e42-41ee-ae20-919f15fffe55-model-cache\") pod \"simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d\" (UID: \"e63fad41-0e42-41ee-ae20-919f15fffe55\") " pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" Oct 30 22:01:07.521407 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:07.521312 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e63fad41-0e42-41ee-ae20-919f15fffe55-tmp-dir\") pod \"simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d\" (UID: \"e63fad41-0e42-41ee-ae20-919f15fffe55\") " pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" Oct 30 22:01:07.521407 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:07.521390 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e63fad41-0e42-41ee-ae20-919f15fffe55-home\") pod \"simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d\" (UID: \"e63fad41-0e42-41ee-ae20-919f15fffe55\") " pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" Oct 30 22:01:07.523029 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:07.523003 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e63fad41-0e42-41ee-ae20-919f15fffe55-dshm\") pod \"simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d\" (UID: \"e63fad41-0e42-41ee-ae20-919f15fffe55\") " pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" Oct 30 22:01:07.523400 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:07.523380 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e63fad41-0e42-41ee-ae20-919f15fffe55-tls-certs\") pod \"simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d\" (UID: \"e63fad41-0e42-41ee-ae20-919f15fffe55\") " pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" Oct 30 22:01:07.587639 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:07.587566 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjk5d\" (UniqueName: \"kubernetes.io/projected/e63fad41-0e42-41ee-ae20-919f15fffe55-kube-api-access-mjk5d\") pod \"simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d\" (UID: \"e63fad41-0e42-41ee-ae20-919f15fffe55\") " pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" Oct 30 22:01:07.625741 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:07.625701 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" Oct 30 22:01:08.004158 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:08.004130 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d"] Oct 30 22:01:08.008782 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:01:08.008750 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode63fad41_0e42_41ee_ae20_919f15fffe55.slice/crio-b97d4f1a3493cab2bc97dfde29b65e12290e83bedb2d674090064b8f3da4fc43 WatchSource:0}: Error finding container b97d4f1a3493cab2bc97dfde29b65e12290e83bedb2d674090064b8f3da4fc43: Status 404 returned error can't find the container with id b97d4f1a3493cab2bc97dfde29b65e12290e83bedb2d674090064b8f3da4fc43 Oct 30 22:01:08.028458 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:08.028428 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" event={"ID":"e63fad41-0e42-41ee-ae20-919f15fffe55","Type":"ContainerStarted","Data":"b97d4f1a3493cab2bc97dfde29b65e12290e83bedb2d674090064b8f3da4fc43"} Oct 30 22:01:11.048801 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:11.048762 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" event={"ID":"e63fad41-0e42-41ee-ae20-919f15fffe55","Type":"ContainerStarted","Data":"3db2b01524a206c993016abd96340146235b614665b091d2842a0e7a398612cd"} Oct 30 22:01:11.195594 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:11.195538 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" podStartSLOduration=2.083936935 podStartE2EDuration="4.195523865s" totalImagesPullingTime="2.11158693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:11.194451215 +0000 UTC m=+631.465149310" watchObservedRunningTime="2026-10-30 22:01:11.195523865 +0000 UTC m=+631.466221951" Oct 30 22:01:11.626398 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:11.626361 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" Oct 30 22:01:17.626571 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:17.626527 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" Oct 30 22:01:17.640088 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:17.640057 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" Oct 30 22:01:18.094833 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:18.094807 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" Oct 30 22:01:21.273982 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:01:21.273897 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:01:37.030150 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:37.030117 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp"] Oct 30 22:01:37.033830 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:37.033814 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" Oct 30 22:01:37.136086 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:37.136053 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5ae2434-f161-4a5c-8086-2f0711678bb4-model-cache\") pod \"simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp\" (UID: \"f5ae2434-f161-4a5c-8086-2f0711678bb4\") " pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" Oct 30 22:01:37.136293 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:37.136249 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5ae2434-f161-4a5c-8086-2f0711678bb4-tls-certs\") pod \"simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp\" (UID: \"f5ae2434-f161-4a5c-8086-2f0711678bb4\") " pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" Oct 30 22:01:37.136293 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:37.136278 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5ae2434-f161-4a5c-8086-2f0711678bb4-tmp-dir\") pod \"simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp\" (UID: \"f5ae2434-f161-4a5c-8086-2f0711678bb4\") " pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" Oct 30 22:01:37.136405 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:37.136308 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5ae2434-f161-4a5c-8086-2f0711678bb4-home\") pod \"simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp\" (UID: \"f5ae2434-f161-4a5c-8086-2f0711678bb4\") " pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" Oct 30 22:01:37.136453 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:37.136420 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5ae2434-f161-4a5c-8086-2f0711678bb4-dshm\") pod \"simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp\" (UID: \"f5ae2434-f161-4a5c-8086-2f0711678bb4\") " pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" Oct 30 22:01:37.136453 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:37.136442 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqqmk\" (UniqueName: \"kubernetes.io/projected/f5ae2434-f161-4a5c-8086-2f0711678bb4-kube-api-access-dqqmk\") pod \"simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp\" (UID: \"f5ae2434-f161-4a5c-8086-2f0711678bb4\") " pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" Oct 30 22:01:37.138597 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:37.138564 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6a0a09\"/\"simulator-w0-6a0a09-kserve-self-signed-certs\"" Oct 30 22:01:37.138698 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:37.138636 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-6a0a09\"/\"openshift-service-ca.crt\"" Oct 30 22:01:37.138698 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:37.138636 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6a0a09\"/\"default-dockercfg-k92f6\"" Oct 30 22:01:37.138698 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:37.138677 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-6a0a09\"/\"kube-root-ca.crt\"" Oct 30 22:01:37.141124 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:37.141102 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp"] Oct 30 22:01:37.237693 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:37.237661 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5ae2434-f161-4a5c-8086-2f0711678bb4-tls-certs\") pod \"simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp\" (UID: \"f5ae2434-f161-4a5c-8086-2f0711678bb4\") " pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" Oct 30 22:01:37.237693 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:37.237697 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5ae2434-f161-4a5c-8086-2f0711678bb4-tmp-dir\") pod \"simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp\" (UID: \"f5ae2434-f161-4a5c-8086-2f0711678bb4\") " pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" Oct 30 22:01:37.237965 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:37.237724 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5ae2434-f161-4a5c-8086-2f0711678bb4-home\") pod \"simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp\" (UID: \"f5ae2434-f161-4a5c-8086-2f0711678bb4\") " pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" Oct 30 22:01:37.237965 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:37.237779 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5ae2434-f161-4a5c-8086-2f0711678bb4-dshm\") pod \"simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp\" (UID: \"f5ae2434-f161-4a5c-8086-2f0711678bb4\") " pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" Oct 30 22:01:37.237965 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:37.237807 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqqmk\" (UniqueName: \"kubernetes.io/projected/f5ae2434-f161-4a5c-8086-2f0711678bb4-kube-api-access-dqqmk\") pod \"simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp\" (UID: \"f5ae2434-f161-4a5c-8086-2f0711678bb4\") " pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" Oct 30 22:01:37.237965 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:37.237842 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5ae2434-f161-4a5c-8086-2f0711678bb4-model-cache\") pod \"simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp\" (UID: \"f5ae2434-f161-4a5c-8086-2f0711678bb4\") " pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" Oct 30 22:01:37.238309 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:37.238282 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5ae2434-f161-4a5c-8086-2f0711678bb4-model-cache\") pod \"simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp\" (UID: \"f5ae2434-f161-4a5c-8086-2f0711678bb4\") " pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" Oct 30 22:01:37.238432 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:37.238316 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5ae2434-f161-4a5c-8086-2f0711678bb4-home\") pod \"simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp\" (UID: \"f5ae2434-f161-4a5c-8086-2f0711678bb4\") " pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" Oct 30 22:01:37.238432 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:37.238373 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5ae2434-f161-4a5c-8086-2f0711678bb4-tmp-dir\") pod \"simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp\" (UID: \"f5ae2434-f161-4a5c-8086-2f0711678bb4\") " pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" Oct 30 22:01:37.240588 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:37.240563 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5ae2434-f161-4a5c-8086-2f0711678bb4-tls-certs\") pod \"simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp\" (UID: \"f5ae2434-f161-4a5c-8086-2f0711678bb4\") " pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" Oct 30 22:01:37.240658 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:37.240609 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5ae2434-f161-4a5c-8086-2f0711678bb4-dshm\") pod \"simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp\" (UID: \"f5ae2434-f161-4a5c-8086-2f0711678bb4\") " pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" Oct 30 22:01:37.302650 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:37.302572 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqqmk\" (UniqueName: \"kubernetes.io/projected/f5ae2434-f161-4a5c-8086-2f0711678bb4-kube-api-access-dqqmk\") pod \"simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp\" (UID: \"f5ae2434-f161-4a5c-8086-2f0711678bb4\") " pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" Oct 30 22:01:37.346285 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:37.346243 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" Oct 30 22:01:37.732359 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:37.732329 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp"] Oct 30 22:01:37.733946 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:01:37.733921 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5ae2434_f161_4a5c_8086_2f0711678bb4.slice/crio-97bd66c9fc59bc7269c4e58207ead9982082ac6ab8f397e56d8a2708e608f249 WatchSource:0}: Error finding container 97bd66c9fc59bc7269c4e58207ead9982082ac6ab8f397e56d8a2708e608f249: Status 404 returned error can't find the container with id 97bd66c9fc59bc7269c4e58207ead9982082ac6ab8f397e56d8a2708e608f249 Oct 30 22:01:38.195452 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:38.195414 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" event={"ID":"f5ae2434-f161-4a5c-8086-2f0711678bb4","Type":"ContainerStarted","Data":"24eff2da57e35ab3006bd0b98e55a9db408b1d66be3971dc22a9c68228f6f4e9"} Oct 30 22:01:38.195835 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:38.195461 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" event={"ID":"f5ae2434-f161-4a5c-8086-2f0711678bb4","Type":"ContainerStarted","Data":"97bd66c9fc59bc7269c4e58207ead9982082ac6ab8f397e56d8a2708e608f249"} Oct 30 22:01:38.335272 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:38.335188 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" podStartSLOduration=2.335173779 podStartE2EDuration="2.335173779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:38.332519136 +0000 UTC m=+658.603217221" watchObservedRunningTime="2026-10-30 22:01:38.335173779 +0000 UTC m=+658.605871862" Oct 30 22:01:38.346562 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:38.346536 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" Oct 30 22:01:47.346680 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:47.346647 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" Oct 30 22:01:47.359859 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:47.359837 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" Oct 30 22:01:48.260564 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:01:48.260536 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" Oct 30 22:01:51.275480 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:01:51.275440 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:02:01.873566 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:01.873529 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc"] Oct 30 22:02:01.877187 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:01.877164 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" Oct 30 22:02:01.981533 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:01.981490 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-9cff60\"/\"openshift-service-ca.crt\"" Oct 30 22:02:01.981533 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:01.981524 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-9cff60\"/\"kube-root-ca.crt\"" Oct 30 22:02:01.981892 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:01.981591 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-9cff60\"/\"default-dockercfg-gwrpm\"" Oct 30 22:02:01.981892 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:01.981600 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-9cff60\"/\"simulator-w1-9cff60-kserve-self-signed-certs\"" Oct 30 22:02:01.984141 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:01.984120 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc"] Oct 30 22:02:01.997170 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:01.997131 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-dshm\") pod \"simulator-w1-9cff60-kserve-7c7f9c446b-n79jc\" (UID: \"21ef7a75-4005-4d56-9b35-783dc4d9cd2a\") " pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" Oct 30 22:02:01.997347 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:01.997323 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-tmp-dir\") pod \"simulator-w1-9cff60-kserve-7c7f9c446b-n79jc\" (UID: \"21ef7a75-4005-4d56-9b35-783dc4d9cd2a\") " pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" Oct 30 22:02:01.997461 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:01.997443 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-tls-certs\") pod \"simulator-w1-9cff60-kserve-7c7f9c446b-n79jc\" (UID: \"21ef7a75-4005-4d56-9b35-783dc4d9cd2a\") " pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" Oct 30 22:02:01.997519 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:01.997482 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-home\") pod \"simulator-w1-9cff60-kserve-7c7f9c446b-n79jc\" (UID: \"21ef7a75-4005-4d56-9b35-783dc4d9cd2a\") " pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" Oct 30 22:02:01.997519 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:01.997499 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjfnr\" (UniqueName: \"kubernetes.io/projected/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-kube-api-access-jjfnr\") pod \"simulator-w1-9cff60-kserve-7c7f9c446b-n79jc\" (UID: \"21ef7a75-4005-4d56-9b35-783dc4d9cd2a\") " pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" Oct 30 22:02:01.997627 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:01.997519 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-model-cache\") pod \"simulator-w1-9cff60-kserve-7c7f9c446b-n79jc\" (UID: \"21ef7a75-4005-4d56-9b35-783dc4d9cd2a\") " pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" Oct 30 22:02:02.098776 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:02.098732 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-dshm\") pod \"simulator-w1-9cff60-kserve-7c7f9c446b-n79jc\" (UID: \"21ef7a75-4005-4d56-9b35-783dc4d9cd2a\") " pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" Oct 30 22:02:02.098990 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:02.098797 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-tmp-dir\") pod \"simulator-w1-9cff60-kserve-7c7f9c446b-n79jc\" (UID: \"21ef7a75-4005-4d56-9b35-783dc4d9cd2a\") " pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" Oct 30 22:02:02.098990 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:02.098851 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-tls-certs\") pod \"simulator-w1-9cff60-kserve-7c7f9c446b-n79jc\" (UID: \"21ef7a75-4005-4d56-9b35-783dc4d9cd2a\") " pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" Oct 30 22:02:02.098990 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:02.098899 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-home\") pod \"simulator-w1-9cff60-kserve-7c7f9c446b-n79jc\" (UID: \"21ef7a75-4005-4d56-9b35-783dc4d9cd2a\") " pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" Oct 30 22:02:02.098990 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:02.098930 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjfnr\" (UniqueName: \"kubernetes.io/projected/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-kube-api-access-jjfnr\") pod \"simulator-w1-9cff60-kserve-7c7f9c446b-n79jc\" (UID: \"21ef7a75-4005-4d56-9b35-783dc4d9cd2a\") " pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" Oct 30 22:02:02.098990 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:02.098953 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-model-cache\") pod \"simulator-w1-9cff60-kserve-7c7f9c446b-n79jc\" (UID: \"21ef7a75-4005-4d56-9b35-783dc4d9cd2a\") " pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" Oct 30 22:02:02.099365 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:02.099338 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-tmp-dir\") pod \"simulator-w1-9cff60-kserve-7c7f9c446b-n79jc\" (UID: \"21ef7a75-4005-4d56-9b35-783dc4d9cd2a\") " pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" Oct 30 22:02:02.099448 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:02.099425 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-home\") pod \"simulator-w1-9cff60-kserve-7c7f9c446b-n79jc\" (UID: \"21ef7a75-4005-4d56-9b35-783dc4d9cd2a\") " pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" Oct 30 22:02:02.099515 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:02.099472 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-model-cache\") pod \"simulator-w1-9cff60-kserve-7c7f9c446b-n79jc\" (UID: \"21ef7a75-4005-4d56-9b35-783dc4d9cd2a\") " pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" Oct 30 22:02:02.101307 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:02.101279 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-dshm\") pod \"simulator-w1-9cff60-kserve-7c7f9c446b-n79jc\" (UID: \"21ef7a75-4005-4d56-9b35-783dc4d9cd2a\") " pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" Oct 30 22:02:02.101541 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:02.101523 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-tls-certs\") pod \"simulator-w1-9cff60-kserve-7c7f9c446b-n79jc\" (UID: \"21ef7a75-4005-4d56-9b35-783dc4d9cd2a\") " pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" Oct 30 22:02:02.172869 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:02.172779 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjfnr\" (UniqueName: \"kubernetes.io/projected/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-kube-api-access-jjfnr\") pod \"simulator-w1-9cff60-kserve-7c7f9c446b-n79jc\" (UID: \"21ef7a75-4005-4d56-9b35-783dc4d9cd2a\") " pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" Oct 30 22:02:02.189873 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:02.189832 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" Oct 30 22:02:02.588716 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:02.588692 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc"] Oct 30 22:02:02.590820 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:02:02.590791 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21ef7a75_4005_4d56_9b35_783dc4d9cd2a.slice/crio-d704f3b0464688bdd244e7bab42514c8313a0953dc3140d89267c7ac7fceb73a WatchSource:0}: Error finding container d704f3b0464688bdd244e7bab42514c8313a0953dc3140d89267c7ac7fceb73a: Status 404 returned error can't find the container with id d704f3b0464688bdd244e7bab42514c8313a0953dc3140d89267c7ac7fceb73a Oct 30 22:02:03.332639 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:03.332598 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" event={"ID":"21ef7a75-4005-4d56-9b35-783dc4d9cd2a","Type":"ContainerStarted","Data":"9dc189e679b85d3f1848c3085a7f7fe37aa4d302f3af979411743e95a16ac390"} Oct 30 22:02:03.332639 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:03.332652 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" event={"ID":"21ef7a75-4005-4d56-9b35-783dc4d9cd2a","Type":"ContainerStarted","Data":"d704f3b0464688bdd244e7bab42514c8313a0953dc3140d89267c7ac7fceb73a"} Oct 30 22:02:03.452754 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:03.452702 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" podStartSLOduration=2.452688478 podStartE2EDuration="2.452688478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:03.45178314 +0000 UTC m=+683.722481277" watchObservedRunningTime="2026-10-30 22:02:03.452688478 +0000 UTC m=+683.723386562" Oct 30 22:02:04.190968 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:04.190936 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" Oct 30 22:02:12.190044 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:12.189995 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" Oct 30 22:02:12.202853 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:12.202817 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" Oct 30 22:02:12.395198 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:12.395164 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" Oct 30 22:02:21.275467 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:02:21.275429 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:02:45.087383 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:45.087342 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7"] Oct 30 22:02:45.091320 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:45.091298 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" Oct 30 22:02:45.146195 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:45.146168 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-3783c2\"/\"kube-root-ca.crt\"" Oct 30 22:02:45.147706 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:45.147680 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-home\") pod \"premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7\" (UID: \"1a1ae368-0e8c-408d-bfcd-8f4a8194464f\") " pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" Oct 30 22:02:45.147862 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:45.147758 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-tls-certs\") pod \"premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7\" (UID: \"1a1ae368-0e8c-408d-bfcd-8f4a8194464f\") " pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" Oct 30 22:02:45.147862 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:45.147849 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-tmp-dir\") pod \"premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7\" (UID: \"1a1ae368-0e8c-408d-bfcd-8f4a8194464f\") " pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" Oct 30 22:02:45.147969 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:45.147895 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-dshm\") pod \"premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7\" (UID: \"1a1ae368-0e8c-408d-bfcd-8f4a8194464f\") " pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" Oct 30 22:02:45.148010 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:45.147978 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5q4f\" (UniqueName: \"kubernetes.io/projected/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-kube-api-access-x5q4f\") pod \"premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7\" (UID: \"1a1ae368-0e8c-408d-bfcd-8f4a8194464f\") " pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" Oct 30 22:02:45.148010 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:45.148004 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-model-cache\") pod \"premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7\" (UID: \"1a1ae368-0e8c-408d-bfcd-8f4a8194464f\") " pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" Oct 30 22:02:45.195487 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:45.195448 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-3783c2\"/\"openshift-service-ca.crt\"" Oct 30 22:02:45.195487 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:45.195451 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-3783c2\"/\"default-dockercfg-f8swn\"" Oct 30 22:02:45.195695 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:45.195666 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-3783c2\"/\"premium-w5-3783c2-kserve-self-signed-certs\"" Oct 30 22:02:45.197829 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:45.197811 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7"] Oct 30 22:02:45.248968 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:45.248935 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-home\") pod \"premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7\" (UID: \"1a1ae368-0e8c-408d-bfcd-8f4a8194464f\") " pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" Oct 30 22:02:45.249171 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:45.249003 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-tls-certs\") pod \"premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7\" (UID: \"1a1ae368-0e8c-408d-bfcd-8f4a8194464f\") " pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" Oct 30 22:02:45.249171 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:45.249029 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-tmp-dir\") pod \"premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7\" (UID: \"1a1ae368-0e8c-408d-bfcd-8f4a8194464f\") " pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" Oct 30 22:02:45.249171 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:45.249054 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-dshm\") pod \"premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7\" (UID: \"1a1ae368-0e8c-408d-bfcd-8f4a8194464f\") " pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" Oct 30 22:02:45.249171 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:45.249082 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5q4f\" (UniqueName: \"kubernetes.io/projected/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-kube-api-access-x5q4f\") pod \"premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7\" (UID: \"1a1ae368-0e8c-408d-bfcd-8f4a8194464f\") " pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" Oct 30 22:02:45.249171 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:45.249099 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-model-cache\") pod \"premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7\" (UID: \"1a1ae368-0e8c-408d-bfcd-8f4a8194464f\") " pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" Oct 30 22:02:45.249544 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:45.249516 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-home\") pod \"premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7\" (UID: \"1a1ae368-0e8c-408d-bfcd-8f4a8194464f\") " pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" Oct 30 22:02:45.249606 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:45.249590 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-tmp-dir\") pod \"premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7\" (UID: \"1a1ae368-0e8c-408d-bfcd-8f4a8194464f\") " pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" Oct 30 22:02:45.249659 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:45.249602 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-model-cache\") pod \"premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7\" (UID: \"1a1ae368-0e8c-408d-bfcd-8f4a8194464f\") " pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" Oct 30 22:02:45.251561 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:45.251533 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-dshm\") pod \"premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7\" (UID: \"1a1ae368-0e8c-408d-bfcd-8f4a8194464f\") " pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" Oct 30 22:02:45.251891 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:45.251875 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-tls-certs\") pod \"premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7\" (UID: \"1a1ae368-0e8c-408d-bfcd-8f4a8194464f\") " pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" Oct 30 22:02:45.317871 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:45.317846 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5q4f\" (UniqueName: \"kubernetes.io/projected/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-kube-api-access-x5q4f\") pod \"premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7\" (UID: \"1a1ae368-0e8c-408d-bfcd-8f4a8194464f\") " pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" Oct 30 22:02:45.404448 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:45.404355 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" Oct 30 22:02:45.800324 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:45.800298 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7"] Oct 30 22:02:45.802062 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:02:45.802035 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a1ae368_0e8c_408d_bfcd_8f4a8194464f.slice/crio-cf6888bafbe6361d1ff63975e91ebd8b9db291cd3f14f22b1f44d9c4aedfee5d WatchSource:0}: Error finding container cf6888bafbe6361d1ff63975e91ebd8b9db291cd3f14f22b1f44d9c4aedfee5d: Status 404 returned error can't find the container with id cf6888bafbe6361d1ff63975e91ebd8b9db291cd3f14f22b1f44d9c4aedfee5d Oct 30 22:02:46.565385 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:46.565347 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" event={"ID":"1a1ae368-0e8c-408d-bfcd-8f4a8194464f","Type":"ContainerStarted","Data":"2cbd5b84590571170aa2cce358162c27b1d27732e139afd2f596b7ce8a387bc3"} Oct 30 22:02:46.565385 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:46.565390 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" event={"ID":"1a1ae368-0e8c-408d-bfcd-8f4a8194464f","Type":"ContainerStarted","Data":"cf6888bafbe6361d1ff63975e91ebd8b9db291cd3f14f22b1f44d9c4aedfee5d"} Oct 30 22:02:46.686826 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:46.686771 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" podStartSLOduration=1.6867563840000002 podStartE2EDuration="1.686756384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:46.685590022 +0000 UTC m=+726.956288110" watchObservedRunningTime="2026-10-30 22:02:46.686756384 +0000 UTC m=+726.957454467" Oct 30 22:02:47.405044 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:47.405006 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" Oct 30 22:02:51.274668 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:02:51.274580 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:02:55.405201 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:55.405169 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" Oct 30 22:02:55.418468 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:55.418441 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" Oct 30 22:02:55.627895 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:02:55.627867 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" Oct 30 22:03:14.108732 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:14.108696 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh"] Oct 30 22:03:14.112684 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:14.112665 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" Oct 30 22:03:14.167776 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:14.167744 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-d8837a-openshift-default-dockercfg-p6b8g\"" Oct 30 22:03:14.169474 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:14.169453 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-istio-data\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh\" (UID: \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" Oct 30 22:03:14.169553 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:14.169494 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-istio-podinfo\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh\" (UID: \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" Oct 30 22:03:14.169553 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:14.169516 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-istiod-ca-cert\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh\" (UID: \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" Oct 30 22:03:14.169658 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:14.169636 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-istio-token\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh\" (UID: \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" Oct 30 22:03:14.169708 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:14.169690 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-istio-envoy\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh\" (UID: \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" Oct 30 22:03:14.169746 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:14.169729 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-credential-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh\" (UID: \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" Oct 30 22:03:14.169801 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:14.169747 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-workload-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh\" (UID: \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" Oct 30 22:03:14.169801 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:14.169776 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75vgf\" (UniqueName: \"kubernetes.io/projected/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-kube-api-access-75vgf\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh\" (UID: \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" Oct 30 22:03:14.169890 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:14.169876 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-workload-certs\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh\" (UID: \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" Oct 30 22:03:14.219006 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:14.218971 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh"] Oct 30 22:03:14.271399 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:14.271361 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-istio-podinfo\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh\" (UID: \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" Oct 30 22:03:14.271399 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:14.271405 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-istiod-ca-cert\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh\" (UID: \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" Oct 30 22:03:14.271637 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:14.271456 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-istio-token\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh\" (UID: \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" Oct 30 22:03:14.271637 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:14.271485 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-istio-envoy\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh\" (UID: \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" Oct 30 22:03:14.271637 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:14.271541 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-credential-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh\" (UID: \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" Oct 30 22:03:14.271637 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:14.271563 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-workload-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh\" (UID: \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" Oct 30 22:03:14.271637 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:14.271601 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75vgf\" (UniqueName: \"kubernetes.io/projected/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-kube-api-access-75vgf\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh\" (UID: \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" Oct 30 22:03:14.271890 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:14.271701 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-workload-certs\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh\" (UID: \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" Oct 30 22:03:14.271890 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:14.271754 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-istio-data\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh\" (UID: \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" Oct 30 22:03:14.272120 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:14.272090 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-workload-certs\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh\" (UID: \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" Oct 30 22:03:14.272306 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:14.272150 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-istio-data\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh\" (UID: \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" Oct 30 22:03:14.272449 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:14.272393 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-credential-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh\" (UID: \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" Oct 30 22:03:14.272532 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:14.272496 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-workload-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh\" (UID: \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" Oct 30 22:03:14.273073 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:14.273051 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-istiod-ca-cert\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh\" (UID: \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" Oct 30 22:03:14.274129 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:14.274111 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-istio-envoy\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh\" (UID: \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" Oct 30 22:03:14.274204 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:14.274188 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-istio-podinfo\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh\" (UID: \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" Oct 30 22:03:14.335025 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:14.334995 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-istio-token\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh\" (UID: \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" Oct 30 22:03:14.335150 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:14.335104 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75vgf\" (UniqueName: \"kubernetes.io/projected/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-kube-api-access-75vgf\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh\" (UID: \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" Oct 30 22:03:14.426458 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:14.426383 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" Oct 30 22:03:14.857595 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:14.857570 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh"] Oct 30 22:03:14.860042 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:03:14.860016 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7dd97a6_507e_4441_b9d2_381bee4d6dd2.slice/crio-26f4faa0b9d6592718615fb692565002d6612030ea99d2000289055f26e4c630 WatchSource:0}: Error finding container 26f4faa0b9d6592718615fb692565002d6612030ea99d2000289055f26e4c630: Status 404 returned error can't find the container with id 26f4faa0b9d6592718615fb692565002d6612030ea99d2000289055f26e4c630 Oct 30 22:03:14.862428 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:14.862395 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:14.862511 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:14.862458 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:14.862511 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:14.862502 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:15.726103 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:15.726065 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" event={"ID":"a7dd97a6-507e-4441-b9d2-381bee4d6dd2","Type":"ContainerStarted","Data":"b0a42735700db7d8cec94372b4225d403b4ca362494598a87ecaec393ef504d4"} Oct 30 22:03:15.726570 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:15.726146 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" event={"ID":"a7dd97a6-507e-4441-b9d2-381bee4d6dd2","Type":"ContainerStarted","Data":"26f4faa0b9d6592718615fb692565002d6612030ea99d2000289055f26e4c630"} Oct 30 22:03:15.852009 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:15.851959 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" podStartSLOduration=1.851945503 podStartE2EDuration="1.851945503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:15.850624414 +0000 UTC m=+756.121322499" watchObservedRunningTime="2026-10-30 22:03:15.851945503 +0000 UTC m=+756.122643590" Oct 30 22:03:16.427262 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:16.427226 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" Oct 30 22:03:16.428963 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:16.428938 5918 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 22:03:16.429089 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:16.428984 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" podUID="a7dd97a6-507e-4441-b9d2-381bee4d6dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 22:03:17.426835 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:17.426788 5918 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 22:03:17.427254 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:17.426842 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" podUID="a7dd97a6-507e-4441-b9d2-381bee4d6dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 22:03:18.426749 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:18.426711 5918 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 22:03:18.426927 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:18.426779 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" podUID="a7dd97a6-507e-4441-b9d2-381bee4d6dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 22:03:19.427830 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:19.427786 5918 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 22:03:19.428258 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:19.427849 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" podUID="a7dd97a6-507e-4441-b9d2-381bee4d6dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 22:03:20.427001 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:20.426967 5918 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 22:03:20.427205 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:20.427027 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" podUID="a7dd97a6-507e-4441-b9d2-381bee4d6dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 22:03:21.276021 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:03:21.275977 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:03:21.533133 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:21.533016 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" Oct 30 22:03:21.534463 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:21.534441 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" Oct 30 22:03:21.534990 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:21.534969 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" Oct 30 22:03:51.275956 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:03:51.275921 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:03:59.157193 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:59.157155 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf"] Oct 30 22:03:59.161468 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:59.161451 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" Oct 30 22:03:59.216051 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:59.216025 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-6a0a09-openshift-default-dockercfg-h7ds4\"" Oct 30 22:03:59.267972 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:59.267945 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf"] Oct 30 22:03:59.281854 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:59.281825 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a77bc09f-2593-4ed7-89fe-21fffab26adf-istio-data\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf\" (UID: \"a77bc09f-2593-4ed7-89fe-21fffab26adf\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" Oct 30 22:03:59.282007 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:59.281910 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77bc09f-2593-4ed7-89fe-21fffab26adf-credential-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf\" (UID: \"a77bc09f-2593-4ed7-89fe-21fffab26adf\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" Oct 30 22:03:59.282007 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:59.281979 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qpdr\" (UniqueName: \"kubernetes.io/projected/a77bc09f-2593-4ed7-89fe-21fffab26adf-kube-api-access-2qpdr\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf\" (UID: \"a77bc09f-2593-4ed7-89fe-21fffab26adf\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" Oct 30 22:03:59.282133 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:59.282086 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a77bc09f-2593-4ed7-89fe-21fffab26adf-istio-envoy\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf\" (UID: \"a77bc09f-2593-4ed7-89fe-21fffab26adf\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" Oct 30 22:03:59.282191 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:59.282154 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a77bc09f-2593-4ed7-89fe-21fffab26adf-istiod-ca-cert\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf\" (UID: \"a77bc09f-2593-4ed7-89fe-21fffab26adf\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" Oct 30 22:03:59.282264 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:59.282231 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77bc09f-2593-4ed7-89fe-21fffab26adf-workload-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf\" (UID: \"a77bc09f-2593-4ed7-89fe-21fffab26adf\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" Oct 30 22:03:59.282264 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:59.282258 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a77bc09f-2593-4ed7-89fe-21fffab26adf-workload-certs\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf\" (UID: \"a77bc09f-2593-4ed7-89fe-21fffab26adf\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" Oct 30 22:03:59.282358 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:59.282300 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a77bc09f-2593-4ed7-89fe-21fffab26adf-istio-token\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf\" (UID: \"a77bc09f-2593-4ed7-89fe-21fffab26adf\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" Oct 30 22:03:59.282417 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:59.282393 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a77bc09f-2593-4ed7-89fe-21fffab26adf-istio-podinfo\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf\" (UID: \"a77bc09f-2593-4ed7-89fe-21fffab26adf\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" Oct 30 22:03:59.383855 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:59.383816 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a77bc09f-2593-4ed7-89fe-21fffab26adf-istio-podinfo\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf\" (UID: \"a77bc09f-2593-4ed7-89fe-21fffab26adf\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" Oct 30 22:03:59.384039 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:59.383871 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a77bc09f-2593-4ed7-89fe-21fffab26adf-istio-data\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf\" (UID: \"a77bc09f-2593-4ed7-89fe-21fffab26adf\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" Oct 30 22:03:59.384039 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:59.383967 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77bc09f-2593-4ed7-89fe-21fffab26adf-credential-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf\" (UID: \"a77bc09f-2593-4ed7-89fe-21fffab26adf\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" Oct 30 22:03:59.384039 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:59.383992 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qpdr\" (UniqueName: \"kubernetes.io/projected/a77bc09f-2593-4ed7-89fe-21fffab26adf-kube-api-access-2qpdr\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf\" (UID: \"a77bc09f-2593-4ed7-89fe-21fffab26adf\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" Oct 30 22:03:59.384039 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:59.384029 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a77bc09f-2593-4ed7-89fe-21fffab26adf-istio-envoy\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf\" (UID: \"a77bc09f-2593-4ed7-89fe-21fffab26adf\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" Oct 30 22:03:59.384333 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:59.384076 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a77bc09f-2593-4ed7-89fe-21fffab26adf-istiod-ca-cert\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf\" (UID: \"a77bc09f-2593-4ed7-89fe-21fffab26adf\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" Oct 30 22:03:59.384333 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:59.384133 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77bc09f-2593-4ed7-89fe-21fffab26adf-workload-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf\" (UID: \"a77bc09f-2593-4ed7-89fe-21fffab26adf\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" Oct 30 22:03:59.384333 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:59.384156 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a77bc09f-2593-4ed7-89fe-21fffab26adf-workload-certs\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf\" (UID: \"a77bc09f-2593-4ed7-89fe-21fffab26adf\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" Oct 30 22:03:59.384333 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:59.384196 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a77bc09f-2593-4ed7-89fe-21fffab26adf-istio-token\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf\" (UID: \"a77bc09f-2593-4ed7-89fe-21fffab26adf\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" Oct 30 22:03:59.384333 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:59.384273 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a77bc09f-2593-4ed7-89fe-21fffab26adf-istio-data\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf\" (UID: \"a77bc09f-2593-4ed7-89fe-21fffab26adf\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" Oct 30 22:03:59.384695 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:59.384357 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77bc09f-2593-4ed7-89fe-21fffab26adf-credential-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf\" (UID: \"a77bc09f-2593-4ed7-89fe-21fffab26adf\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" Oct 30 22:03:59.384695 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:59.384532 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77bc09f-2593-4ed7-89fe-21fffab26adf-workload-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf\" (UID: \"a77bc09f-2593-4ed7-89fe-21fffab26adf\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" Oct 30 22:03:59.384695 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:59.384658 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a77bc09f-2593-4ed7-89fe-21fffab26adf-workload-certs\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf\" (UID: \"a77bc09f-2593-4ed7-89fe-21fffab26adf\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" Oct 30 22:03:59.384950 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:59.384929 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a77bc09f-2593-4ed7-89fe-21fffab26adf-istiod-ca-cert\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf\" (UID: \"a77bc09f-2593-4ed7-89fe-21fffab26adf\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" Oct 30 22:03:59.386473 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:59.386447 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a77bc09f-2593-4ed7-89fe-21fffab26adf-istio-envoy\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf\" (UID: \"a77bc09f-2593-4ed7-89fe-21fffab26adf\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" Oct 30 22:03:59.387031 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:59.387010 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a77bc09f-2593-4ed7-89fe-21fffab26adf-istio-podinfo\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf\" (UID: \"a77bc09f-2593-4ed7-89fe-21fffab26adf\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" Oct 30 22:03:59.453042 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:59.453010 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a77bc09f-2593-4ed7-89fe-21fffab26adf-istio-token\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf\" (UID: \"a77bc09f-2593-4ed7-89fe-21fffab26adf\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" Oct 30 22:03:59.453163 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:59.453152 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qpdr\" (UniqueName: \"kubernetes.io/projected/a77bc09f-2593-4ed7-89fe-21fffab26adf-kube-api-access-2qpdr\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf\" (UID: \"a77bc09f-2593-4ed7-89fe-21fffab26adf\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" Oct 30 22:03:59.473931 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:59.473912 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" Oct 30 22:03:59.904106 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:59.904072 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf"] Oct 30 22:03:59.905382 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:03:59.905356 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda77bc09f_2593_4ed7_89fe_21fffab26adf.slice/crio-30c7cf577b6692095cc9d10824ba4a49c7d29b3d7992a13b4257da0a72c753fe WatchSource:0}: Error finding container 30c7cf577b6692095cc9d10824ba4a49c7d29b3d7992a13b4257da0a72c753fe: Status 404 returned error can't find the container with id 30c7cf577b6692095cc9d10824ba4a49c7d29b3d7992a13b4257da0a72c753fe Oct 30 22:03:59.907796 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:59.907759 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:59.907907 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:59.907838 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:59.907907 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:59.907883 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:59.975775 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:03:59.975707 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" event={"ID":"a77bc09f-2593-4ed7-89fe-21fffab26adf","Type":"ContainerStarted","Data":"30c7cf577b6692095cc9d10824ba4a49c7d29b3d7992a13b4257da0a72c753fe"} Oct 30 22:04:00.984620 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:00.984584 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" event={"ID":"a77bc09f-2593-4ed7-89fe-21fffab26adf","Type":"ContainerStarted","Data":"eb18f53ae7c5aea2e647c4eb64c2a86ef4f6494f916928d1ef45635137bcdfd2"} Oct 30 22:04:01.164431 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:01.164376 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" podStartSLOduration=2.164360924 podStartE2EDuration="2.164360924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:01.161996313 +0000 UTC m=+801.432694399" watchObservedRunningTime="2026-10-30 22:04:01.164360924 +0000 UTC m=+801.435059007" Oct 30 22:04:01.475023 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:01.474990 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" Oct 30 22:04:01.476427 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:01.476397 5918 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 22:04:01.476567 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:01.476450 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" podUID="a77bc09f-2593-4ed7-89fe-21fffab26adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 22:04:02.475049 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:02.475008 5918 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 22:04:02.475538 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:02.475073 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" podUID="a77bc09f-2593-4ed7-89fe-21fffab26adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 22:04:03.474823 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:03.474789 5918 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 22:04:03.474995 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:03.474846 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" podUID="a77bc09f-2593-4ed7-89fe-21fffab26adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 22:04:04.474662 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:04.474599 5918 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 22:04:04.475246 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:04.474679 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" podUID="a77bc09f-2593-4ed7-89fe-21fffab26adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 22:04:05.311287 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:05.304798 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb"] Oct 30 22:04:05.311287 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:05.305034 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" podUID="7707ff63-b08b-4233-afd0-4d71f7a19835" containerName="istio-proxy" containerID="cri-o://46314bbf9833f7a9727665fb4e0686ed50195676b49c1f14ebcc3442efcb65b5" gracePeriod=30 Oct 30 22:04:05.474673 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:05.474643 5918 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 22:04:05.475080 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:05.474694 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" podUID="a77bc09f-2593-4ed7-89fe-21fffab26adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 22:04:06.474846 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:06.474812 5918 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 22:04:06.475271 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:06.474869 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" podUID="a77bc09f-2593-4ed7-89fe-21fffab26adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 22:04:07.479255 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:07.479225 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" Oct 30 22:04:07.479654 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:07.479593 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" Oct 30 22:04:07.480428 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:07.480409 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" Oct 30 22:04:10.570342 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:10.570312 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" Oct 30 22:04:10.606644 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:10.606566 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7707ff63-b08b-4233-afd0-4d71f7a19835-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7707ff63-b08b-4233-afd0-4d71f7a19835-workload-socket\") pod \"7707ff63-b08b-4233-afd0-4d71f7a19835\" (UID: \"7707ff63-b08b-4233-afd0-4d71f7a19835\") " Oct 30 22:04:10.606644 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:10.606612 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7707ff63-b08b-4233-afd0-4d71f7a19835-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7707ff63-b08b-4233-afd0-4d71f7a19835-istiod-ca-cert\") pod \"7707ff63-b08b-4233-afd0-4d71f7a19835\" (UID: \"7707ff63-b08b-4233-afd0-4d71f7a19835\") " Oct 30 22:04:10.606830 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:10.606644 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7707ff63-b08b-4233-afd0-4d71f7a19835-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7707ff63-b08b-4233-afd0-4d71f7a19835-credential-socket\") pod \"7707ff63-b08b-4233-afd0-4d71f7a19835\" (UID: \"7707ff63-b08b-4233-afd0-4d71f7a19835\") " Oct 30 22:04:10.606830 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:10.606690 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7707ff63-b08b-4233-afd0-4d71f7a19835-kube-api-access-mbs77\" (UniqueName: \"kubernetes.io/projected/7707ff63-b08b-4233-afd0-4d71f7a19835-kube-api-access-mbs77\") pod \"7707ff63-b08b-4233-afd0-4d71f7a19835\" (UID: \"7707ff63-b08b-4233-afd0-4d71f7a19835\") " Oct 30 22:04:10.606830 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:10.606752 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7707ff63-b08b-4233-afd0-4d71f7a19835-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7707ff63-b08b-4233-afd0-4d71f7a19835-istio-data\") pod \"7707ff63-b08b-4233-afd0-4d71f7a19835\" (UID: \"7707ff63-b08b-4233-afd0-4d71f7a19835\") " Oct 30 22:04:10.606830 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:10.606796 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7707ff63-b08b-4233-afd0-4d71f7a19835-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7707ff63-b08b-4233-afd0-4d71f7a19835-workload-certs\") pod \"7707ff63-b08b-4233-afd0-4d71f7a19835\" (UID: \"7707ff63-b08b-4233-afd0-4d71f7a19835\") " Oct 30 22:04:10.607033 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:10.606841 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7707ff63-b08b-4233-afd0-4d71f7a19835-workload-socket" pod "7707ff63-b08b-4233-afd0-4d71f7a19835" (UID: "7707ff63-b08b-4233-afd0-4d71f7a19835"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:10.607033 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:10.606893 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7707ff63-b08b-4233-afd0-4d71f7a19835-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7707ff63-b08b-4233-afd0-4d71f7a19835-istio-envoy\") pod \"7707ff63-b08b-4233-afd0-4d71f7a19835\" (UID: \"7707ff63-b08b-4233-afd0-4d71f7a19835\") " Oct 30 22:04:10.607033 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:10.606936 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7707ff63-b08b-4233-afd0-4d71f7a19835-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7707ff63-b08b-4233-afd0-4d71f7a19835-istio-podinfo\") pod \"7707ff63-b08b-4233-afd0-4d71f7a19835\" (UID: \"7707ff63-b08b-4233-afd0-4d71f7a19835\") " Oct 30 22:04:10.607033 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:10.606947 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7707ff63-b08b-4233-afd0-4d71f7a19835-credential-socket" pod "7707ff63-b08b-4233-afd0-4d71f7a19835" (UID: "7707ff63-b08b-4233-afd0-4d71f7a19835"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:10.607033 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:10.606980 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7707ff63-b08b-4233-afd0-4d71f7a19835-istio-token\" (UniqueName: \"kubernetes.io/projected/7707ff63-b08b-4233-afd0-4d71f7a19835-istio-token\") pod \"7707ff63-b08b-4233-afd0-4d71f7a19835\" (UID: \"7707ff63-b08b-4233-afd0-4d71f7a19835\") " Oct 30 22:04:10.607272 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:10.607074 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7707ff63-b08b-4233-afd0-4d71f7a19835-istiod-ca-cert" pod "7707ff63-b08b-4233-afd0-4d71f7a19835" (UID: "7707ff63-b08b-4233-afd0-4d71f7a19835"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:10.607272 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:10.607127 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7707ff63-b08b-4233-afd0-4d71f7a19835-istio-data" pod "7707ff63-b08b-4233-afd0-4d71f7a19835" (UID: "7707ff63-b08b-4233-afd0-4d71f7a19835"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:10.607272 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:10.607152 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7707ff63-b08b-4233-afd0-4d71f7a19835-workload-certs" pod "7707ff63-b08b-4233-afd0-4d71f7a19835" (UID: "7707ff63-b08b-4233-afd0-4d71f7a19835"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:10.607470 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:10.607449 5918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7707ff63-b08b-4233-afd0-4d71f7a19835-workload-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:10.607545 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:10.607473 5918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7707ff63-b08b-4233-afd0-4d71f7a19835-istiod-ca-cert\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:10.607545 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:10.607493 5918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7707ff63-b08b-4233-afd0-4d71f7a19835-credential-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:10.607545 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:10.607510 5918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7707ff63-b08b-4233-afd0-4d71f7a19835-istio-data\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:10.607545 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:10.607523 5918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7707ff63-b08b-4233-afd0-4d71f7a19835-workload-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:10.609194 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:10.609165 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7707ff63-b08b-4233-afd0-4d71f7a19835-kube-api-access-mbs77" pod "7707ff63-b08b-4233-afd0-4d71f7a19835" (UID: "7707ff63-b08b-4233-afd0-4d71f7a19835"). InnerVolumeSpecName "kube-api-access-mbs77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:10.609352 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:10.609335 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7707ff63-b08b-4233-afd0-4d71f7a19835-istio-podinfo" pod "7707ff63-b08b-4233-afd0-4d71f7a19835" (UID: "7707ff63-b08b-4233-afd0-4d71f7a19835"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:04:10.609433 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:10.609386 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7707ff63-b08b-4233-afd0-4d71f7a19835-istio-envoy" pod "7707ff63-b08b-4233-afd0-4d71f7a19835" (UID: "7707ff63-b08b-4233-afd0-4d71f7a19835"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:10.609433 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:10.609397 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7707ff63-b08b-4233-afd0-4d71f7a19835-istio-token" pod "7707ff63-b08b-4233-afd0-4d71f7a19835" (UID: "7707ff63-b08b-4233-afd0-4d71f7a19835"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:10.708128 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:10.708098 5918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7707ff63-b08b-4233-afd0-4d71f7a19835-istio-envoy\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:10.708128 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:10.708122 5918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7707ff63-b08b-4233-afd0-4d71f7a19835-istio-podinfo\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:10.708128 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:10.708131 5918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7707ff63-b08b-4233-afd0-4d71f7a19835-istio-token\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:10.708408 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:10.708140 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbs77\" (UniqueName: \"kubernetes.io/projected/7707ff63-b08b-4233-afd0-4d71f7a19835-kube-api-access-mbs77\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:11.072783 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:11.072746 5918 generic.go:356] "Generic (PLEG): container finished" podID="7707ff63-b08b-4233-afd0-4d71f7a19835" containerID="46314bbf9833f7a9727665fb4e0686ed50195676b49c1f14ebcc3442efcb65b5" exitCode=0 Oct 30 22:04:11.072959 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:11.072797 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" Oct 30 22:04:11.072959 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:11.072836 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" event={"ID":"7707ff63-b08b-4233-afd0-4d71f7a19835","Type":"ContainerDied","Data":"46314bbf9833f7a9727665fb4e0686ed50195676b49c1f14ebcc3442efcb65b5"} Oct 30 22:04:11.072959 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:11.072903 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb" event={"ID":"7707ff63-b08b-4233-afd0-4d71f7a19835","Type":"ContainerDied","Data":"e28b3c5b0aa1bb83d9b688757ff93b0ebe2091f2ae27c0d8e68de98e5277cf8c"} Oct 30 22:04:11.072959 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:11.072933 5918 scope.go:122] "RemoveContainer" containerID="46314bbf9833f7a9727665fb4e0686ed50195676b49c1f14ebcc3442efcb65b5" Oct 30 22:04:11.083137 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:11.083117 5918 scope.go:122] "RemoveContainer" containerID="46314bbf9833f7a9727665fb4e0686ed50195676b49c1f14ebcc3442efcb65b5" Oct 30 22:04:11.083441 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:04:11.083421 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46314bbf9833f7a9727665fb4e0686ed50195676b49c1f14ebcc3442efcb65b5\": container with ID starting with 46314bbf9833f7a9727665fb4e0686ed50195676b49c1f14ebcc3442efcb65b5 not found: ID does not exist" containerID="46314bbf9833f7a9727665fb4e0686ed50195676b49c1f14ebcc3442efcb65b5" Oct 30 22:04:11.083501 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:11.083450 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46314bbf9833f7a9727665fb4e0686ed50195676b49c1f14ebcc3442efcb65b5"} err="failed to get container status \"46314bbf9833f7a9727665fb4e0686ed50195676b49c1f14ebcc3442efcb65b5\": rpc error: code = NotFound desc = could not find container \"46314bbf9833f7a9727665fb4e0686ed50195676b49c1f14ebcc3442efcb65b5\": container with ID starting with 46314bbf9833f7a9727665fb4e0686ed50195676b49c1f14ebcc3442efcb65b5 not found: ID does not exist" Oct 30 22:04:11.243880 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:11.243843 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb"] Oct 30 22:04:11.249243 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:11.249205 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ckkdb"] Oct 30 22:04:11.850731 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:11.850699 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7707ff63-b08b-4233-afd0-4d71f7a19835" path="/var/lib/kubelet/pods/7707ff63-b08b-4233-afd0-4d71f7a19835/volumes" Oct 30 22:04:21.275460 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:04:21.275377 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:04:30.002804 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.002767 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4"] Oct 30 22:04:30.003293 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.003279 5918 cpu_manager.go:414] "containerMap: removing container" podUID="7707ff63-b08b-4233-afd0-4d71f7a19835" containerName="istio-proxy" Oct 30 22:04:30.003360 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.003294 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7707ff63-b08b-4233-afd0-4d71f7a19835" containerName="istio-proxy" Oct 30 22:04:30.003400 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.003373 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="7707ff63-b08b-4233-afd0-4d71f7a19835" containerName="istio-proxy" Oct 30 22:04:30.006490 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.006475 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" Oct 30 22:04:30.061269 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.061235 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-3783c2-openshift-default-dockercfg-4c7q9\"" Oct 30 22:04:30.109382 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.109343 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-workload-certs\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4\" (UID: \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" Oct 30 22:04:30.109561 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.109414 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-istio-envoy\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4\" (UID: \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" Oct 30 22:04:30.109561 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.109540 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-istio-data\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4\" (UID: \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" Oct 30 22:04:30.109643 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.109587 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-istiod-ca-cert\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4\" (UID: \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" Oct 30 22:04:30.109759 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.109734 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-credential-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4\" (UID: \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" Oct 30 22:04:30.109823 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.109775 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48snv\" (UniqueName: \"kubernetes.io/projected/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-kube-api-access-48snv\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4\" (UID: \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" Oct 30 22:04:30.109874 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.109837 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-workload-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4\" (UID: \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" Oct 30 22:04:30.109874 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.109871 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-istio-token\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4\" (UID: \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" Oct 30 22:04:30.109938 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.109908 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-istio-podinfo\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4\" (UID: \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" Oct 30 22:04:30.112584 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.112562 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4"] Oct 30 22:04:30.210827 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.210793 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-istio-envoy\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4\" (UID: \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" Oct 30 22:04:30.210972 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.210850 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-istio-data\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4\" (UID: \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" Oct 30 22:04:30.210972 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.210877 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-istiod-ca-cert\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4\" (UID: \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" Oct 30 22:04:30.210972 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.210917 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-credential-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4\" (UID: \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" Oct 30 22:04:30.211086 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.211043 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48snv\" (UniqueName: \"kubernetes.io/projected/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-kube-api-access-48snv\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4\" (UID: \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" Oct 30 22:04:30.211154 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.211125 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-workload-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4\" (UID: \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" Oct 30 22:04:30.211208 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.211169 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-istio-token\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4\" (UID: \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" Oct 30 22:04:30.211295 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.211264 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-istio-podinfo\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4\" (UID: \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" Oct 30 22:04:30.211357 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.211303 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-workload-certs\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4\" (UID: \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" Oct 30 22:04:30.211357 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.211343 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-istio-data\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4\" (UID: \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" Oct 30 22:04:30.211563 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.211461 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-credential-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4\" (UID: \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" Oct 30 22:04:30.211563 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.211502 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-workload-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4\" (UID: \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" Oct 30 22:04:30.211731 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.211682 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-workload-certs\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4\" (UID: \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" Oct 30 22:04:30.211731 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.211699 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-istiod-ca-cert\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4\" (UID: \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" Oct 30 22:04:30.213490 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.213463 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-istio-envoy\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4\" (UID: \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" Oct 30 22:04:30.214056 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.214036 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-istio-podinfo\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4\" (UID: \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" Oct 30 22:04:30.274843 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.274773 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-istio-token\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4\" (UID: \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" Oct 30 22:04:30.275065 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.275038 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48snv\" (UniqueName: \"kubernetes.io/projected/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-kube-api-access-48snv\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4\" (UID: \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" Oct 30 22:04:30.319977 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.319947 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" Oct 30 22:04:30.751778 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.751738 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4"] Oct 30 22:04:30.787155 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:04:30.787127 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf83c719c_83ad_4b5e_be59_a3242cd7ecaf.slice/crio-41f10dc257690b86511682ccb66c43ce553f0db882b031edd29026e101e5dd8f WatchSource:0}: Error finding container 41f10dc257690b86511682ccb66c43ce553f0db882b031edd29026e101e5dd8f: Status 404 returned error can't find the container with id 41f10dc257690b86511682ccb66c43ce553f0db882b031edd29026e101e5dd8f Oct 30 22:04:30.789501 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.789466 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:30.789583 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.789536 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:30.789583 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:30.789574 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:31.193034 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:31.193004 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" event={"ID":"f83c719c-83ad-4b5e-be59-a3242cd7ecaf","Type":"ContainerStarted","Data":"863bd38fff60a4cf064d4be13aa9928b6109fdf653d3d80ee606e25ec7dc1fda"} Oct 30 22:04:31.193446 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:31.193060 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" event={"ID":"f83c719c-83ad-4b5e-be59-a3242cd7ecaf","Type":"ContainerStarted","Data":"41f10dc257690b86511682ccb66c43ce553f0db882b031edd29026e101e5dd8f"} Oct 30 22:04:31.317779 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:31.317731 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" podStartSLOduration=2.317717747 podStartE2EDuration="2.317717747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:31.315859751 +0000 UTC m=+831.586557847" watchObservedRunningTime="2026-10-30 22:04:31.317717747 +0000 UTC m=+831.588415830" Oct 30 22:04:31.321080 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:31.321051 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" Oct 30 22:04:31.322310 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:31.322284 5918 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 22:04:31.322409 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:31.322331 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" podUID="f83c719c-83ad-4b5e-be59-a3242cd7ecaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 22:04:32.320770 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:32.320736 5918 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 22:04:32.321179 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:32.320792 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" podUID="f83c719c-83ad-4b5e-be59-a3242cd7ecaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 22:04:33.320627 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:33.320591 5918 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 22:04:33.320805 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:33.320646 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" podUID="f83c719c-83ad-4b5e-be59-a3242cd7ecaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 22:04:34.320768 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:34.320729 5918 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 22:04:34.320956 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:34.320801 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" podUID="f83c719c-83ad-4b5e-be59-a3242cd7ecaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 22:04:35.320728 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:35.320686 5918 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 22:04:35.320899 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:35.320749 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" podUID="f83c719c-83ad-4b5e-be59-a3242cd7ecaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 22:04:36.320769 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:36.320734 5918 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 22:04:36.321133 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:36.320790 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" podUID="f83c719c-83ad-4b5e-be59-a3242cd7ecaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 22:04:37.320566 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:37.320531 5918 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 22:04:37.320743 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:37.320596 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" podUID="f83c719c-83ad-4b5e-be59-a3242cd7ecaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 22:04:38.325272 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:38.325241 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" Oct 30 22:04:38.325711 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:38.325577 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" Oct 30 22:04:38.326208 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:04:38.326190 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" Oct 30 22:04:51.274888 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:04:51.274852 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:05:00.992513 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:00.992480 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj"] Oct 30 22:05:00.996361 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:00.996345 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" Oct 30 22:05:01.046818 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:01.046781 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcx7q\" (UniqueName: \"kubernetes.io/projected/39983d32-e560-445c-8d17-d70986d3f6e7-kube-api-access-qcx7q\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj\" (UID: \"39983d32-e560-445c-8d17-d70986d3f6e7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" Oct 30 22:05:01.046985 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:01.046832 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39983d32-e560-445c-8d17-d70986d3f6e7-workload-certs\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj\" (UID: \"39983d32-e560-445c-8d17-d70986d3f6e7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" Oct 30 22:05:01.046985 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:01.046898 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39983d32-e560-445c-8d17-d70986d3f6e7-istio-data\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj\" (UID: \"39983d32-e560-445c-8d17-d70986d3f6e7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" Oct 30 22:05:01.046985 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:01.046920 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39983d32-e560-445c-8d17-d70986d3f6e7-istio-podinfo\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj\" (UID: \"39983d32-e560-445c-8d17-d70986d3f6e7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" Oct 30 22:05:01.046985 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:01.046948 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39983d32-e560-445c-8d17-d70986d3f6e7-istiod-ca-cert\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj\" (UID: \"39983d32-e560-445c-8d17-d70986d3f6e7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" Oct 30 22:05:01.047118 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:01.047022 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39983d32-e560-445c-8d17-d70986d3f6e7-istio-envoy\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj\" (UID: \"39983d32-e560-445c-8d17-d70986d3f6e7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" Oct 30 22:05:01.047118 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:01.047078 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39983d32-e560-445c-8d17-d70986d3f6e7-credential-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj\" (UID: \"39983d32-e560-445c-8d17-d70986d3f6e7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" Oct 30 22:05:01.047190 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:01.047144 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39983d32-e560-445c-8d17-d70986d3f6e7-istio-token\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj\" (UID: \"39983d32-e560-445c-8d17-d70986d3f6e7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" Oct 30 22:05:01.047252 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:01.047195 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39983d32-e560-445c-8d17-d70986d3f6e7-workload-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj\" (UID: \"39983d32-e560-445c-8d17-d70986d3f6e7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" Oct 30 22:05:01.051625 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:01.051609 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-bce513-openshift-default-dockercfg-jq86d\"" Oct 30 22:05:01.103898 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:01.103871 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj"] Oct 30 22:05:01.148169 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:01.148125 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39983d32-e560-445c-8d17-d70986d3f6e7-credential-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj\" (UID: \"39983d32-e560-445c-8d17-d70986d3f6e7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" Oct 30 22:05:01.148347 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:01.148271 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39983d32-e560-445c-8d17-d70986d3f6e7-istio-token\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj\" (UID: \"39983d32-e560-445c-8d17-d70986d3f6e7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" Oct 30 22:05:01.148347 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:01.148323 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39983d32-e560-445c-8d17-d70986d3f6e7-workload-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj\" (UID: \"39983d32-e560-445c-8d17-d70986d3f6e7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" Oct 30 22:05:01.148461 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:01.148427 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcx7q\" (UniqueName: \"kubernetes.io/projected/39983d32-e560-445c-8d17-d70986d3f6e7-kube-api-access-qcx7q\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj\" (UID: \"39983d32-e560-445c-8d17-d70986d3f6e7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" Oct 30 22:05:01.148519 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:01.148472 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39983d32-e560-445c-8d17-d70986d3f6e7-workload-certs\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj\" (UID: \"39983d32-e560-445c-8d17-d70986d3f6e7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" Oct 30 22:05:01.148519 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:01.148508 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39983d32-e560-445c-8d17-d70986d3f6e7-istio-data\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj\" (UID: \"39983d32-e560-445c-8d17-d70986d3f6e7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" Oct 30 22:05:01.148627 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:01.148537 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39983d32-e560-445c-8d17-d70986d3f6e7-credential-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj\" (UID: \"39983d32-e560-445c-8d17-d70986d3f6e7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" Oct 30 22:05:01.148627 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:01.148540 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39983d32-e560-445c-8d17-d70986d3f6e7-istio-podinfo\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj\" (UID: \"39983d32-e560-445c-8d17-d70986d3f6e7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" Oct 30 22:05:01.148627 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:01.148619 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39983d32-e560-445c-8d17-d70986d3f6e7-istiod-ca-cert\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj\" (UID: \"39983d32-e560-445c-8d17-d70986d3f6e7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" Oct 30 22:05:01.148767 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:01.148654 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39983d32-e560-445c-8d17-d70986d3f6e7-istio-envoy\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj\" (UID: \"39983d32-e560-445c-8d17-d70986d3f6e7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" Oct 30 22:05:01.148767 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:01.148705 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39983d32-e560-445c-8d17-d70986d3f6e7-workload-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj\" (UID: \"39983d32-e560-445c-8d17-d70986d3f6e7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" Oct 30 22:05:01.148897 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:01.148876 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39983d32-e560-445c-8d17-d70986d3f6e7-workload-certs\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj\" (UID: \"39983d32-e560-445c-8d17-d70986d3f6e7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" Oct 30 22:05:01.148966 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:01.148944 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39983d32-e560-445c-8d17-d70986d3f6e7-istio-data\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj\" (UID: \"39983d32-e560-445c-8d17-d70986d3f6e7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" Oct 30 22:05:01.149424 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:01.149300 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39983d32-e560-445c-8d17-d70986d3f6e7-istiod-ca-cert\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj\" (UID: \"39983d32-e560-445c-8d17-d70986d3f6e7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" Oct 30 22:05:01.150971 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:01.150947 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39983d32-e560-445c-8d17-d70986d3f6e7-istio-envoy\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj\" (UID: \"39983d32-e560-445c-8d17-d70986d3f6e7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" Oct 30 22:05:01.151314 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:01.151294 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39983d32-e560-445c-8d17-d70986d3f6e7-istio-podinfo\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj\" (UID: \"39983d32-e560-445c-8d17-d70986d3f6e7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" Oct 30 22:05:01.217672 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:01.217643 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39983d32-e560-445c-8d17-d70986d3f6e7-istio-token\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj\" (UID: \"39983d32-e560-445c-8d17-d70986d3f6e7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" Oct 30 22:05:01.219692 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:01.219664 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcx7q\" (UniqueName: \"kubernetes.io/projected/39983d32-e560-445c-8d17-d70986d3f6e7-kube-api-access-qcx7q\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj\" (UID: \"39983d32-e560-445c-8d17-d70986d3f6e7\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" Oct 30 22:05:01.309404 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:01.309332 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" Oct 30 22:05:01.743739 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:01.743710 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj"] Oct 30 22:05:01.745485 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:05:01.745462 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39983d32_e560_445c_8d17_d70986d3f6e7.slice/crio-9d3fefb69c3640c84be323fd0ef109db6c029c3f7af47724a2efc93e0b902dd2 WatchSource:0}: Error finding container 9d3fefb69c3640c84be323fd0ef109db6c029c3f7af47724a2efc93e0b902dd2: Status 404 returned error can't find the container with id 9d3fefb69c3640c84be323fd0ef109db6c029c3f7af47724a2efc93e0b902dd2 Oct 30 22:05:01.747868 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:01.747831 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:01.747966 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:01.747903 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:01.747966 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:01.747962 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:02.411097 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:02.411058 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" event={"ID":"39983d32-e560-445c-8d17-d70986d3f6e7","Type":"ContainerStarted","Data":"2375a87f710334934f99f193e11c19fff952c5c2d25c27302a9140c032b2e5cc"} Oct 30 22:05:02.411615 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:02.411115 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" event={"ID":"39983d32-e560-445c-8d17-d70986d3f6e7","Type":"ContainerStarted","Data":"9d3fefb69c3640c84be323fd0ef109db6c029c3f7af47724a2efc93e0b902dd2"} Oct 30 22:05:02.537192 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:02.537139 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" podStartSLOduration=2.537125606 podStartE2EDuration="2.537125606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:02.535111054 +0000 UTC m=+862.805809143" watchObservedRunningTime="2026-10-30 22:05:02.537125606 +0000 UTC m=+862.807823689" Oct 30 22:05:03.310951 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:03.309947 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" Oct 30 22:05:03.312311 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:03.311990 5918 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 22:05:03.312311 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:03.312041 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" podUID="39983d32-e560-445c-8d17-d70986d3f6e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 22:05:04.310666 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:04.310620 5918 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 22:05:04.311301 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:04.311263 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" podUID="39983d32-e560-445c-8d17-d70986d3f6e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 22:05:05.311707 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:05.311659 5918 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 22:05:05.312133 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:05.311737 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" podUID="39983d32-e560-445c-8d17-d70986d3f6e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 22:05:05.838842 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:05.838808 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989"] Oct 30 22:05:05.842762 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:05.842746 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" Oct 30 22:05:05.920942 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:05.920903 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb62082-3ac2-4985-854f-9a63194f65e1-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-9q989\" (UID: \"5fb62082-3ac2-4985-854f-9a63194f65e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" Oct 30 22:05:05.920942 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:05.920946 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb62082-3ac2-4985-854f-9a63194f65e1-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-9q989\" (UID: \"5fb62082-3ac2-4985-854f-9a63194f65e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" Oct 30 22:05:05.921202 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:05.921040 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb62082-3ac2-4985-854f-9a63194f65e1-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-9q989\" (UID: \"5fb62082-3ac2-4985-854f-9a63194f65e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" Oct 30 22:05:05.921202 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:05.921190 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkrp7\" (UniqueName: \"kubernetes.io/projected/5fb62082-3ac2-4985-854f-9a63194f65e1-kube-api-access-tkrp7\") pod \"maas-default-gateway-openshift-default-67bf986f99-9q989\" (UID: \"5fb62082-3ac2-4985-854f-9a63194f65e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" Oct 30 22:05:05.921341 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:05.921299 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb62082-3ac2-4985-854f-9a63194f65e1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-9q989\" (UID: \"5fb62082-3ac2-4985-854f-9a63194f65e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" Oct 30 22:05:05.921381 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:05.921338 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb62082-3ac2-4985-854f-9a63194f65e1-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-9q989\" (UID: \"5fb62082-3ac2-4985-854f-9a63194f65e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" Oct 30 22:05:05.921485 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:05.921458 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb62082-3ac2-4985-854f-9a63194f65e1-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-9q989\" (UID: \"5fb62082-3ac2-4985-854f-9a63194f65e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" Oct 30 22:05:05.921555 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:05.921534 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb62082-3ac2-4985-854f-9a63194f65e1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-9q989\" (UID: \"5fb62082-3ac2-4985-854f-9a63194f65e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" Oct 30 22:05:05.921618 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:05.921584 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb62082-3ac2-4985-854f-9a63194f65e1-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-9q989\" (UID: \"5fb62082-3ac2-4985-854f-9a63194f65e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" Oct 30 22:05:06.023309 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:06.023268 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb62082-3ac2-4985-854f-9a63194f65e1-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-9q989\" (UID: \"5fb62082-3ac2-4985-854f-9a63194f65e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" Oct 30 22:05:06.023309 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:06.023311 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb62082-3ac2-4985-854f-9a63194f65e1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-9q989\" (UID: \"5fb62082-3ac2-4985-854f-9a63194f65e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" Oct 30 22:05:06.023576 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:06.023343 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb62082-3ac2-4985-854f-9a63194f65e1-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-9q989\" (UID: \"5fb62082-3ac2-4985-854f-9a63194f65e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" Oct 30 22:05:06.023576 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:06.023381 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb62082-3ac2-4985-854f-9a63194f65e1-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-9q989\" (UID: \"5fb62082-3ac2-4985-854f-9a63194f65e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" Oct 30 22:05:06.023576 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:06.023409 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb62082-3ac2-4985-854f-9a63194f65e1-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-9q989\" (UID: \"5fb62082-3ac2-4985-854f-9a63194f65e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" Oct 30 22:05:06.023576 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:06.023562 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb62082-3ac2-4985-854f-9a63194f65e1-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-9q989\" (UID: \"5fb62082-3ac2-4985-854f-9a63194f65e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" Oct 30 22:05:06.023793 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:06.023708 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkrp7\" (UniqueName: \"kubernetes.io/projected/5fb62082-3ac2-4985-854f-9a63194f65e1-kube-api-access-tkrp7\") pod \"maas-default-gateway-openshift-default-67bf986f99-9q989\" (UID: \"5fb62082-3ac2-4985-854f-9a63194f65e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" Oct 30 22:05:06.023888 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:06.023862 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb62082-3ac2-4985-854f-9a63194f65e1-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-9q989\" (UID: \"5fb62082-3ac2-4985-854f-9a63194f65e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" Oct 30 22:05:06.023958 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:06.023906 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb62082-3ac2-4985-854f-9a63194f65e1-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-9q989\" (UID: \"5fb62082-3ac2-4985-854f-9a63194f65e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" Oct 30 22:05:06.024121 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:06.024098 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb62082-3ac2-4985-854f-9a63194f65e1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-9q989\" (UID: \"5fb62082-3ac2-4985-854f-9a63194f65e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" Oct 30 22:05:06.024121 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:06.024110 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb62082-3ac2-4985-854f-9a63194f65e1-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-9q989\" (UID: \"5fb62082-3ac2-4985-854f-9a63194f65e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" Oct 30 22:05:06.024273 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:06.024134 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb62082-3ac2-4985-854f-9a63194f65e1-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-9q989\" (UID: \"5fb62082-3ac2-4985-854f-9a63194f65e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" Oct 30 22:05:06.024273 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:06.024204 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb62082-3ac2-4985-854f-9a63194f65e1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-9q989\" (UID: \"5fb62082-3ac2-4985-854f-9a63194f65e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" Oct 30 22:05:06.024394 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:06.024204 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb62082-3ac2-4985-854f-9a63194f65e1-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-9q989\" (UID: \"5fb62082-3ac2-4985-854f-9a63194f65e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" Oct 30 22:05:06.026570 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:06.026540 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb62082-3ac2-4985-854f-9a63194f65e1-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-9q989\" (UID: \"5fb62082-3ac2-4985-854f-9a63194f65e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" Oct 30 22:05:06.029701 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:06.029675 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb62082-3ac2-4985-854f-9a63194f65e1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-9q989\" (UID: \"5fb62082-3ac2-4985-854f-9a63194f65e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" Oct 30 22:05:06.076359 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:06.076329 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989"] Oct 30 22:05:06.118398 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:06.118303 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkrp7\" (UniqueName: \"kubernetes.io/projected/5fb62082-3ac2-4985-854f-9a63194f65e1-kube-api-access-tkrp7\") pod \"maas-default-gateway-openshift-default-67bf986f99-9q989\" (UID: \"5fb62082-3ac2-4985-854f-9a63194f65e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" Oct 30 22:05:06.120894 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:06.120870 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb62082-3ac2-4985-854f-9a63194f65e1-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-9q989\" (UID: \"5fb62082-3ac2-4985-854f-9a63194f65e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" Oct 30 22:05:06.157312 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:06.157286 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" Oct 30 22:05:06.311714 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:06.311679 5918 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 22:05:06.312258 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:06.311762 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" podUID="39983d32-e560-445c-8d17-d70986d3f6e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 22:05:06.611249 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:06.609099 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989"] Oct 30 22:05:06.616413 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:05:06.616381 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fb62082_3ac2_4985_854f_9a63194f65e1.slice/crio-6104f54022db0a1c50ee371e66edf79899a76fe12619ed44f1ea592561039219 WatchSource:0}: Error finding container 6104f54022db0a1c50ee371e66edf79899a76fe12619ed44f1ea592561039219: Status 404 returned error can't find the container with id 6104f54022db0a1c50ee371e66edf79899a76fe12619ed44f1ea592561039219 Oct 30 22:05:06.619176 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:06.619143 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:06.619274 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:06.619245 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:06.619328 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:06.619274 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:07.314238 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:07.313633 5918 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 22:05:07.314238 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:07.313693 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" podUID="39983d32-e560-445c-8d17-d70986d3f6e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 22:05:07.447479 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:07.447439 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" event={"ID":"5fb62082-3ac2-4985-854f-9a63194f65e1","Type":"ContainerStarted","Data":"663f531ae3de7c53bf3a5f4ab0340b921d2e083202e15a6d3c67da244b456012"} Oct 30 22:05:07.447655 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:07.447488 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" event={"ID":"5fb62082-3ac2-4985-854f-9a63194f65e1","Type":"ContainerStarted","Data":"6104f54022db0a1c50ee371e66edf79899a76fe12619ed44f1ea592561039219"} Oct 30 22:05:07.679153 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:07.679110 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" podStartSLOduration=2.679092458 podStartE2EDuration="2.679092458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:07.628706233 +0000 UTC m=+867.899404353" watchObservedRunningTime="2026-10-30 22:05:07.679092458 +0000 UTC m=+867.949790542" Oct 30 22:05:08.158252 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:08.158138 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" Oct 30 22:05:08.159721 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:08.159696 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9q989 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 22:05:08.159836 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:08.159743 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" podUID="5fb62082-3ac2-4985-854f-9a63194f65e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 22:05:08.311617 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:08.311580 5918 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 22:05:08.311872 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:08.311844 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" podUID="39983d32-e560-445c-8d17-d70986d3f6e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 22:05:09.158103 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:09.158069 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9q989 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 22:05:09.158582 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:09.158125 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" podUID="5fb62082-3ac2-4985-854f-9a63194f65e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 22:05:09.315902 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:09.315865 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" Oct 30 22:05:09.316261 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:09.316238 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" Oct 30 22:05:09.316991 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:09.316971 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" Oct 30 22:05:10.157953 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:10.157917 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9q989 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 22:05:10.158133 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:10.157976 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" podUID="5fb62082-3ac2-4985-854f-9a63194f65e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 22:05:11.158596 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:11.158558 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9q989 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 22:05:11.159077 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:11.158624 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" podUID="5fb62082-3ac2-4985-854f-9a63194f65e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 22:05:12.157767 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:12.157731 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9q989 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 22:05:12.157965 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:12.157789 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" podUID="5fb62082-3ac2-4985-854f-9a63194f65e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 22:05:13.158524 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:13.158487 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9q989 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 22:05:13.159002 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:13.158547 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" podUID="5fb62082-3ac2-4985-854f-9a63194f65e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 22:05:14.158053 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:14.158015 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9q989 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 22:05:14.158210 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:14.158082 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" podUID="5fb62082-3ac2-4985-854f-9a63194f65e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 22:05:15.162147 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:15.162115 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" Oct 30 22:05:15.162584 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:15.162446 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" Oct 30 22:05:15.163162 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:15.163146 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" Oct 30 22:05:20.944335 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:20.944303 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc"] Oct 30 22:05:20.948062 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:20.948042 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" Oct 30 22:05:21.015424 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:21.015396 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-plwzc\" (UID: \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" Oct 30 22:05:21.015540 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:21.015434 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-plwzc\" (UID: \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" Oct 30 22:05:21.015540 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:21.015463 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-plwzc\" (UID: \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" Oct 30 22:05:21.015616 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:21.015564 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-plwzc\" (UID: \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" Oct 30 22:05:21.015651 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:21.015635 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-plwzc\" (UID: \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" Oct 30 22:05:21.015688 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:21.015657 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpm57\" (UniqueName: \"kubernetes.io/projected/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-kube-api-access-cpm57\") pod \"maas-default-gateway-openshift-default-67bf986f99-plwzc\" (UID: \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" Oct 30 22:05:21.015688 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:21.015680 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-plwzc\" (UID: \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" Oct 30 22:05:21.015748 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:21.015702 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-plwzc\" (UID: \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" Oct 30 22:05:21.015780 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:21.015765 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-plwzc\" (UID: \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" Oct 30 22:05:21.053262 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:21.053205 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc"] Oct 30 22:05:21.116619 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:21.116591 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-plwzc\" (UID: \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" Oct 30 22:05:21.116762 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:21.116623 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpm57\" (UniqueName: \"kubernetes.io/projected/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-kube-api-access-cpm57\") pod \"maas-default-gateway-openshift-default-67bf986f99-plwzc\" (UID: \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" Oct 30 22:05:21.116762 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:21.116650 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-plwzc\" (UID: \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" Oct 30 22:05:21.116762 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:21.116673 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-plwzc\" (UID: \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" Oct 30 22:05:21.116762 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:21.116701 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-plwzc\" (UID: \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" Oct 30 22:05:21.116998 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:21.116820 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-plwzc\" (UID: \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" Oct 30 22:05:21.116998 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:21.116859 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-plwzc\" (UID: \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" Oct 30 22:05:21.116998 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:21.116889 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-plwzc\" (UID: \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" Oct 30 22:05:21.116998 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:21.116920 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-plwzc\" (UID: \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" Oct 30 22:05:21.117237 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:21.117067 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-plwzc\" (UID: \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" Oct 30 22:05:21.117237 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:21.117115 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-plwzc\" (UID: \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" Oct 30 22:05:21.117452 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:21.117428 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-plwzc\" (UID: \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" Oct 30 22:05:21.117638 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:21.117483 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-plwzc\" (UID: \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" Oct 30 22:05:21.117915 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:21.117888 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-plwzc\" (UID: \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" Oct 30 22:05:21.119168 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:21.119142 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-plwzc\" (UID: \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" Oct 30 22:05:21.119493 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:21.119476 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-plwzc\" (UID: \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" Oct 30 22:05:21.186160 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:21.186130 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-plwzc\" (UID: \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" Oct 30 22:05:21.186348 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:21.186201 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpm57\" (UniqueName: \"kubernetes.io/projected/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-kube-api-access-cpm57\") pod \"maas-default-gateway-openshift-default-67bf986f99-plwzc\" (UID: \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" Oct 30 22:05:21.261588 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:21.261479 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" Oct 30 22:05:21.277108 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:05:21.277074 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:05:21.696592 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:21.696567 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc"] Oct 30 22:05:21.698311 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:05:21.698285 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74d3f33a_98ec_4b95_88d4_6e836fe59fd5.slice/crio-793ec52634ae1090a7ecf2bff251328fdc208f18036f2d2d20662520180d1da0 WatchSource:0}: Error finding container 793ec52634ae1090a7ecf2bff251328fdc208f18036f2d2d20662520180d1da0: Status 404 returned error can't find the container with id 793ec52634ae1090a7ecf2bff251328fdc208f18036f2d2d20662520180d1da0 Oct 30 22:05:21.700620 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:21.700586 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:21.700712 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:21.700649 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:21.700712 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:21.700683 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:22.534698 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:22.534666 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" event={"ID":"74d3f33a-98ec-4b95-88d4-6e836fe59fd5","Type":"ContainerStarted","Data":"a418702ab277f4d16865f6fa931acc75dd3ce2a386637c4a38ce7d4311380f3f"} Oct 30 22:05:22.535124 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:22.534715 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" event={"ID":"74d3f33a-98ec-4b95-88d4-6e836fe59fd5","Type":"ContainerStarted","Data":"793ec52634ae1090a7ecf2bff251328fdc208f18036f2d2d20662520180d1da0"} Oct 30 22:05:22.666380 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:22.666322 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" podStartSLOduration=2.666306884 podStartE2EDuration="2.666306884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:22.663520417 +0000 UTC m=+882.934218502" watchObservedRunningTime="2026-10-30 22:05:22.666306884 +0000 UTC m=+882.937004964" Oct 30 22:05:23.261756 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:23.261727 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" Oct 30 22:05:23.263283 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:23.263259 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-plwzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 22:05:23.263401 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:23.263315 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" podUID="74d3f33a-98ec-4b95-88d4-6e836fe59fd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 22:05:24.262455 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:24.262421 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-plwzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 22:05:24.262970 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:24.262490 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" podUID="74d3f33a-98ec-4b95-88d4-6e836fe59fd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 22:05:25.262060 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:25.262021 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-plwzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 22:05:25.262273 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:25.262090 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" podUID="74d3f33a-98ec-4b95-88d4-6e836fe59fd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 22:05:26.263034 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:26.263000 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-plwzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 22:05:26.263471 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:26.263063 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" podUID="74d3f33a-98ec-4b95-88d4-6e836fe59fd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 22:05:27.262361 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:27.262329 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-plwzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 22:05:27.262535 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:27.262393 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" podUID="74d3f33a-98ec-4b95-88d4-6e836fe59fd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 22:05:28.262135 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:28.262098 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-plwzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 22:05:28.262563 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:28.262163 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" podUID="74d3f33a-98ec-4b95-88d4-6e836fe59fd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 22:05:30.305341 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:30.305305 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-plwzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:05:30.305886 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:30.305858 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" podUID="74d3f33a-98ec-4b95-88d4-6e836fe59fd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:30.548410 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:30.548380 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" Oct 30 22:05:30.549783 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:30.549748 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" Oct 30 22:05:30.550939 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:05:30.550918 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" Oct 30 22:05:51.275115 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:05:51.275040 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:06:01.731406 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.731372 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76"] Oct 30 22:06:01.735175 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.735153 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" Oct 30 22:06:01.781268 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.781200 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm"] Oct 30 22:06:01.785269 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.785245 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" Oct 30 22:06:01.796623 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.796590 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-2182a5-openshift-default-dockercfg-ssq65\"" Oct 30 22:06:01.798247 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.798201 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed6f9c77-47bb-4e52-9e6e-1493201547dd-istio-data\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76\" (UID: \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" Oct 30 22:06:01.798372 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.798311 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed6f9c77-47bb-4e52-9e6e-1493201547dd-istio-envoy\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76\" (UID: \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" Oct 30 22:06:01.798438 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.798375 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed6f9c77-47bb-4e52-9e6e-1493201547dd-istio-token\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76\" (UID: \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" Oct 30 22:06:01.798540 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.798521 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed6f9c77-47bb-4e52-9e6e-1493201547dd-credential-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76\" (UID: \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" Oct 30 22:06:01.798588 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.798559 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed6f9c77-47bb-4e52-9e6e-1493201547dd-istiod-ca-cert\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76\" (UID: \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" Oct 30 22:06:01.798648 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.798586 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wd98\" (UniqueName: \"kubernetes.io/projected/ed6f9c77-47bb-4e52-9e6e-1493201547dd-kube-api-access-2wd98\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76\" (UID: \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" Oct 30 22:06:01.798708 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.798684 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed6f9c77-47bb-4e52-9e6e-1493201547dd-workload-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76\" (UID: \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" Oct 30 22:06:01.798823 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.798811 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed6f9c77-47bb-4e52-9e6e-1493201547dd-workload-certs\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76\" (UID: \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" Oct 30 22:06:01.798861 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.798833 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed6f9c77-47bb-4e52-9e6e-1493201547dd-istio-podinfo\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76\" (UID: \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" Oct 30 22:06:01.833376 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.833341 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76"] Oct 30 22:06:01.865281 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.865236 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm"] Oct 30 22:06:01.899758 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.899721 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed6f9c77-47bb-4e52-9e6e-1493201547dd-credential-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76\" (UID: \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" Oct 30 22:06:01.899758 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.899758 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e8dced5-a68f-436b-ab03-da8d8e879315-istio-token\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm\" (UID: \"2e8dced5-a68f-436b-ab03-da8d8e879315\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" Oct 30 22:06:01.899966 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.899785 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed6f9c77-47bb-4e52-9e6e-1493201547dd-istiod-ca-cert\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76\" (UID: \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" Oct 30 22:06:01.899966 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.899862 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wd98\" (UniqueName: \"kubernetes.io/projected/ed6f9c77-47bb-4e52-9e6e-1493201547dd-kube-api-access-2wd98\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76\" (UID: \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" Oct 30 22:06:01.899966 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.899910 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed6f9c77-47bb-4e52-9e6e-1493201547dd-workload-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76\" (UID: \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" Oct 30 22:06:01.899966 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.899953 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e8dced5-a68f-436b-ab03-da8d8e879315-istio-envoy\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm\" (UID: \"2e8dced5-a68f-436b-ab03-da8d8e879315\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" Oct 30 22:06:01.900169 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.899990 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed6f9c77-47bb-4e52-9e6e-1493201547dd-workload-certs\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76\" (UID: \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" Oct 30 22:06:01.900169 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.900016 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed6f9c77-47bb-4e52-9e6e-1493201547dd-istio-podinfo\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76\" (UID: \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" Oct 30 22:06:01.900169 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.900050 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e8dced5-a68f-436b-ab03-da8d8e879315-istiod-ca-cert\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm\" (UID: \"2e8dced5-a68f-436b-ab03-da8d8e879315\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" Oct 30 22:06:01.900169 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.900100 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwxcl\" (UniqueName: \"kubernetes.io/projected/2e8dced5-a68f-436b-ab03-da8d8e879315-kube-api-access-lwxcl\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm\" (UID: \"2e8dced5-a68f-436b-ab03-da8d8e879315\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" Oct 30 22:06:01.900169 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.900128 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8dced5-a68f-436b-ab03-da8d8e879315-credential-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm\" (UID: \"2e8dced5-a68f-436b-ab03-da8d8e879315\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" Oct 30 22:06:01.900169 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.900154 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8dced5-a68f-436b-ab03-da8d8e879315-workload-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm\" (UID: \"2e8dced5-a68f-436b-ab03-da8d8e879315\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" Oct 30 22:06:01.900539 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.900190 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e8dced5-a68f-436b-ab03-da8d8e879315-workload-certs\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm\" (UID: \"2e8dced5-a68f-436b-ab03-da8d8e879315\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" Oct 30 22:06:01.900539 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.900191 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed6f9c77-47bb-4e52-9e6e-1493201547dd-credential-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76\" (UID: \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" Oct 30 22:06:01.900539 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.900308 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e8dced5-a68f-436b-ab03-da8d8e879315-istio-data\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm\" (UID: \"2e8dced5-a68f-436b-ab03-da8d8e879315\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" Oct 30 22:06:01.900539 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.900355 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed6f9c77-47bb-4e52-9e6e-1493201547dd-workload-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76\" (UID: \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" Oct 30 22:06:01.900539 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.900357 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed6f9c77-47bb-4e52-9e6e-1493201547dd-istio-data\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76\" (UID: \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" Oct 30 22:06:01.900539 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.900440 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed6f9c77-47bb-4e52-9e6e-1493201547dd-istio-envoy\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76\" (UID: \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" Oct 30 22:06:01.900539 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.900485 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed6f9c77-47bb-4e52-9e6e-1493201547dd-istio-token\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76\" (UID: \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" Oct 30 22:06:01.900539 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.900493 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed6f9c77-47bb-4e52-9e6e-1493201547dd-workload-certs\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76\" (UID: \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" Oct 30 22:06:01.900539 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.900517 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e8dced5-a68f-436b-ab03-da8d8e879315-istio-podinfo\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm\" (UID: \"2e8dced5-a68f-436b-ab03-da8d8e879315\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" Oct 30 22:06:01.900887 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.900602 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed6f9c77-47bb-4e52-9e6e-1493201547dd-istiod-ca-cert\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76\" (UID: \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" Oct 30 22:06:01.900887 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.900633 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed6f9c77-47bb-4e52-9e6e-1493201547dd-istio-data\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76\" (UID: \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" Oct 30 22:06:01.902769 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.902750 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed6f9c77-47bb-4e52-9e6e-1493201547dd-istio-envoy\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76\" (UID: \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" Oct 30 22:06:01.902884 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.902867 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed6f9c77-47bb-4e52-9e6e-1493201547dd-istio-podinfo\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76\" (UID: \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" Oct 30 22:06:01.963426 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.963388 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed6f9c77-47bb-4e52-9e6e-1493201547dd-istio-token\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76\" (UID: \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" Oct 30 22:06:01.963576 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:01.963501 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wd98\" (UniqueName: \"kubernetes.io/projected/ed6f9c77-47bb-4e52-9e6e-1493201547dd-kube-api-access-2wd98\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76\" (UID: \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" Oct 30 22:06:02.002080 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:02.001995 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e8dced5-a68f-436b-ab03-da8d8e879315-istiod-ca-cert\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm\" (UID: \"2e8dced5-a68f-436b-ab03-da8d8e879315\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" Oct 30 22:06:02.002080 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:02.002051 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwxcl\" (UniqueName: \"kubernetes.io/projected/2e8dced5-a68f-436b-ab03-da8d8e879315-kube-api-access-lwxcl\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm\" (UID: \"2e8dced5-a68f-436b-ab03-da8d8e879315\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" Oct 30 22:06:02.002320 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:02.002094 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8dced5-a68f-436b-ab03-da8d8e879315-credential-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm\" (UID: \"2e8dced5-a68f-436b-ab03-da8d8e879315\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" Oct 30 22:06:02.002320 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:02.002128 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8dced5-a68f-436b-ab03-da8d8e879315-workload-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm\" (UID: \"2e8dced5-a68f-436b-ab03-da8d8e879315\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" Oct 30 22:06:02.002320 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:02.002170 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e8dced5-a68f-436b-ab03-da8d8e879315-workload-certs\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm\" (UID: \"2e8dced5-a68f-436b-ab03-da8d8e879315\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" Oct 30 22:06:02.002320 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:02.002192 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e8dced5-a68f-436b-ab03-da8d8e879315-istio-data\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm\" (UID: \"2e8dced5-a68f-436b-ab03-da8d8e879315\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" Oct 30 22:06:02.002320 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:02.002269 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e8dced5-a68f-436b-ab03-da8d8e879315-istio-podinfo\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm\" (UID: \"2e8dced5-a68f-436b-ab03-da8d8e879315\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" Oct 30 22:06:02.002539 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:02.002368 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e8dced5-a68f-436b-ab03-da8d8e879315-istio-token\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm\" (UID: \"2e8dced5-a68f-436b-ab03-da8d8e879315\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" Oct 30 22:06:02.002539 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:02.002423 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e8dced5-a68f-436b-ab03-da8d8e879315-istio-envoy\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm\" (UID: \"2e8dced5-a68f-436b-ab03-da8d8e879315\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" Oct 30 22:06:02.002649 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:02.002537 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8dced5-a68f-436b-ab03-da8d8e879315-credential-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm\" (UID: \"2e8dced5-a68f-436b-ab03-da8d8e879315\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" Oct 30 22:06:02.002649 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:02.002634 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8dced5-a68f-436b-ab03-da8d8e879315-workload-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm\" (UID: \"2e8dced5-a68f-436b-ab03-da8d8e879315\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" Oct 30 22:06:02.002760 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:02.002705 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e8dced5-a68f-436b-ab03-da8d8e879315-workload-certs\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm\" (UID: \"2e8dced5-a68f-436b-ab03-da8d8e879315\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" Oct 30 22:06:02.002830 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:02.002801 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e8dced5-a68f-436b-ab03-da8d8e879315-istio-data\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm\" (UID: \"2e8dced5-a68f-436b-ab03-da8d8e879315\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" Oct 30 22:06:02.002900 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:02.002878 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e8dced5-a68f-436b-ab03-da8d8e879315-istiod-ca-cert\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm\" (UID: \"2e8dced5-a68f-436b-ab03-da8d8e879315\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" Oct 30 22:06:02.004851 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:02.004828 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e8dced5-a68f-436b-ab03-da8d8e879315-istio-envoy\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm\" (UID: \"2e8dced5-a68f-436b-ab03-da8d8e879315\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" Oct 30 22:06:02.005042 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:02.005024 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e8dced5-a68f-436b-ab03-da8d8e879315-istio-podinfo\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm\" (UID: \"2e8dced5-a68f-436b-ab03-da8d8e879315\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" Oct 30 22:06:02.049733 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:02.049699 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" Oct 30 22:06:02.068513 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:02.068489 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e8dced5-a68f-436b-ab03-da8d8e879315-istio-token\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm\" (UID: \"2e8dced5-a68f-436b-ab03-da8d8e879315\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" Oct 30 22:06:02.068760 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:02.068731 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwxcl\" (UniqueName: \"kubernetes.io/projected/2e8dced5-a68f-436b-ab03-da8d8e879315-kube-api-access-lwxcl\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm\" (UID: \"2e8dced5-a68f-436b-ab03-da8d8e879315\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" Oct 30 22:06:02.099705 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:02.099671 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" Oct 30 22:06:02.480805 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:02.480776 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76"] Oct 30 22:06:02.482490 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:06:02.482455 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded6f9c77_47bb_4e52_9e6e_1493201547dd.slice/crio-2514a25795306ec0595357aaf06ccb4503f1f6665b95a5b9ba01688bad6c1015 WatchSource:0}: Error finding container 2514a25795306ec0595357aaf06ccb4503f1f6665b95a5b9ba01688bad6c1015: Status 404 returned error can't find the container with id 2514a25795306ec0595357aaf06ccb4503f1f6665b95a5b9ba01688bad6c1015 Oct 30 22:06:02.485203 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:02.485170 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:06:02.485318 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:02.485268 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:06:02.485318 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:02.485298 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:06:02.534668 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:02.534641 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm"] Oct 30 22:06:02.535653 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:06:02.535630 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e8dced5_a68f_436b_ab03_da8d8e879315.slice/crio-bb4de6e5731e838563b5a663fc334d40c884d02c576ff4b27c5d2d743fd2decb WatchSource:0}: Error finding container bb4de6e5731e838563b5a663fc334d40c884d02c576ff4b27c5d2d743fd2decb: Status 404 returned error can't find the container with id bb4de6e5731e838563b5a663fc334d40c884d02c576ff4b27c5d2d743fd2decb Oct 30 22:06:02.538090 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:02.538060 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:06:02.538159 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:02.538117 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:06:02.538159 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:02.538149 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:06:02.776923 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:02.776819 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" event={"ID":"ed6f9c77-47bb-4e52-9e6e-1493201547dd","Type":"ContainerStarted","Data":"f7cb38a1c3461af578cf7006f1647421d30cfd936731adcea5240184d9b98871"} Oct 30 22:06:02.776923 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:02.776887 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" event={"ID":"ed6f9c77-47bb-4e52-9e6e-1493201547dd","Type":"ContainerStarted","Data":"2514a25795306ec0595357aaf06ccb4503f1f6665b95a5b9ba01688bad6c1015"} Oct 30 22:06:02.778851 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:02.778826 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" event={"ID":"2e8dced5-a68f-436b-ab03-da8d8e879315","Type":"ContainerStarted","Data":"c30a776aa45e76a131a1ae8b3e8b9727e5edd0d45e6cd298873b1e6be976bc30"} Oct 30 22:06:02.778968 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:02.778868 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" event={"ID":"2e8dced5-a68f-436b-ab03-da8d8e879315","Type":"ContainerStarted","Data":"bb4de6e5731e838563b5a663fc334d40c884d02c576ff4b27c5d2d743fd2decb"} Oct 30 22:06:02.901765 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:02.901707 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" podStartSLOduration=1.901687693 podStartE2EDuration="1.901687693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:02.90024597 +0000 UTC m=+923.170944049" watchObservedRunningTime="2026-10-30 22:06:02.901687693 +0000 UTC m=+923.172385780" Oct 30 22:06:03.025510 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:03.025454 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" podStartSLOduration=2.02543628 podStartE2EDuration="2.02543628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:03.024654521 +0000 UTC m=+923.295352606" watchObservedRunningTime="2026-10-30 22:06:03.02543628 +0000 UTC m=+923.296134357" Oct 30 22:06:03.050947 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:03.050850 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" Oct 30 22:06:03.052536 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:03.052508 5918 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 22:06:03.052669 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:03.052558 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" podUID="ed6f9c77-47bb-4e52-9e6e-1493201547dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 22:06:03.100258 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:03.100111 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" Oct 30 22:06:03.101370 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:03.101347 5918 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 22:06:03.101453 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:03.101396 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" podUID="2e8dced5-a68f-436b-ab03-da8d8e879315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 22:06:04.050708 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:04.050662 5918 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 22:06:04.051197 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:04.050725 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" podUID="ed6f9c77-47bb-4e52-9e6e-1493201547dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 22:06:04.104241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:04.103628 5918 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 22:06:04.104241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:04.103690 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" podUID="2e8dced5-a68f-436b-ab03-da8d8e879315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 22:06:05.051190 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:05.051149 5918 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 22:06:05.051702 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:05.051245 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" podUID="ed6f9c77-47bb-4e52-9e6e-1493201547dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 22:06:05.103243 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:05.101169 5918 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 22:06:05.103243 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:05.101254 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" podUID="2e8dced5-a68f-436b-ab03-da8d8e879315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 22:06:06.050455 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:06.050419 5918 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 22:06:06.050639 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:06.050474 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" podUID="ed6f9c77-47bb-4e52-9e6e-1493201547dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 22:06:06.103241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:06.101555 5918 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 22:06:06.103241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:06.101613 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" podUID="2e8dced5-a68f-436b-ab03-da8d8e879315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 22:06:07.050956 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:07.050917 5918 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 22:06:07.051115 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:07.050980 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" podUID="ed6f9c77-47bb-4e52-9e6e-1493201547dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 22:06:07.111621 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:07.111584 5918 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 22:06:07.112187 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:07.112159 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" podUID="2e8dced5-a68f-436b-ab03-da8d8e879315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 22:06:08.050397 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:08.050368 5918 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 22:06:08.050623 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:08.050594 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" podUID="ed6f9c77-47bb-4e52-9e6e-1493201547dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 22:06:08.100629 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:08.100592 5918 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 22:06:08.100923 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:08.100894 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" podUID="2e8dced5-a68f-436b-ab03-da8d8e879315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 22:06:09.050766 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:09.050725 5918 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 22:06:09.051161 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:09.050794 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" podUID="ed6f9c77-47bb-4e52-9e6e-1493201547dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 22:06:09.100072 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:09.100030 5918 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 22:06:09.100246 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:09.100093 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" podUID="2e8dced5-a68f-436b-ab03-da8d8e879315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 22:06:10.055896 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:10.055864 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" Oct 30 22:06:10.056307 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:10.056159 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" Oct 30 22:06:10.056821 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:10.056805 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" Oct 30 22:06:10.104690 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:10.104662 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" Oct 30 22:06:10.104953 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:10.104936 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" Oct 30 22:06:10.105710 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:10.105690 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" Oct 30 22:06:20.492739 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:20.492706 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb"] Oct 30 22:06:20.496584 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:20.496564 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" Oct 30 22:06:20.594952 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:20.594920 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-home\") pod \"distinct-w0-6a0a09-kserve-b97fffc48-wnhcb\" (UID: \"6536df0a-a317-4b24-aaa9-c4315d2f7dd5\") " pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" Oct 30 22:06:20.594952 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:20.594954 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-model-cache\") pod \"distinct-w0-6a0a09-kserve-b97fffc48-wnhcb\" (UID: \"6536df0a-a317-4b24-aaa9-c4315d2f7dd5\") " pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" Oct 30 22:06:20.595191 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:20.594987 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl8rn\" (UniqueName: \"kubernetes.io/projected/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-kube-api-access-dl8rn\") pod \"distinct-w0-6a0a09-kserve-b97fffc48-wnhcb\" (UID: \"6536df0a-a317-4b24-aaa9-c4315d2f7dd5\") " pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" Oct 30 22:06:20.595191 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:20.595010 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-dshm\") pod \"distinct-w0-6a0a09-kserve-b97fffc48-wnhcb\" (UID: \"6536df0a-a317-4b24-aaa9-c4315d2f7dd5\") " pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" Oct 30 22:06:20.595308 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:20.595268 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-tmp-dir\") pod \"distinct-w0-6a0a09-kserve-b97fffc48-wnhcb\" (UID: \"6536df0a-a317-4b24-aaa9-c4315d2f7dd5\") " pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" Oct 30 22:06:20.595394 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:20.595379 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-tls-certs\") pod \"distinct-w0-6a0a09-kserve-b97fffc48-wnhcb\" (UID: \"6536df0a-a317-4b24-aaa9-c4315d2f7dd5\") " pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" Oct 30 22:06:20.600583 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:20.600565 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6a0a09\"/\"distinct-w0-6a0a09-kserve-self-signed-certs\"" Oct 30 22:06:20.602960 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:20.602938 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb"] Oct 30 22:06:20.696982 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:20.696937 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-tmp-dir\") pod \"distinct-w0-6a0a09-kserve-b97fffc48-wnhcb\" (UID: \"6536df0a-a317-4b24-aaa9-c4315d2f7dd5\") " pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" Oct 30 22:06:20.697186 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:20.697052 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-tls-certs\") pod \"distinct-w0-6a0a09-kserve-b97fffc48-wnhcb\" (UID: \"6536df0a-a317-4b24-aaa9-c4315d2f7dd5\") " pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" Oct 30 22:06:20.697186 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:20.697118 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-home\") pod \"distinct-w0-6a0a09-kserve-b97fffc48-wnhcb\" (UID: \"6536df0a-a317-4b24-aaa9-c4315d2f7dd5\") " pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" Oct 30 22:06:20.697186 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:20.697140 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-model-cache\") pod \"distinct-w0-6a0a09-kserve-b97fffc48-wnhcb\" (UID: \"6536df0a-a317-4b24-aaa9-c4315d2f7dd5\") " pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" Oct 30 22:06:20.697379 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:20.697188 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl8rn\" (UniqueName: \"kubernetes.io/projected/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-kube-api-access-dl8rn\") pod \"distinct-w0-6a0a09-kserve-b97fffc48-wnhcb\" (UID: \"6536df0a-a317-4b24-aaa9-c4315d2f7dd5\") " pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" Oct 30 22:06:20.697379 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:20.697244 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-dshm\") pod \"distinct-w0-6a0a09-kserve-b97fffc48-wnhcb\" (UID: \"6536df0a-a317-4b24-aaa9-c4315d2f7dd5\") " pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" Oct 30 22:06:20.697488 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:20.697460 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-tmp-dir\") pod \"distinct-w0-6a0a09-kserve-b97fffc48-wnhcb\" (UID: \"6536df0a-a317-4b24-aaa9-c4315d2f7dd5\") " pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" Oct 30 22:06:20.697543 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:20.697491 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-home\") pod \"distinct-w0-6a0a09-kserve-b97fffc48-wnhcb\" (UID: \"6536df0a-a317-4b24-aaa9-c4315d2f7dd5\") " pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" Oct 30 22:06:20.697880 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:20.697855 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-model-cache\") pod \"distinct-w0-6a0a09-kserve-b97fffc48-wnhcb\" (UID: \"6536df0a-a317-4b24-aaa9-c4315d2f7dd5\") " pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" Oct 30 22:06:20.699724 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:20.699700 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-dshm\") pod \"distinct-w0-6a0a09-kserve-b97fffc48-wnhcb\" (UID: \"6536df0a-a317-4b24-aaa9-c4315d2f7dd5\") " pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" Oct 30 22:06:20.699856 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:20.699760 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-tls-certs\") pod \"distinct-w0-6a0a09-kserve-b97fffc48-wnhcb\" (UID: \"6536df0a-a317-4b24-aaa9-c4315d2f7dd5\") " pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" Oct 30 22:06:20.761144 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:20.761065 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl8rn\" (UniqueName: \"kubernetes.io/projected/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-kube-api-access-dl8rn\") pod \"distinct-w0-6a0a09-kserve-b97fffc48-wnhcb\" (UID: \"6536df0a-a317-4b24-aaa9-c4315d2f7dd5\") " pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" Oct 30 22:06:20.809547 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:20.809506 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" Oct 30 22:06:21.241589 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:21.241560 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb"] Oct 30 22:06:21.243756 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:06:21.243727 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6536df0a_a317_4b24_aaa9_c4315d2f7dd5.slice/crio-ac6cd2cff8546d6e1900af41898112e5e675af7ba8d8aba015ce09861519dce1 WatchSource:0}: Error finding container ac6cd2cff8546d6e1900af41898112e5e675af7ba8d8aba015ce09861519dce1: Status 404 returned error can't find the container with id ac6cd2cff8546d6e1900af41898112e5e675af7ba8d8aba015ce09861519dce1 Oct 30 22:06:21.276203 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:06:21.276169 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:06:21.938864 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:21.938826 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" event={"ID":"6536df0a-a317-4b24-aaa9-c4315d2f7dd5","Type":"ContainerStarted","Data":"8fa29214bd7bfb44e6c0beeddc5a7d31b2526af18d0966e9703f6d81379a0b29"} Oct 30 22:06:21.938864 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:21.938870 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" event={"ID":"6536df0a-a317-4b24-aaa9-c4315d2f7dd5","Type":"ContainerStarted","Data":"ac6cd2cff8546d6e1900af41898112e5e675af7ba8d8aba015ce09861519dce1"} Oct 30 22:06:22.060525 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:22.060459 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" podStartSLOduration=2.060445655 podStartE2EDuration="2.060445655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:22.059668087 +0000 UTC m=+942.330366171" watchObservedRunningTime="2026-10-30 22:06:22.060445655 +0000 UTC m=+942.331143739" Oct 30 22:06:22.809748 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:22.809713 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" Oct 30 22:06:29.720022 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:29.719987 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z"] Oct 30 22:06:29.723948 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:29.723927 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" Oct 30 22:06:29.778498 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:29.778472 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-d8837a\"/\"openshift-service-ca.crt\"" Oct 30 22:06:29.803831 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:29.803803 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-dshm\") pod \"distinct-w2-d8837a-kserve-7f67bfb654-ph88z\" (UID: \"dc20dd97-eb75-42c5-8e50-6b0b5cb54b45\") " pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" Oct 30 22:06:29.803979 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:29.803839 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-model-cache\") pod \"distinct-w2-d8837a-kserve-7f67bfb654-ph88z\" (UID: \"dc20dd97-eb75-42c5-8e50-6b0b5cb54b45\") " pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" Oct 30 22:06:29.803979 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:29.803922 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvkrd\" (UniqueName: \"kubernetes.io/projected/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-kube-api-access-jvkrd\") pod \"distinct-w2-d8837a-kserve-7f67bfb654-ph88z\" (UID: \"dc20dd97-eb75-42c5-8e50-6b0b5cb54b45\") " pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" Oct 30 22:06:29.804081 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:29.804061 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-home\") pod \"distinct-w2-d8837a-kserve-7f67bfb654-ph88z\" (UID: \"dc20dd97-eb75-42c5-8e50-6b0b5cb54b45\") " pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" Oct 30 22:06:29.804122 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:29.804109 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-tmp-dir\") pod \"distinct-w2-d8837a-kserve-7f67bfb654-ph88z\" (UID: \"dc20dd97-eb75-42c5-8e50-6b0b5cb54b45\") " pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" Oct 30 22:06:29.804191 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:29.804181 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-tls-certs\") pod \"distinct-w2-d8837a-kserve-7f67bfb654-ph88z\" (UID: \"dc20dd97-eb75-42c5-8e50-6b0b5cb54b45\") " pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" Oct 30 22:06:29.825591 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:29.825565 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d8837a\"/\"default-dockercfg-47zx6\"" Oct 30 22:06:29.828234 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:29.828201 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d8837a\"/\"distinct-w2-d8837a-kserve-self-signed-certs\"" Oct 30 22:06:29.830535 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:29.830515 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z"] Oct 30 22:06:29.879262 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:29.879231 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-d8837a\"/\"kube-root-ca.crt\"" Oct 30 22:06:29.905627 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:29.905598 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvkrd\" (UniqueName: \"kubernetes.io/projected/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-kube-api-access-jvkrd\") pod \"distinct-w2-d8837a-kserve-7f67bfb654-ph88z\" (UID: \"dc20dd97-eb75-42c5-8e50-6b0b5cb54b45\") " pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" Oct 30 22:06:29.905808 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:29.905677 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-home\") pod \"distinct-w2-d8837a-kserve-7f67bfb654-ph88z\" (UID: \"dc20dd97-eb75-42c5-8e50-6b0b5cb54b45\") " pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" Oct 30 22:06:29.905808 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:29.905708 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-tmp-dir\") pod \"distinct-w2-d8837a-kserve-7f67bfb654-ph88z\" (UID: \"dc20dd97-eb75-42c5-8e50-6b0b5cb54b45\") " pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" Oct 30 22:06:29.905808 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:29.905744 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-tls-certs\") pod \"distinct-w2-d8837a-kserve-7f67bfb654-ph88z\" (UID: \"dc20dd97-eb75-42c5-8e50-6b0b5cb54b45\") " pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" Oct 30 22:06:29.905808 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:29.905803 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-dshm\") pod \"distinct-w2-d8837a-kserve-7f67bfb654-ph88z\" (UID: \"dc20dd97-eb75-42c5-8e50-6b0b5cb54b45\") " pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" Oct 30 22:06:29.906009 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:29.905820 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-model-cache\") pod \"distinct-w2-d8837a-kserve-7f67bfb654-ph88z\" (UID: \"dc20dd97-eb75-42c5-8e50-6b0b5cb54b45\") " pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" Oct 30 22:06:29.906110 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:29.906090 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-home\") pod \"distinct-w2-d8837a-kserve-7f67bfb654-ph88z\" (UID: \"dc20dd97-eb75-42c5-8e50-6b0b5cb54b45\") " pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" Oct 30 22:06:29.906181 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:29.906162 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-tmp-dir\") pod \"distinct-w2-d8837a-kserve-7f67bfb654-ph88z\" (UID: \"dc20dd97-eb75-42c5-8e50-6b0b5cb54b45\") " pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" Oct 30 22:06:29.906276 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:29.906261 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-model-cache\") pod \"distinct-w2-d8837a-kserve-7f67bfb654-ph88z\" (UID: \"dc20dd97-eb75-42c5-8e50-6b0b5cb54b45\") " pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" Oct 30 22:06:29.908297 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:29.908274 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-dshm\") pod \"distinct-w2-d8837a-kserve-7f67bfb654-ph88z\" (UID: \"dc20dd97-eb75-42c5-8e50-6b0b5cb54b45\") " pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" Oct 30 22:06:29.909013 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:29.908987 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-tls-certs\") pod \"distinct-w2-d8837a-kserve-7f67bfb654-ph88z\" (UID: \"dc20dd97-eb75-42c5-8e50-6b0b5cb54b45\") " pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" Oct 30 22:06:29.969702 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:29.969666 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvkrd\" (UniqueName: \"kubernetes.io/projected/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-kube-api-access-jvkrd\") pod \"distinct-w2-d8837a-kserve-7f67bfb654-ph88z\" (UID: \"dc20dd97-eb75-42c5-8e50-6b0b5cb54b45\") " pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" Oct 30 22:06:30.038138 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:30.038048 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" Oct 30 22:06:30.460287 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:30.460261 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z"] Oct 30 22:06:30.461284 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:06:30.461255 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc20dd97_eb75_42c5_8e50_6b0b5cb54b45.slice/crio-21fffba930719395b19afb49792b4c169d99fb63b76ceec954625edcb36e3f85 WatchSource:0}: Error finding container 21fffba930719395b19afb49792b4c169d99fb63b76ceec954625edcb36e3f85: Status 404 returned error can't find the container with id 21fffba930719395b19afb49792b4c169d99fb63b76ceec954625edcb36e3f85 Oct 30 22:06:30.810336 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:30.810252 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" Oct 30 22:06:30.822969 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:30.822943 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" Oct 30 22:06:30.997691 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:30.997655 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" event={"ID":"dc20dd97-eb75-42c5-8e50-6b0b5cb54b45","Type":"ContainerStarted","Data":"e35ba9adf08cfe1479cba187071134c39cdcbec689be01b01a492a5a363b367e"} Oct 30 22:06:30.997877 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:30.997700 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" event={"ID":"dc20dd97-eb75-42c5-8e50-6b0b5cb54b45","Type":"ContainerStarted","Data":"21fffba930719395b19afb49792b4c169d99fb63b76ceec954625edcb36e3f85"} Oct 30 22:06:31.009414 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:31.009385 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" Oct 30 22:06:31.038382 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:31.038355 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" Oct 30 22:06:31.119431 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:31.119326 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" podStartSLOduration=2.119312876 podStartE2EDuration="2.119312876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:31.11744967 +0000 UTC m=+951.388147766" watchObservedRunningTime="2026-10-30 22:06:31.119312876 +0000 UTC m=+951.390010960" Oct 30 22:06:40.038410 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:40.038380 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" Oct 30 22:06:40.051169 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:40.051145 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" Oct 30 22:06:40.065149 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:40.065122 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" Oct 30 22:06:45.605828 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:45.605796 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2"] Oct 30 22:06:45.610901 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:45.610883 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" Oct 30 22:06:45.668168 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:45.668132 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-9cff60\"/\"distinct-w1-9cff60-kserve-self-signed-certs\"" Oct 30 22:06:45.670141 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:45.670111 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2"] Oct 30 22:06:45.720731 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:45.720690 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05ab563b-c4c2-4fa7-85e4-423141587664-model-cache\") pod \"distinct-w1-9cff60-kserve-759f8f574b-hcgn2\" (UID: \"05ab563b-c4c2-4fa7-85e4-423141587664\") " pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" Oct 30 22:06:45.720932 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:45.720751 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05ab563b-c4c2-4fa7-85e4-423141587664-dshm\") pod \"distinct-w1-9cff60-kserve-759f8f574b-hcgn2\" (UID: \"05ab563b-c4c2-4fa7-85e4-423141587664\") " pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" Oct 30 22:06:45.720932 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:45.720886 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc2c6\" (UniqueName: \"kubernetes.io/projected/05ab563b-c4c2-4fa7-85e4-423141587664-kube-api-access-tc2c6\") pod \"distinct-w1-9cff60-kserve-759f8f574b-hcgn2\" (UID: \"05ab563b-c4c2-4fa7-85e4-423141587664\") " pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" Oct 30 22:06:45.721089 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:45.721070 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05ab563b-c4c2-4fa7-85e4-423141587664-tls-certs\") pod \"distinct-w1-9cff60-kserve-759f8f574b-hcgn2\" (UID: \"05ab563b-c4c2-4fa7-85e4-423141587664\") " pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" Oct 30 22:06:45.721155 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:45.721132 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05ab563b-c4c2-4fa7-85e4-423141587664-tmp-dir\") pod \"distinct-w1-9cff60-kserve-759f8f574b-hcgn2\" (UID: \"05ab563b-c4c2-4fa7-85e4-423141587664\") " pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" Oct 30 22:06:45.721238 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:45.721200 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05ab563b-c4c2-4fa7-85e4-423141587664-home\") pod \"distinct-w1-9cff60-kserve-759f8f574b-hcgn2\" (UID: \"05ab563b-c4c2-4fa7-85e4-423141587664\") " pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" Oct 30 22:06:45.791019 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:45.790971 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7"] Oct 30 22:06:45.795767 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:45.795738 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" Oct 30 22:06:45.822761 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:45.822712 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc2c6\" (UniqueName: \"kubernetes.io/projected/05ab563b-c4c2-4fa7-85e4-423141587664-kube-api-access-tc2c6\") pod \"distinct-w1-9cff60-kserve-759f8f574b-hcgn2\" (UID: \"05ab563b-c4c2-4fa7-85e4-423141587664\") " pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" Oct 30 22:06:45.822958 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:45.822893 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05ab563b-c4c2-4fa7-85e4-423141587664-tls-certs\") pod \"distinct-w1-9cff60-kserve-759f8f574b-hcgn2\" (UID: \"05ab563b-c4c2-4fa7-85e4-423141587664\") " pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" Oct 30 22:06:45.822958 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:45.822948 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05ab563b-c4c2-4fa7-85e4-423141587664-tmp-dir\") pod \"distinct-w1-9cff60-kserve-759f8f574b-hcgn2\" (UID: \"05ab563b-c4c2-4fa7-85e4-423141587664\") " pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" Oct 30 22:06:45.823109 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:45.822995 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05ab563b-c4c2-4fa7-85e4-423141587664-home\") pod \"distinct-w1-9cff60-kserve-759f8f574b-hcgn2\" (UID: \"05ab563b-c4c2-4fa7-85e4-423141587664\") " pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" Oct 30 22:06:45.823109 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:45.823048 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05ab563b-c4c2-4fa7-85e4-423141587664-model-cache\") pod \"distinct-w1-9cff60-kserve-759f8f574b-hcgn2\" (UID: \"05ab563b-c4c2-4fa7-85e4-423141587664\") " pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" Oct 30 22:06:45.823109 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:45.823098 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05ab563b-c4c2-4fa7-85e4-423141587664-dshm\") pod \"distinct-w1-9cff60-kserve-759f8f574b-hcgn2\" (UID: \"05ab563b-c4c2-4fa7-85e4-423141587664\") " pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" Oct 30 22:06:45.824859 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:45.824830 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05ab563b-c4c2-4fa7-85e4-423141587664-model-cache\") pod \"distinct-w1-9cff60-kserve-759f8f574b-hcgn2\" (UID: \"05ab563b-c4c2-4fa7-85e4-423141587664\") " pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" Oct 30 22:06:45.825022 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:45.825003 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05ab563b-c4c2-4fa7-85e4-423141587664-home\") pod \"distinct-w1-9cff60-kserve-759f8f574b-hcgn2\" (UID: \"05ab563b-c4c2-4fa7-85e4-423141587664\") " pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" Oct 30 22:06:45.825588 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:45.825564 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05ab563b-c4c2-4fa7-85e4-423141587664-tmp-dir\") pod \"distinct-w1-9cff60-kserve-759f8f574b-hcgn2\" (UID: \"05ab563b-c4c2-4fa7-85e4-423141587664\") " pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" Oct 30 22:06:45.830033 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:45.829942 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05ab563b-c4c2-4fa7-85e4-423141587664-tls-certs\") pod \"distinct-w1-9cff60-kserve-759f8f574b-hcgn2\" (UID: \"05ab563b-c4c2-4fa7-85e4-423141587664\") " pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" Oct 30 22:06:45.830441 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:45.830418 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05ab563b-c4c2-4fa7-85e4-423141587664-dshm\") pod \"distinct-w1-9cff60-kserve-759f8f574b-hcgn2\" (UID: \"05ab563b-c4c2-4fa7-85e4-423141587664\") " pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" Oct 30 22:06:45.859665 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:45.859591 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7"] Oct 30 22:06:45.889405 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:45.889373 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc2c6\" (UniqueName: \"kubernetes.io/projected/05ab563b-c4c2-4fa7-85e4-423141587664-kube-api-access-tc2c6\") pod \"distinct-w1-9cff60-kserve-759f8f574b-hcgn2\" (UID: \"05ab563b-c4c2-4fa7-85e4-423141587664\") " pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" Oct 30 22:06:45.924317 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:45.924283 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/845afe35-58c3-43c9-b276-66247d39c613-istio-token\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7\" (UID: \"845afe35-58c3-43c9-b276-66247d39c613\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" Oct 30 22:06:45.924528 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:45.924412 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/845afe35-58c3-43c9-b276-66247d39c613-istio-podinfo\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7\" (UID: \"845afe35-58c3-43c9-b276-66247d39c613\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" Oct 30 22:06:45.924528 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:45.924469 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5mqm\" (UniqueName: \"kubernetes.io/projected/845afe35-58c3-43c9-b276-66247d39c613-kube-api-access-h5mqm\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7\" (UID: \"845afe35-58c3-43c9-b276-66247d39c613\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" Oct 30 22:06:45.924651 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:45.924569 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/845afe35-58c3-43c9-b276-66247d39c613-credential-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7\" (UID: \"845afe35-58c3-43c9-b276-66247d39c613\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" Oct 30 22:06:45.924714 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:45.924669 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/845afe35-58c3-43c9-b276-66247d39c613-workload-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7\" (UID: \"845afe35-58c3-43c9-b276-66247d39c613\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" Oct 30 22:06:45.924714 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:45.924696 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/845afe35-58c3-43c9-b276-66247d39c613-istio-data\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7\" (UID: \"845afe35-58c3-43c9-b276-66247d39c613\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" Oct 30 22:06:45.924871 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:45.924846 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/845afe35-58c3-43c9-b276-66247d39c613-workload-certs\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7\" (UID: \"845afe35-58c3-43c9-b276-66247d39c613\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" Oct 30 22:06:45.924925 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:45.924915 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/845afe35-58c3-43c9-b276-66247d39c613-istiod-ca-cert\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7\" (UID: \"845afe35-58c3-43c9-b276-66247d39c613\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" Oct 30 22:06:45.924978 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:45.924941 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/845afe35-58c3-43c9-b276-66247d39c613-istio-envoy\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7\" (UID: \"845afe35-58c3-43c9-b276-66247d39c613\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" Oct 30 22:06:45.927096 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:45.927070 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" Oct 30 22:06:46.049250 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.034122 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/845afe35-58c3-43c9-b276-66247d39c613-istio-token\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7\" (UID: \"845afe35-58c3-43c9-b276-66247d39c613\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" Oct 30 22:06:46.049250 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.034209 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/845afe35-58c3-43c9-b276-66247d39c613-istio-podinfo\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7\" (UID: \"845afe35-58c3-43c9-b276-66247d39c613\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" Oct 30 22:06:46.049250 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.034271 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5mqm\" (UniqueName: \"kubernetes.io/projected/845afe35-58c3-43c9-b276-66247d39c613-kube-api-access-h5mqm\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7\" (UID: \"845afe35-58c3-43c9-b276-66247d39c613\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" Oct 30 22:06:46.049250 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.034331 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/845afe35-58c3-43c9-b276-66247d39c613-credential-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7\" (UID: \"845afe35-58c3-43c9-b276-66247d39c613\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" Oct 30 22:06:46.049250 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.034392 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/845afe35-58c3-43c9-b276-66247d39c613-workload-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7\" (UID: \"845afe35-58c3-43c9-b276-66247d39c613\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" Oct 30 22:06:46.049250 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.034417 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/845afe35-58c3-43c9-b276-66247d39c613-istio-data\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7\" (UID: \"845afe35-58c3-43c9-b276-66247d39c613\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" Oct 30 22:06:46.049250 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.034547 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/845afe35-58c3-43c9-b276-66247d39c613-workload-certs\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7\" (UID: \"845afe35-58c3-43c9-b276-66247d39c613\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" Oct 30 22:06:46.049250 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.034614 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/845afe35-58c3-43c9-b276-66247d39c613-istiod-ca-cert\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7\" (UID: \"845afe35-58c3-43c9-b276-66247d39c613\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" Oct 30 22:06:46.049250 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.034646 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/845afe35-58c3-43c9-b276-66247d39c613-istio-envoy\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7\" (UID: \"845afe35-58c3-43c9-b276-66247d39c613\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" Oct 30 22:06:46.049250 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.036399 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/845afe35-58c3-43c9-b276-66247d39c613-credential-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7\" (UID: \"845afe35-58c3-43c9-b276-66247d39c613\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" Oct 30 22:06:46.049250 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.040701 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/845afe35-58c3-43c9-b276-66247d39c613-istio-podinfo\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7\" (UID: \"845afe35-58c3-43c9-b276-66247d39c613\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" Oct 30 22:06:46.049250 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.041241 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/845afe35-58c3-43c9-b276-66247d39c613-istio-data\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7\" (UID: \"845afe35-58c3-43c9-b276-66247d39c613\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" Oct 30 22:06:46.049250 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.041477 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/845afe35-58c3-43c9-b276-66247d39c613-workload-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7\" (UID: \"845afe35-58c3-43c9-b276-66247d39c613\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" Oct 30 22:06:46.049250 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.041700 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/845afe35-58c3-43c9-b276-66247d39c613-workload-certs\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7\" (UID: \"845afe35-58c3-43c9-b276-66247d39c613\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" Oct 30 22:06:46.049250 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.042092 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/845afe35-58c3-43c9-b276-66247d39c613-istiod-ca-cert\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7\" (UID: \"845afe35-58c3-43c9-b276-66247d39c613\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" Oct 30 22:06:46.049250 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.046417 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/845afe35-58c3-43c9-b276-66247d39c613-istio-envoy\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7\" (UID: \"845afe35-58c3-43c9-b276-66247d39c613\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" Oct 30 22:06:46.106849 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.106812 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/845afe35-58c3-43c9-b276-66247d39c613-istio-token\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7\" (UID: \"845afe35-58c3-43c9-b276-66247d39c613\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" Oct 30 22:06:46.107950 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.107923 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5mqm\" (UniqueName: \"kubernetes.io/projected/845afe35-58c3-43c9-b276-66247d39c613-kube-api-access-h5mqm\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7\" (UID: \"845afe35-58c3-43c9-b276-66247d39c613\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" Oct 30 22:06:46.111714 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.111657 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" Oct 30 22:06:46.148423 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.148387 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs"] Oct 30 22:06:46.153679 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.153648 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" Oct 30 22:06:46.216465 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.216303 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs"] Oct 30 22:06:46.237370 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.237343 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba68152b-0571-406a-8355-8fc4d3aae15c-istiod-ca-cert\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs\" (UID: \"ba68152b-0571-406a-8355-8fc4d3aae15c\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" Oct 30 22:06:46.237526 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.237376 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba68152b-0571-406a-8355-8fc4d3aae15c-istio-envoy\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs\" (UID: \"ba68152b-0571-406a-8355-8fc4d3aae15c\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" Oct 30 22:06:46.237526 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.237412 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba68152b-0571-406a-8355-8fc4d3aae15c-credential-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs\" (UID: \"ba68152b-0571-406a-8355-8fc4d3aae15c\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" Oct 30 22:06:46.237526 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.237488 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba68152b-0571-406a-8355-8fc4d3aae15c-workload-certs\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs\" (UID: \"ba68152b-0571-406a-8355-8fc4d3aae15c\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" Oct 30 22:06:46.237628 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.237601 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba68152b-0571-406a-8355-8fc4d3aae15c-workload-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs\" (UID: \"ba68152b-0571-406a-8355-8fc4d3aae15c\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" Oct 30 22:06:46.237701 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.237669 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba68152b-0571-406a-8355-8fc4d3aae15c-istio-data\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs\" (UID: \"ba68152b-0571-406a-8355-8fc4d3aae15c\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" Oct 30 22:06:46.237793 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.237773 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba68152b-0571-406a-8355-8fc4d3aae15c-istio-podinfo\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs\" (UID: \"ba68152b-0571-406a-8355-8fc4d3aae15c\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" Oct 30 22:06:46.237828 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.237817 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcgqj\" (UniqueName: \"kubernetes.io/projected/ba68152b-0571-406a-8355-8fc4d3aae15c-kube-api-access-fcgqj\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs\" (UID: \"ba68152b-0571-406a-8355-8fc4d3aae15c\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" Oct 30 22:06:46.237861 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.237846 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba68152b-0571-406a-8355-8fc4d3aae15c-istio-token\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs\" (UID: \"ba68152b-0571-406a-8355-8fc4d3aae15c\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" Oct 30 22:06:46.332110 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.332086 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2"] Oct 30 22:06:46.333137 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:06:46.333114 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05ab563b_c4c2_4fa7_85e4_423141587664.slice/crio-f80482c7e8ebd6dba8823f6014803286bd4cb2db08eb044c2c4be1378d51f4a3 WatchSource:0}: Error finding container f80482c7e8ebd6dba8823f6014803286bd4cb2db08eb044c2c4be1378d51f4a3: Status 404 returned error can't find the container with id f80482c7e8ebd6dba8823f6014803286bd4cb2db08eb044c2c4be1378d51f4a3 Oct 30 22:06:46.338526 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.338501 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba68152b-0571-406a-8355-8fc4d3aae15c-istiod-ca-cert\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs\" (UID: \"ba68152b-0571-406a-8355-8fc4d3aae15c\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" Oct 30 22:06:46.338634 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.338539 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba68152b-0571-406a-8355-8fc4d3aae15c-istio-envoy\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs\" (UID: \"ba68152b-0571-406a-8355-8fc4d3aae15c\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" Oct 30 22:06:46.338634 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.338573 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba68152b-0571-406a-8355-8fc4d3aae15c-credential-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs\" (UID: \"ba68152b-0571-406a-8355-8fc4d3aae15c\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" Oct 30 22:06:46.338634 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.338593 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba68152b-0571-406a-8355-8fc4d3aae15c-workload-certs\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs\" (UID: \"ba68152b-0571-406a-8355-8fc4d3aae15c\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" Oct 30 22:06:46.338778 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.338673 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba68152b-0571-406a-8355-8fc4d3aae15c-workload-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs\" (UID: \"ba68152b-0571-406a-8355-8fc4d3aae15c\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" Oct 30 22:06:46.338778 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.338765 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba68152b-0571-406a-8355-8fc4d3aae15c-istio-data\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs\" (UID: \"ba68152b-0571-406a-8355-8fc4d3aae15c\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" Oct 30 22:06:46.338865 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.338798 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba68152b-0571-406a-8355-8fc4d3aae15c-istio-podinfo\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs\" (UID: \"ba68152b-0571-406a-8355-8fc4d3aae15c\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" Oct 30 22:06:46.338865 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.338841 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcgqj\" (UniqueName: \"kubernetes.io/projected/ba68152b-0571-406a-8355-8fc4d3aae15c-kube-api-access-fcgqj\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs\" (UID: \"ba68152b-0571-406a-8355-8fc4d3aae15c\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" Oct 30 22:06:46.338965 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.338873 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba68152b-0571-406a-8355-8fc4d3aae15c-istio-token\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs\" (UID: \"ba68152b-0571-406a-8355-8fc4d3aae15c\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" Oct 30 22:06:46.339449 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.339052 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba68152b-0571-406a-8355-8fc4d3aae15c-credential-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs\" (UID: \"ba68152b-0571-406a-8355-8fc4d3aae15c\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" Oct 30 22:06:46.339449 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.339093 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba68152b-0571-406a-8355-8fc4d3aae15c-workload-certs\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs\" (UID: \"ba68152b-0571-406a-8355-8fc4d3aae15c\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" Oct 30 22:06:46.339449 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.339251 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba68152b-0571-406a-8355-8fc4d3aae15c-workload-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs\" (UID: \"ba68152b-0571-406a-8355-8fc4d3aae15c\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" Oct 30 22:06:46.339449 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.339396 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba68152b-0571-406a-8355-8fc4d3aae15c-istio-data\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs\" (UID: \"ba68152b-0571-406a-8355-8fc4d3aae15c\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" Oct 30 22:06:46.339672 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.339381 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba68152b-0571-406a-8355-8fc4d3aae15c-istiod-ca-cert\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs\" (UID: \"ba68152b-0571-406a-8355-8fc4d3aae15c\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" Oct 30 22:06:46.340965 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.340944 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba68152b-0571-406a-8355-8fc4d3aae15c-istio-envoy\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs\" (UID: \"ba68152b-0571-406a-8355-8fc4d3aae15c\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" Oct 30 22:06:46.341509 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.341489 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba68152b-0571-406a-8355-8fc4d3aae15c-istio-podinfo\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs\" (UID: \"ba68152b-0571-406a-8355-8fc4d3aae15c\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" Oct 30 22:06:46.401806 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.401781 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba68152b-0571-406a-8355-8fc4d3aae15c-istio-token\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs\" (UID: \"ba68152b-0571-406a-8355-8fc4d3aae15c\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" Oct 30 22:06:46.402245 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.402189 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcgqj\" (UniqueName: \"kubernetes.io/projected/ba68152b-0571-406a-8355-8fc4d3aae15c-kube-api-access-fcgqj\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs\" (UID: \"ba68152b-0571-406a-8355-8fc4d3aae15c\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" Oct 30 22:06:46.472246 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.472192 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" Oct 30 22:06:46.510494 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.510463 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7"] Oct 30 22:06:46.511276 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:06:46.511207 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod845afe35_58c3_43c9_b276_66247d39c613.slice/crio-022e83e2c0b3895bc2dc12755e4af19a43f259233af25085fc338c6ff5516310 WatchSource:0}: Error finding container 022e83e2c0b3895bc2dc12755e4af19a43f259233af25085fc338c6ff5516310: Status 404 returned error can't find the container with id 022e83e2c0b3895bc2dc12755e4af19a43f259233af25085fc338c6ff5516310 Oct 30 22:06:46.514461 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.514427 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:06:46.514564 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.514491 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:06:46.514564 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.514521 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:06:46.868683 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.868500 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs"] Oct 30 22:06:46.870801 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:06:46.870770 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba68152b_0571_406a_8355_8fc4d3aae15c.slice/crio-b7bb40545b2f3fff0babd928a4125ebb6757cb725bfa4aa2afc675944f532320 WatchSource:0}: Error finding container b7bb40545b2f3fff0babd928a4125ebb6757cb725bfa4aa2afc675944f532320: Status 404 returned error can't find the container with id b7bb40545b2f3fff0babd928a4125ebb6757cb725bfa4aa2afc675944f532320 Oct 30 22:06:46.873563 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.873515 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:06:46.873651 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.873610 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:06:46.873691 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:46.873654 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:06:47.123752 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:47.123643 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" event={"ID":"05ab563b-c4c2-4fa7-85e4-423141587664","Type":"ContainerStarted","Data":"014f7ad8d5c88a866b43cf490faf12cd4c9fdcfc1e82acff8afa65b3c1a4d96d"} Oct 30 22:06:47.123752 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:47.123690 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" event={"ID":"05ab563b-c4c2-4fa7-85e4-423141587664","Type":"ContainerStarted","Data":"f80482c7e8ebd6dba8823f6014803286bd4cb2db08eb044c2c4be1378d51f4a3"} Oct 30 22:06:47.125503 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:47.125476 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" event={"ID":"ba68152b-0571-406a-8355-8fc4d3aae15c","Type":"ContainerStarted","Data":"bfeec9849da7882ff42c68e26d34aed161cfabafebd6beeafe2ad36ef7f4367c"} Oct 30 22:06:47.125645 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:47.125524 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" event={"ID":"ba68152b-0571-406a-8355-8fc4d3aae15c","Type":"ContainerStarted","Data":"b7bb40545b2f3fff0babd928a4125ebb6757cb725bfa4aa2afc675944f532320"} Oct 30 22:06:47.127203 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:47.127178 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" event={"ID":"845afe35-58c3-43c9-b276-66247d39c613","Type":"ContainerStarted","Data":"2d42a7149e6fb2265b23cd05a63f2afbb7efbf0c1b9ecac8680ecc4fa20369f0"} Oct 30 22:06:47.127330 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:47.127244 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" event={"ID":"845afe35-58c3-43c9-b276-66247d39c613","Type":"ContainerStarted","Data":"022e83e2c0b3895bc2dc12755e4af19a43f259233af25085fc338c6ff5516310"} Oct 30 22:06:47.245712 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:47.245659 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" podStartSLOduration=2.245643806 podStartE2EDuration="2.245643806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:47.24387995 +0000 UTC m=+967.514578035" watchObservedRunningTime="2026-10-30 22:06:47.245643806 +0000 UTC m=+967.516341890" Oct 30 22:06:47.472910 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:47.472877 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" Oct 30 22:06:47.475191 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:47.475147 5918 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 22:06:47.475355 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:47.475206 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" podUID="ba68152b-0571-406a-8355-8fc4d3aae15c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 22:06:47.530125 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:47.528829 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" podStartSLOduration=2.528804258 podStartE2EDuration="2.528804258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:47.373593308 +0000 UTC m=+967.644291393" watchObservedRunningTime="2026-10-30 22:06:47.528804258 +0000 UTC m=+967.799502344" Oct 30 22:06:47.932425 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:47.932376 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" Oct 30 22:06:48.119243 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:48.112154 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" Oct 30 22:06:48.140270 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:48.139401 5918 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 22:06:48.140270 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:48.139488 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" podUID="845afe35-58c3-43c9-b276-66247d39c613" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 22:06:48.473061 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:48.473030 5918 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 22:06:48.473282 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:48.473079 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" podUID="ba68152b-0571-406a-8355-8fc4d3aae15c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 22:06:49.125242 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:49.124622 5918 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 22:06:49.125242 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:49.124681 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" podUID="845afe35-58c3-43c9-b276-66247d39c613" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 22:06:49.473311 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:49.473272 5918 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 22:06:49.473493 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:49.473340 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" podUID="ba68152b-0571-406a-8355-8fc4d3aae15c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 22:06:50.115241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:50.114641 5918 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 22:06:50.115241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:50.114705 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" podUID="845afe35-58c3-43c9-b276-66247d39c613" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 22:06:50.473072 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:50.472705 5918 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 22:06:50.473072 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:50.472767 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" podUID="ba68152b-0571-406a-8355-8fc4d3aae15c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 22:06:51.112650 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:51.112615 5918 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 22:06:51.112944 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:51.112915 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" podUID="845afe35-58c3-43c9-b276-66247d39c613" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 22:06:51.274928 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:06:51.274898 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:06:51.472906 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:51.472871 5918 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 22:06:51.473126 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:51.472925 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" podUID="ba68152b-0571-406a-8355-8fc4d3aae15c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 22:06:52.119250 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:52.118795 5918 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 22:06:52.119250 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:52.118876 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" podUID="845afe35-58c3-43c9-b276-66247d39c613" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 22:06:52.473036 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:52.473007 5918 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 22:06:52.473231 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:52.473056 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" podUID="ba68152b-0571-406a-8355-8fc4d3aae15c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 22:06:53.112384 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:53.112339 5918 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 22:06:53.112601 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:53.112440 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" podUID="845afe35-58c3-43c9-b276-66247d39c613" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 22:06:53.472885 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:53.472839 5918 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 22:06:53.473107 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:53.472895 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" podUID="ba68152b-0571-406a-8355-8fc4d3aae15c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 22:06:54.116690 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:54.116653 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" Oct 30 22:06:54.117552 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:54.117475 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" Oct 30 22:06:54.118203 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:54.118177 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" Oct 30 22:06:54.343204 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:54.343146 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" podStartSLOduration=8.343133427 podStartE2EDuration="8.343133427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:47.500552418 +0000 UTC m=+967.771250502" watchObservedRunningTime="2026-10-30 22:06:54.343133427 +0000 UTC m=+974.613831519" Oct 30 22:06:54.478490 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:54.478461 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" Oct 30 22:06:54.478894 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:54.478861 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" Oct 30 22:06:54.479722 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:54.479704 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" Oct 30 22:06:55.927202 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:55.927173 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" Oct 30 22:06:55.941255 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:55.941230 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" Oct 30 22:06:56.207444 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:06:56.207359 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" Oct 30 22:07:21.275639 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:07:21.275563 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:07:51.275988 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:07:51.275948 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:07:59.153731 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:07:59.153697 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz"] Oct 30 22:07:59.157762 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:07:59.157738 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" Oct 30 22:07:59.239170 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:07:59.239141 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/395b4d7d-4aeb-4576-b616-67638bb1a2c2-model-cache\") pod \"distinct-2-w0-6a0a09-kserve-744f74cf9-784kz\" (UID: \"395b4d7d-4aeb-4576-b616-67638bb1a2c2\") " pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" Oct 30 22:07:59.239170 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:07:59.239172 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-276vk\" (UniqueName: \"kubernetes.io/projected/395b4d7d-4aeb-4576-b616-67638bb1a2c2-kube-api-access-276vk\") pod \"distinct-2-w0-6a0a09-kserve-744f74cf9-784kz\" (UID: \"395b4d7d-4aeb-4576-b616-67638bb1a2c2\") " pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" Oct 30 22:07:59.239416 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:07:59.239201 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/395b4d7d-4aeb-4576-b616-67638bb1a2c2-tls-certs\") pod \"distinct-2-w0-6a0a09-kserve-744f74cf9-784kz\" (UID: \"395b4d7d-4aeb-4576-b616-67638bb1a2c2\") " pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" Oct 30 22:07:59.239416 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:07:59.239365 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/395b4d7d-4aeb-4576-b616-67638bb1a2c2-tmp-dir\") pod \"distinct-2-w0-6a0a09-kserve-744f74cf9-784kz\" (UID: \"395b4d7d-4aeb-4576-b616-67638bb1a2c2\") " pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" Oct 30 22:07:59.239416 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:07:59.239405 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/395b4d7d-4aeb-4576-b616-67638bb1a2c2-dshm\") pod \"distinct-2-w0-6a0a09-kserve-744f74cf9-784kz\" (UID: \"395b4d7d-4aeb-4576-b616-67638bb1a2c2\") " pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" Oct 30 22:07:59.239555 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:07:59.239430 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/395b4d7d-4aeb-4576-b616-67638bb1a2c2-home\") pod \"distinct-2-w0-6a0a09-kserve-744f74cf9-784kz\" (UID: \"395b4d7d-4aeb-4576-b616-67638bb1a2c2\") " pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" Oct 30 22:07:59.261701 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:07:59.261673 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6a0a09\"/\"distinct-2-w0-6a0a09-kserve-self-signed-certs\"" Oct 30 22:07:59.264500 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:07:59.264475 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz"] Oct 30 22:07:59.340312 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:07:59.340281 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/395b4d7d-4aeb-4576-b616-67638bb1a2c2-tls-certs\") pod \"distinct-2-w0-6a0a09-kserve-744f74cf9-784kz\" (UID: \"395b4d7d-4aeb-4576-b616-67638bb1a2c2\") " pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" Oct 30 22:07:59.340490 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:07:59.340370 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/395b4d7d-4aeb-4576-b616-67638bb1a2c2-tmp-dir\") pod \"distinct-2-w0-6a0a09-kserve-744f74cf9-784kz\" (UID: \"395b4d7d-4aeb-4576-b616-67638bb1a2c2\") " pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" Oct 30 22:07:59.340490 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:07:59.340409 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/395b4d7d-4aeb-4576-b616-67638bb1a2c2-dshm\") pod \"distinct-2-w0-6a0a09-kserve-744f74cf9-784kz\" (UID: \"395b4d7d-4aeb-4576-b616-67638bb1a2c2\") " pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" Oct 30 22:07:59.340490 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:07:59.340441 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/395b4d7d-4aeb-4576-b616-67638bb1a2c2-home\") pod \"distinct-2-w0-6a0a09-kserve-744f74cf9-784kz\" (UID: \"395b4d7d-4aeb-4576-b616-67638bb1a2c2\") " pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" Oct 30 22:07:59.340690 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:07:59.340675 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/395b4d7d-4aeb-4576-b616-67638bb1a2c2-model-cache\") pod \"distinct-2-w0-6a0a09-kserve-744f74cf9-784kz\" (UID: \"395b4d7d-4aeb-4576-b616-67638bb1a2c2\") " pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" Oct 30 22:07:59.340798 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:07:59.340709 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-276vk\" (UniqueName: \"kubernetes.io/projected/395b4d7d-4aeb-4576-b616-67638bb1a2c2-kube-api-access-276vk\") pod \"distinct-2-w0-6a0a09-kserve-744f74cf9-784kz\" (UID: \"395b4d7d-4aeb-4576-b616-67638bb1a2c2\") " pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" Oct 30 22:07:59.340798 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:07:59.340744 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/395b4d7d-4aeb-4576-b616-67638bb1a2c2-tmp-dir\") pod \"distinct-2-w0-6a0a09-kserve-744f74cf9-784kz\" (UID: \"395b4d7d-4aeb-4576-b616-67638bb1a2c2\") " pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" Oct 30 22:07:59.340911 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:07:59.340825 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/395b4d7d-4aeb-4576-b616-67638bb1a2c2-home\") pod \"distinct-2-w0-6a0a09-kserve-744f74cf9-784kz\" (UID: \"395b4d7d-4aeb-4576-b616-67638bb1a2c2\") " pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" Oct 30 22:07:59.341052 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:07:59.341027 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/395b4d7d-4aeb-4576-b616-67638bb1a2c2-model-cache\") pod \"distinct-2-w0-6a0a09-kserve-744f74cf9-784kz\" (UID: \"395b4d7d-4aeb-4576-b616-67638bb1a2c2\") " pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" Oct 30 22:07:59.342864 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:07:59.342845 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/395b4d7d-4aeb-4576-b616-67638bb1a2c2-dshm\") pod \"distinct-2-w0-6a0a09-kserve-744f74cf9-784kz\" (UID: \"395b4d7d-4aeb-4576-b616-67638bb1a2c2\") " pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" Oct 30 22:07:59.343117 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:07:59.343100 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/395b4d7d-4aeb-4576-b616-67638bb1a2c2-tls-certs\") pod \"distinct-2-w0-6a0a09-kserve-744f74cf9-784kz\" (UID: \"395b4d7d-4aeb-4576-b616-67638bb1a2c2\") " pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" Oct 30 22:07:59.405513 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:07:59.405436 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-276vk\" (UniqueName: \"kubernetes.io/projected/395b4d7d-4aeb-4576-b616-67638bb1a2c2-kube-api-access-276vk\") pod \"distinct-2-w0-6a0a09-kserve-744f74cf9-784kz\" (UID: \"395b4d7d-4aeb-4576-b616-67638bb1a2c2\") " pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" Oct 30 22:07:59.469725 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:07:59.469685 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" Oct 30 22:07:59.896001 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:07:59.895972 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz"] Oct 30 22:07:59.897273 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:07:59.897246 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod395b4d7d_4aeb_4576_b616_67638bb1a2c2.slice/crio-92eafdef17522c187b7992f548775a539c62e64bfae1fdbd6670dd7fa3cb2aba WatchSource:0}: Error finding container 92eafdef17522c187b7992f548775a539c62e64bfae1fdbd6670dd7fa3cb2aba: Status 404 returned error can't find the container with id 92eafdef17522c187b7992f548775a539c62e64bfae1fdbd6670dd7fa3cb2aba Oct 30 22:08:00.653273 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:00.653233 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" event={"ID":"395b4d7d-4aeb-4576-b616-67638bb1a2c2","Type":"ContainerStarted","Data":"fe1b8c437772f9d9f45f5d72ead538d279183e43337d3ab38bc2a80f719d123d"} Oct 30 22:08:00.653784 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:00.653287 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" event={"ID":"395b4d7d-4aeb-4576-b616-67638bb1a2c2","Type":"ContainerStarted","Data":"92eafdef17522c187b7992f548775a539c62e64bfae1fdbd6670dd7fa3cb2aba"} Oct 30 22:08:00.775170 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:00.775112 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" podStartSLOduration=1.775098783 podStartE2EDuration="1.775098783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:00.773550451 +0000 UTC m=+1041.044248543" watchObservedRunningTime="2026-10-30 22:08:00.775098783 +0000 UTC m=+1041.045796868" Oct 30 22:08:01.470581 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:01.470546 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" Oct 30 22:08:09.470647 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:09.470612 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" Oct 30 22:08:09.483421 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:09.483393 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" Oct 30 22:08:09.718604 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:09.718573 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" Oct 30 22:08:21.274824 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:08:21.274784 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:08:31.356733 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.356696 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8"] Oct 30 22:08:31.360711 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.360693 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" Oct 30 22:08:31.404575 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.404543 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r"] Oct 30 22:08:31.409096 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.409078 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" Oct 30 22:08:31.438466 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.438438 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-istio-podinfo\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8\" (UID: \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" Oct 30 22:08:31.438639 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.438563 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-istio-token\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8\" (UID: \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" Oct 30 22:08:31.438639 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.438626 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-istio-data\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8\" (UID: \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" Oct 30 22:08:31.438756 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.438677 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2jtc\" (UniqueName: \"kubernetes.io/projected/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-kube-api-access-l2jtc\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8\" (UID: \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" Oct 30 22:08:31.438815 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.438758 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-istio-envoy\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8\" (UID: \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" Oct 30 22:08:31.438815 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.438783 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-workload-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8\" (UID: \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" Oct 30 22:08:31.438815 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.438808 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-credential-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8\" (UID: \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" Oct 30 22:08:31.438971 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.438825 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-istiod-ca-cert\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8\" (UID: \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" Oct 30 22:08:31.439032 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.438977 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-workload-certs\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8\" (UID: \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" Oct 30 22:08:31.454642 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.454617 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8"] Oct 30 22:08:31.540591 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.540554 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-istio-token\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8\" (UID: \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" Oct 30 22:08:31.540763 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.540606 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1181a098-41e1-4d6a-8582-bda860f8bd99-istio-envoy\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r\" (UID: \"1181a098-41e1-4d6a-8582-bda860f8bd99\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" Oct 30 22:08:31.540763 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.540628 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1181a098-41e1-4d6a-8582-bda860f8bd99-istio-podinfo\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r\" (UID: \"1181a098-41e1-4d6a-8582-bda860f8bd99\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" Oct 30 22:08:31.540763 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.540655 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1181a098-41e1-4d6a-8582-bda860f8bd99-credential-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r\" (UID: \"1181a098-41e1-4d6a-8582-bda860f8bd99\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" Oct 30 22:08:31.540910 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.540796 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq8qd\" (UniqueName: \"kubernetes.io/projected/1181a098-41e1-4d6a-8582-bda860f8bd99-kube-api-access-vq8qd\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r\" (UID: \"1181a098-41e1-4d6a-8582-bda860f8bd99\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" Oct 30 22:08:31.540910 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.540862 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-istio-data\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8\" (UID: \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" Oct 30 22:08:31.541008 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.540960 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2jtc\" (UniqueName: \"kubernetes.io/projected/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-kube-api-access-l2jtc\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8\" (UID: \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" Oct 30 22:08:31.541063 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.541012 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1181a098-41e1-4d6a-8582-bda860f8bd99-istio-token\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r\" (UID: \"1181a098-41e1-4d6a-8582-bda860f8bd99\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" Oct 30 22:08:31.541063 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.541053 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-istio-envoy\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8\" (UID: \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" Oct 30 22:08:31.541174 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.541083 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1181a098-41e1-4d6a-8582-bda860f8bd99-workload-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r\" (UID: \"1181a098-41e1-4d6a-8582-bda860f8bd99\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" Oct 30 22:08:31.541174 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.541159 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-workload-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8\" (UID: \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" Oct 30 22:08:31.541300 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.541227 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-credential-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8\" (UID: \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" Oct 30 22:08:31.541300 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.541255 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-istiod-ca-cert\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8\" (UID: \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" Oct 30 22:08:31.541395 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.541299 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-istio-data\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8\" (UID: \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" Oct 30 22:08:31.541395 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.541342 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-workload-certs\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8\" (UID: \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" Oct 30 22:08:31.541395 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.541375 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1181a098-41e1-4d6a-8582-bda860f8bd99-workload-certs\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r\" (UID: \"1181a098-41e1-4d6a-8582-bda860f8bd99\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" Oct 30 22:08:31.541539 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.541441 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-istio-podinfo\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8\" (UID: \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" Oct 30 22:08:31.541539 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.541519 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1181a098-41e1-4d6a-8582-bda860f8bd99-istio-data\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r\" (UID: \"1181a098-41e1-4d6a-8582-bda860f8bd99\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" Oct 30 22:08:31.541654 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.541559 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-workload-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8\" (UID: \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" Oct 30 22:08:31.541654 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.541587 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-credential-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8\" (UID: \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" Oct 30 22:08:31.541654 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.541634 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1181a098-41e1-4d6a-8582-bda860f8bd99-istiod-ca-cert\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r\" (UID: \"1181a098-41e1-4d6a-8582-bda860f8bd99\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" Oct 30 22:08:31.541971 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.541950 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-workload-certs\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8\" (UID: \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" Oct 30 22:08:31.542075 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.542011 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-istiod-ca-cert\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8\" (UID: \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" Oct 30 22:08:31.544157 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.544132 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-istio-podinfo\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8\" (UID: \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" Oct 30 22:08:31.544268 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.544141 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-istio-envoy\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8\" (UID: \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" Oct 30 22:08:31.555593 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.555567 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r"] Oct 30 22:08:31.605674 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.605634 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-istio-token\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8\" (UID: \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" Oct 30 22:08:31.605918 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.605895 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2jtc\" (UniqueName: \"kubernetes.io/projected/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-kube-api-access-l2jtc\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8\" (UID: \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" Oct 30 22:08:31.643315 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.643245 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1181a098-41e1-4d6a-8582-bda860f8bd99-istio-data\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r\" (UID: \"1181a098-41e1-4d6a-8582-bda860f8bd99\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" Oct 30 22:08:31.643315 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.643285 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1181a098-41e1-4d6a-8582-bda860f8bd99-istiod-ca-cert\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r\" (UID: \"1181a098-41e1-4d6a-8582-bda860f8bd99\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" Oct 30 22:08:31.643513 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.643332 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1181a098-41e1-4d6a-8582-bda860f8bd99-istio-envoy\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r\" (UID: \"1181a098-41e1-4d6a-8582-bda860f8bd99\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" Oct 30 22:08:31.643513 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.643348 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1181a098-41e1-4d6a-8582-bda860f8bd99-istio-podinfo\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r\" (UID: \"1181a098-41e1-4d6a-8582-bda860f8bd99\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" Oct 30 22:08:31.643513 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.643480 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1181a098-41e1-4d6a-8582-bda860f8bd99-credential-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r\" (UID: \"1181a098-41e1-4d6a-8582-bda860f8bd99\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" Oct 30 22:08:31.643688 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.643527 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq8qd\" (UniqueName: \"kubernetes.io/projected/1181a098-41e1-4d6a-8582-bda860f8bd99-kube-api-access-vq8qd\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r\" (UID: \"1181a098-41e1-4d6a-8582-bda860f8bd99\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" Oct 30 22:08:31.643688 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.643561 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1181a098-41e1-4d6a-8582-bda860f8bd99-istio-data\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r\" (UID: \"1181a098-41e1-4d6a-8582-bda860f8bd99\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" Oct 30 22:08:31.643688 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.643628 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1181a098-41e1-4d6a-8582-bda860f8bd99-istio-token\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r\" (UID: \"1181a098-41e1-4d6a-8582-bda860f8bd99\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" Oct 30 22:08:31.643688 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.643662 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1181a098-41e1-4d6a-8582-bda860f8bd99-workload-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r\" (UID: \"1181a098-41e1-4d6a-8582-bda860f8bd99\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" Oct 30 22:08:31.643902 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.643779 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1181a098-41e1-4d6a-8582-bda860f8bd99-credential-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r\" (UID: \"1181a098-41e1-4d6a-8582-bda860f8bd99\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" Oct 30 22:08:31.643902 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.643799 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1181a098-41e1-4d6a-8582-bda860f8bd99-workload-certs\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r\" (UID: \"1181a098-41e1-4d6a-8582-bda860f8bd99\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" Oct 30 22:08:31.644022 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.644000 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1181a098-41e1-4d6a-8582-bda860f8bd99-workload-socket\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r\" (UID: \"1181a098-41e1-4d6a-8582-bda860f8bd99\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" Oct 30 22:08:31.644083 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.644040 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1181a098-41e1-4d6a-8582-bda860f8bd99-istiod-ca-cert\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r\" (UID: \"1181a098-41e1-4d6a-8582-bda860f8bd99\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" Oct 30 22:08:31.644124 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.644062 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1181a098-41e1-4d6a-8582-bda860f8bd99-workload-certs\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r\" (UID: \"1181a098-41e1-4d6a-8582-bda860f8bd99\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" Oct 30 22:08:31.645820 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.645797 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1181a098-41e1-4d6a-8582-bda860f8bd99-istio-envoy\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r\" (UID: \"1181a098-41e1-4d6a-8582-bda860f8bd99\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" Oct 30 22:08:31.645891 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.645881 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1181a098-41e1-4d6a-8582-bda860f8bd99-istio-podinfo\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r\" (UID: \"1181a098-41e1-4d6a-8582-bda860f8bd99\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" Oct 30 22:08:31.674921 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.674895 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" Oct 30 22:08:31.707284 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.707258 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1181a098-41e1-4d6a-8582-bda860f8bd99-istio-token\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r\" (UID: \"1181a098-41e1-4d6a-8582-bda860f8bd99\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" Oct 30 22:08:31.707464 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.707427 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq8qd\" (UniqueName: \"kubernetes.io/projected/1181a098-41e1-4d6a-8582-bda860f8bd99-kube-api-access-vq8qd\") pod \"e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r\" (UID: \"1181a098-41e1-4d6a-8582-bda860f8bd99\") " pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" Oct 30 22:08:31.722393 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:31.722367 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" Oct 30 22:08:32.109109 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:32.109076 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8"] Oct 30 22:08:32.146139 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:08:32.146111 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d61cd5_fed8_44a1_9b3a_2e7b942f0947.slice/crio-ee409938e9596464abc253ff337aad034db402cd366ffce835fbef0c93d41bed WatchSource:0}: Error finding container ee409938e9596464abc253ff337aad034db402cd366ffce835fbef0c93d41bed: Status 404 returned error can't find the container with id ee409938e9596464abc253ff337aad034db402cd366ffce835fbef0c93d41bed Oct 30 22:08:32.148591 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:32.148557 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:08:32.148652 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:32.148634 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:08:32.148686 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:32.148667 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:08:32.164418 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:32.164399 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r"] Oct 30 22:08:32.164819 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:08:32.164792 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1181a098_41e1_4d6a_8582_bda860f8bd99.slice/crio-2870d8d975b270efdb62cb0c626adc9a2b03fd7ee3180b77406a5b04dce7245c WatchSource:0}: Error finding container 2870d8d975b270efdb62cb0c626adc9a2b03fd7ee3180b77406a5b04dce7245c: Status 404 returned error can't find the container with id 2870d8d975b270efdb62cb0c626adc9a2b03fd7ee3180b77406a5b04dce7245c Oct 30 22:08:32.166984 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:32.166953 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:08:32.167088 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:32.167024 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:08:32.167088 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:32.167069 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:08:32.853923 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:32.853886 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" event={"ID":"1181a098-41e1-4d6a-8582-bda860f8bd99","Type":"ContainerStarted","Data":"f4347d240efac04d580af138555dd862c6e7309d2e629df6e8b8151255541969"} Oct 30 22:08:32.853923 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:32.853939 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" event={"ID":"1181a098-41e1-4d6a-8582-bda860f8bd99","Type":"ContainerStarted","Data":"2870d8d975b270efdb62cb0c626adc9a2b03fd7ee3180b77406a5b04dce7245c"} Oct 30 22:08:32.855775 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:32.855750 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" event={"ID":"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947","Type":"ContainerStarted","Data":"887d38aef35e0bd61710f72347e4b052ad5345ea4414894c7a7ded41e1dae6ea"} Oct 30 22:08:32.855882 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:32.855793 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" event={"ID":"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947","Type":"ContainerStarted","Data":"ee409938e9596464abc253ff337aad034db402cd366ffce835fbef0c93d41bed"} Oct 30 22:08:32.978788 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:32.978726 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" podStartSLOduration=1.978708912 podStartE2EDuration="1.978708912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:32.976717827 +0000 UTC m=+1073.247415924" watchObservedRunningTime="2026-10-30 22:08:32.978708912 +0000 UTC m=+1073.249406997" Oct 30 22:08:33.102255 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:33.102167 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" podStartSLOduration=2.102145442 podStartE2EDuration="2.102145442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:33.10129093 +0000 UTC m=+1073.371989018" watchObservedRunningTime="2026-10-30 22:08:33.102145442 +0000 UTC m=+1073.372843521" Oct 30 22:08:33.675286 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:33.675250 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" Oct 30 22:08:33.676666 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:33.676641 5918 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 22:08:33.676765 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:33.676702 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" podUID="d5d61cd5-fed8-44a1-9b3a-2e7b942f0947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 22:08:33.726241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:33.725399 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" Oct 30 22:08:33.734278 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:33.730451 5918 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 22:08:33.734278 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:33.730515 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" podUID="1181a098-41e1-4d6a-8582-bda860f8bd99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 22:08:34.675693 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:34.675663 5918 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 22:08:34.676098 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:34.675711 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" podUID="d5d61cd5-fed8-44a1-9b3a-2e7b942f0947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 22:08:34.728240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:34.727582 5918 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 22:08:34.728240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:34.727639 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" podUID="1181a098-41e1-4d6a-8582-bda860f8bd99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 22:08:35.675653 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:35.675613 5918 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 22:08:35.675834 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:35.675677 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" podUID="d5d61cd5-fed8-44a1-9b3a-2e7b942f0947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 22:08:35.730241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:35.729598 5918 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 22:08:35.730241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:35.729662 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" podUID="1181a098-41e1-4d6a-8582-bda860f8bd99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 22:08:36.675713 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:36.675680 5918 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 22:08:36.675901 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:36.675732 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" podUID="d5d61cd5-fed8-44a1-9b3a-2e7b942f0947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 22:08:36.729582 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:36.729544 5918 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 22:08:36.729862 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:36.729834 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" podUID="1181a098-41e1-4d6a-8582-bda860f8bd99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 22:08:37.675997 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:37.675961 5918 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 22:08:37.676423 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:37.676013 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" podUID="d5d61cd5-fed8-44a1-9b3a-2e7b942f0947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 22:08:37.727721 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:37.727572 5918 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 22:08:37.727721 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:37.727634 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" podUID="1181a098-41e1-4d6a-8582-bda860f8bd99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 22:08:38.675566 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:38.675530 5918 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 22:08:38.675738 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:38.675583 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" podUID="d5d61cd5-fed8-44a1-9b3a-2e7b942f0947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 22:08:38.722681 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:38.722644 5918 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 22:08:38.723196 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:38.722700 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" podUID="1181a098-41e1-4d6a-8582-bda860f8bd99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 22:08:40.032306 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:40.032269 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" Oct 30 22:08:40.032825 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:40.032800 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" Oct 30 22:08:40.033980 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:40.033962 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" Oct 30 22:08:40.054413 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:40.054386 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" Oct 30 22:08:40.055374 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:40.055346 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" Oct 30 22:08:40.056312 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:40.056284 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" Oct 30 22:08:51.276633 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:08:51.276552 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:08:56.120055 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:56.120017 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87"] Oct 30 22:08:56.124323 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:56.124298 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" Oct 30 22:08:56.185750 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:56.185715 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87"] Oct 30 22:08:56.227082 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:56.227050 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab957b47-90d6-497a-8334-1d5d4541dc7a-model-cache\") pod \"distinct-2-w2-d8837a-kserve-597c4754df-jxg87\" (UID: \"ab957b47-90d6-497a-8334-1d5d4541dc7a\") " pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" Oct 30 22:08:56.227082 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:56.227080 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab957b47-90d6-497a-8334-1d5d4541dc7a-home\") pod \"distinct-2-w2-d8837a-kserve-597c4754df-jxg87\" (UID: \"ab957b47-90d6-497a-8334-1d5d4541dc7a\") " pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" Oct 30 22:08:56.227314 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:56.227248 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab957b47-90d6-497a-8334-1d5d4541dc7a-dshm\") pod \"distinct-2-w2-d8837a-kserve-597c4754df-jxg87\" (UID: \"ab957b47-90d6-497a-8334-1d5d4541dc7a\") " pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" Oct 30 22:08:56.227363 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:56.227345 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt8m6\" (UniqueName: \"kubernetes.io/projected/ab957b47-90d6-497a-8334-1d5d4541dc7a-kube-api-access-lt8m6\") pod \"distinct-2-w2-d8837a-kserve-597c4754df-jxg87\" (UID: \"ab957b47-90d6-497a-8334-1d5d4541dc7a\") " pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" Oct 30 22:08:56.227412 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:56.227400 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab957b47-90d6-497a-8334-1d5d4541dc7a-tls-certs\") pod \"distinct-2-w2-d8837a-kserve-597c4754df-jxg87\" (UID: \"ab957b47-90d6-497a-8334-1d5d4541dc7a\") " pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" Oct 30 22:08:56.227475 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:56.227461 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab957b47-90d6-497a-8334-1d5d4541dc7a-tmp-dir\") pod \"distinct-2-w2-d8837a-kserve-597c4754df-jxg87\" (UID: \"ab957b47-90d6-497a-8334-1d5d4541dc7a\") " pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" Oct 30 22:08:56.232993 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:56.232976 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d8837a\"/\"distinct-2-w2-d8837a-kserve-self-signed-certs\"" Oct 30 22:08:56.329010 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:56.328967 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab957b47-90d6-497a-8334-1d5d4541dc7a-tls-certs\") pod \"distinct-2-w2-d8837a-kserve-597c4754df-jxg87\" (UID: \"ab957b47-90d6-497a-8334-1d5d4541dc7a\") " pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" Oct 30 22:08:56.329252 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:56.329066 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab957b47-90d6-497a-8334-1d5d4541dc7a-tmp-dir\") pod \"distinct-2-w2-d8837a-kserve-597c4754df-jxg87\" (UID: \"ab957b47-90d6-497a-8334-1d5d4541dc7a\") " pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" Oct 30 22:08:56.329252 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:56.329181 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab957b47-90d6-497a-8334-1d5d4541dc7a-model-cache\") pod \"distinct-2-w2-d8837a-kserve-597c4754df-jxg87\" (UID: \"ab957b47-90d6-497a-8334-1d5d4541dc7a\") " pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" Oct 30 22:08:56.329252 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:56.329203 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab957b47-90d6-497a-8334-1d5d4541dc7a-home\") pod \"distinct-2-w2-d8837a-kserve-597c4754df-jxg87\" (UID: \"ab957b47-90d6-497a-8334-1d5d4541dc7a\") " pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" Oct 30 22:08:56.329418 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:56.329344 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab957b47-90d6-497a-8334-1d5d4541dc7a-dshm\") pod \"distinct-2-w2-d8837a-kserve-597c4754df-jxg87\" (UID: \"ab957b47-90d6-497a-8334-1d5d4541dc7a\") " pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" Oct 30 22:08:56.329481 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:56.329423 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt8m6\" (UniqueName: \"kubernetes.io/projected/ab957b47-90d6-497a-8334-1d5d4541dc7a-kube-api-access-lt8m6\") pod \"distinct-2-w2-d8837a-kserve-597c4754df-jxg87\" (UID: \"ab957b47-90d6-497a-8334-1d5d4541dc7a\") " pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" Oct 30 22:08:56.330370 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:56.330343 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab957b47-90d6-497a-8334-1d5d4541dc7a-model-cache\") pod \"distinct-2-w2-d8837a-kserve-597c4754df-jxg87\" (UID: \"ab957b47-90d6-497a-8334-1d5d4541dc7a\") " pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" Oct 30 22:08:56.330546 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:56.330350 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab957b47-90d6-497a-8334-1d5d4541dc7a-tmp-dir\") pod \"distinct-2-w2-d8837a-kserve-597c4754df-jxg87\" (UID: \"ab957b47-90d6-497a-8334-1d5d4541dc7a\") " pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" Oct 30 22:08:56.330692 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:56.330387 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab957b47-90d6-497a-8334-1d5d4541dc7a-home\") pod \"distinct-2-w2-d8837a-kserve-597c4754df-jxg87\" (UID: \"ab957b47-90d6-497a-8334-1d5d4541dc7a\") " pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" Oct 30 22:08:56.332496 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:56.332474 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab957b47-90d6-497a-8334-1d5d4541dc7a-dshm\") pod \"distinct-2-w2-d8837a-kserve-597c4754df-jxg87\" (UID: \"ab957b47-90d6-497a-8334-1d5d4541dc7a\") " pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" Oct 30 22:08:56.332622 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:56.332601 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab957b47-90d6-497a-8334-1d5d4541dc7a-tls-certs\") pod \"distinct-2-w2-d8837a-kserve-597c4754df-jxg87\" (UID: \"ab957b47-90d6-497a-8334-1d5d4541dc7a\") " pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" Oct 30 22:08:56.395990 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:56.395919 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt8m6\" (UniqueName: \"kubernetes.io/projected/ab957b47-90d6-497a-8334-1d5d4541dc7a-kube-api-access-lt8m6\") pod \"distinct-2-w2-d8837a-kserve-597c4754df-jxg87\" (UID: \"ab957b47-90d6-497a-8334-1d5d4541dc7a\") " pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" Oct 30 22:08:56.437280 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:56.437246 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" Oct 30 22:08:56.821564 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:56.821536 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87"] Oct 30 22:08:56.822686 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:08:56.822662 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab957b47_90d6_497a_8334_1d5d4541dc7a.slice/crio-5fa669a7da27ae5fc98587b81a457e179e098ec5e968a828c1f4f49aced53d67 WatchSource:0}: Error finding container 5fa669a7da27ae5fc98587b81a457e179e098ec5e968a828c1f4f49aced53d67: Status 404 returned error can't find the container with id 5fa669a7da27ae5fc98587b81a457e179e098ec5e968a828c1f4f49aced53d67 Oct 30 22:08:57.009340 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:57.009302 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" event={"ID":"ab957b47-90d6-497a-8334-1d5d4541dc7a","Type":"ContainerStarted","Data":"b5c70fc58409e56bd674ac90b399e2730cb92cf95a979154f6bb834c5aa78d26"} Oct 30 22:08:57.009340 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:57.009344 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" event={"ID":"ab957b47-90d6-497a-8334-1d5d4541dc7a","Type":"ContainerStarted","Data":"5fa669a7da27ae5fc98587b81a457e179e098ec5e968a828c1f4f49aced53d67"} Oct 30 22:08:57.133602 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:57.133510 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" podStartSLOduration=1.133494547 podStartE2EDuration="1.133494547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:57.130959012 +0000 UTC m=+1097.401657097" watchObservedRunningTime="2026-10-30 22:08:57.133494547 +0000 UTC m=+1097.404192630" Oct 30 22:08:57.437622 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:08:57.437596 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" Oct 30 22:09:06.438242 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:06.438180 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" Oct 30 22:09:06.451077 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:06.451049 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" Oct 30 22:09:07.103659 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:07.103628 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" Oct 30 22:09:17.346053 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.346020 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk"] Oct 30 22:09:17.351589 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.351568 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" Oct 30 22:09:17.403488 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.403455 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66"] Oct 30 22:09:17.407460 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.407441 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" Oct 30 22:09:17.432888 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.432853 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-istiod-ca-cert\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk\" (UID: \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" Oct 30 22:09:17.433045 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.432936 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d68726e2-79a3-4c7d-9c64-c06d73bb3077-workload-certs\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66\" (UID: \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" Oct 30 22:09:17.433045 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.432965 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-istio-token\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk\" (UID: \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" Oct 30 22:09:17.433045 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.433010 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68726e2-79a3-4c7d-9c64-c06d73bb3077-credential-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66\" (UID: \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" Oct 30 22:09:17.433207 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.433053 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-istio-data\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk\" (UID: \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" Oct 30 22:09:17.433207 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.433180 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d68726e2-79a3-4c7d-9c64-c06d73bb3077-istio-podinfo\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66\" (UID: \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" Oct 30 22:09:17.433321 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.433282 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlbqn\" (UniqueName: \"kubernetes.io/projected/d68726e2-79a3-4c7d-9c64-c06d73bb3077-kube-api-access-wlbqn\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66\" (UID: \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" Oct 30 22:09:17.433367 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.433324 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-credential-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk\" (UID: \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" Oct 30 22:09:17.433367 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.433342 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-workload-certs\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk\" (UID: \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" Oct 30 22:09:17.433770 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.433747 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d68726e2-79a3-4c7d-9c64-c06d73bb3077-istio-envoy\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66\" (UID: \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" Oct 30 22:09:17.433829 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.433804 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d68726e2-79a3-4c7d-9c64-c06d73bb3077-istiod-ca-cert\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66\" (UID: \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" Oct 30 22:09:17.433913 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.433898 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5bmk\" (UniqueName: \"kubernetes.io/projected/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-kube-api-access-f5bmk\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk\" (UID: \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" Oct 30 22:09:17.433979 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.433938 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-istio-envoy\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk\" (UID: \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" Oct 30 22:09:17.434135 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.434018 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d68726e2-79a3-4c7d-9c64-c06d73bb3077-istio-data\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66\" (UID: \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" Oct 30 22:09:17.434197 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.434145 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d68726e2-79a3-4c7d-9c64-c06d73bb3077-istio-token\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66\" (UID: \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" Oct 30 22:09:17.434271 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.434208 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-workload-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk\" (UID: \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" Oct 30 22:09:17.434329 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.434298 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-istio-podinfo\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk\" (UID: \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" Oct 30 22:09:17.434391 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.434357 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68726e2-79a3-4c7d-9c64-c06d73bb3077-workload-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66\" (UID: \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" Oct 30 22:09:17.454289 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.454268 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk"] Oct 30 22:09:17.506727 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.501549 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66"] Oct 30 22:09:17.535353 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.535322 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-credential-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk\" (UID: \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" Oct 30 22:09:17.535538 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.535358 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-workload-certs\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk\" (UID: \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" Oct 30 22:09:17.535538 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.535384 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d68726e2-79a3-4c7d-9c64-c06d73bb3077-istio-envoy\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66\" (UID: \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" Oct 30 22:09:17.535538 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.535435 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d68726e2-79a3-4c7d-9c64-c06d73bb3077-istiod-ca-cert\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66\" (UID: \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" Oct 30 22:09:17.535538 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.535476 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5bmk\" (UniqueName: \"kubernetes.io/projected/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-kube-api-access-f5bmk\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk\" (UID: \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" Oct 30 22:09:17.535538 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.535508 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-istio-envoy\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk\" (UID: \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" Oct 30 22:09:17.535817 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.535543 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d68726e2-79a3-4c7d-9c64-c06d73bb3077-istio-data\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66\" (UID: \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" Oct 30 22:09:17.535817 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.535634 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d68726e2-79a3-4c7d-9c64-c06d73bb3077-istio-token\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66\" (UID: \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" Oct 30 22:09:17.535817 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.535693 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-workload-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk\" (UID: \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" Oct 30 22:09:17.535817 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.535722 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-istio-podinfo\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk\" (UID: \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" Oct 30 22:09:17.535817 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.535775 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68726e2-79a3-4c7d-9c64-c06d73bb3077-workload-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66\" (UID: \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" Oct 30 22:09:17.535817 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.535779 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-credential-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk\" (UID: \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" Oct 30 22:09:17.536159 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.535825 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-workload-certs\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk\" (UID: \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" Oct 30 22:09:17.536159 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.535853 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-istiod-ca-cert\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk\" (UID: \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" Oct 30 22:09:17.536159 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.535968 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d68726e2-79a3-4c7d-9c64-c06d73bb3077-workload-certs\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66\" (UID: \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" Oct 30 22:09:17.536159 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.536002 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-istio-token\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk\" (UID: \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" Oct 30 22:09:17.536159 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.536063 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68726e2-79a3-4c7d-9c64-c06d73bb3077-credential-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66\" (UID: \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" Oct 30 22:09:17.536159 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.536105 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-istio-data\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk\" (UID: \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" Oct 30 22:09:17.536159 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.536131 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68726e2-79a3-4c7d-9c64-c06d73bb3077-workload-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66\" (UID: \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" Oct 30 22:09:17.536159 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.536140 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d68726e2-79a3-4c7d-9c64-c06d73bb3077-istio-podinfo\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66\" (UID: \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" Oct 30 22:09:17.536159 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.536140 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-workload-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk\" (UID: \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" Oct 30 22:09:17.536159 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.536163 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlbqn\" (UniqueName: \"kubernetes.io/projected/d68726e2-79a3-4c7d-9c64-c06d73bb3077-kube-api-access-wlbqn\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66\" (UID: \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" Oct 30 22:09:17.536779 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.536401 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-istio-data\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk\" (UID: \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" Oct 30 22:09:17.536779 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.536604 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d68726e2-79a3-4c7d-9c64-c06d73bb3077-istio-data\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66\" (UID: \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" Oct 30 22:09:17.536779 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.536655 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68726e2-79a3-4c7d-9c64-c06d73bb3077-credential-socket\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66\" (UID: \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" Oct 30 22:09:17.536986 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.536966 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-istiod-ca-cert\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk\" (UID: \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" Oct 30 22:09:17.537162 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.537128 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d68726e2-79a3-4c7d-9c64-c06d73bb3077-workload-certs\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66\" (UID: \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" Oct 30 22:09:17.537606 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.537544 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d68726e2-79a3-4c7d-9c64-c06d73bb3077-istiod-ca-cert\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66\" (UID: \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" Oct 30 22:09:17.539306 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.539261 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-istio-envoy\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk\" (UID: \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" Oct 30 22:09:17.539415 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.539374 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d68726e2-79a3-4c7d-9c64-c06d73bb3077-istio-podinfo\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66\" (UID: \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" Oct 30 22:09:17.539853 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.539811 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d68726e2-79a3-4c7d-9c64-c06d73bb3077-istio-envoy\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66\" (UID: \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" Oct 30 22:09:17.542143 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.542121 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-istio-podinfo\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk\" (UID: \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" Oct 30 22:09:17.600068 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.600002 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d68726e2-79a3-4c7d-9c64-c06d73bb3077-istio-token\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66\" (UID: \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" Oct 30 22:09:17.600068 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.600000 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-istio-token\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk\" (UID: \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" Oct 30 22:09:17.600321 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.600190 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlbqn\" (UniqueName: \"kubernetes.io/projected/d68726e2-79a3-4c7d-9c64-c06d73bb3077-kube-api-access-wlbqn\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66\" (UID: \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" Oct 30 22:09:17.609783 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.609755 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5bmk\" (UniqueName: \"kubernetes.io/projected/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-kube-api-access-f5bmk\") pod \"e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk\" (UID: \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\") " pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" Oct 30 22:09:17.665679 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.665653 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" Oct 30 22:09:17.728613 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:17.727645 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" Oct 30 22:09:18.103487 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.103453 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk"] Oct 30 22:09:18.137072 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:09:18.137044 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2801c3c_999f_41aa_900e_5a8b1c9fd8c0.slice/crio-f55ceab8e728c734fdbc6f84dc64853ecccf893d2702d2d8b8f1fb2a375da963 WatchSource:0}: Error finding container f55ceab8e728c734fdbc6f84dc64853ecccf893d2702d2d8b8f1fb2a375da963: Status 404 returned error can't find the container with id f55ceab8e728c734fdbc6f84dc64853ecccf893d2702d2d8b8f1fb2a375da963 Oct 30 22:09:18.140398 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.140356 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:09:18.140524 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.140447 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:09:18.140524 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.140492 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:09:18.154504 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.154483 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66"] Oct 30 22:09:18.155356 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:09:18.155334 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd68726e2_79a3_4c7d_9c64_c06d73bb3077.slice/crio-d21273178842a99f9e85808d255b5aaf67abc44d805114006057e5bbca9bbe02 WatchSource:0}: Error finding container d21273178842a99f9e85808d255b5aaf67abc44d805114006057e5bbca9bbe02: Status 404 returned error can't find the container with id d21273178842a99f9e85808d255b5aaf67abc44d805114006057e5bbca9bbe02 Oct 30 22:09:18.157802 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.157772 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:09:18.157851 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.157833 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:09:18.157886 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.157866 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:09:18.169528 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.169497 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" event={"ID":"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0","Type":"ContainerStarted","Data":"f55ceab8e728c734fdbc6f84dc64853ecccf893d2702d2d8b8f1fb2a375da963"} Oct 30 22:09:18.171258 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.171230 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" event={"ID":"d68726e2-79a3-4c7d-9c64-c06d73bb3077","Type":"ContainerStarted","Data":"d21273178842a99f9e85808d255b5aaf67abc44d805114006057e5bbca9bbe02"} Oct 30 22:09:18.328314 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.326305 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728"] Oct 30 22:09:18.330547 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.330528 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" Oct 30 22:09:18.349368 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.349341 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b3bd36-51f3-4b46-bed6-94703557bfdf-istio-envoy\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728\" (UID: \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" Oct 30 22:09:18.349762 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.349375 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b3bd36-51f3-4b46-bed6-94703557bfdf-credential-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728\" (UID: \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" Oct 30 22:09:18.349762 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.349494 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b3bd36-51f3-4b46-bed6-94703557bfdf-istio-podinfo\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728\" (UID: \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" Oct 30 22:09:18.349762 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.349530 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b3bd36-51f3-4b46-bed6-94703557bfdf-istiod-ca-cert\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728\" (UID: \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" Oct 30 22:09:18.349762 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.349618 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbprg\" (UniqueName: \"kubernetes.io/projected/f9b3bd36-51f3-4b46-bed6-94703557bfdf-kube-api-access-hbprg\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728\" (UID: \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" Oct 30 22:09:18.349762 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.349709 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b3bd36-51f3-4b46-bed6-94703557bfdf-istio-data\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728\" (UID: \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" Oct 30 22:09:18.349762 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.349747 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b3bd36-51f3-4b46-bed6-94703557bfdf-workload-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728\" (UID: \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" Oct 30 22:09:18.350059 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.349836 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b3bd36-51f3-4b46-bed6-94703557bfdf-istio-token\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728\" (UID: \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" Oct 30 22:09:18.350059 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.349856 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b3bd36-51f3-4b46-bed6-94703557bfdf-workload-certs\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728\" (UID: \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" Oct 30 22:09:18.397283 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.397172 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728"] Oct 30 22:09:18.461240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.451950 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b3bd36-51f3-4b46-bed6-94703557bfdf-istio-podinfo\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728\" (UID: \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" Oct 30 22:09:18.461240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.451990 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b3bd36-51f3-4b46-bed6-94703557bfdf-istiod-ca-cert\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728\" (UID: \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" Oct 30 22:09:18.461240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.452045 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbprg\" (UniqueName: \"kubernetes.io/projected/f9b3bd36-51f3-4b46-bed6-94703557bfdf-kube-api-access-hbprg\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728\" (UID: \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" Oct 30 22:09:18.461240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.452128 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b3bd36-51f3-4b46-bed6-94703557bfdf-istio-data\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728\" (UID: \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" Oct 30 22:09:18.461240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.452180 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b3bd36-51f3-4b46-bed6-94703557bfdf-workload-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728\" (UID: \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" Oct 30 22:09:18.461240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.453390 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b3bd36-51f3-4b46-bed6-94703557bfdf-istio-token\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728\" (UID: \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" Oct 30 22:09:18.461240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.453459 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b3bd36-51f3-4b46-bed6-94703557bfdf-workload-certs\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728\" (UID: \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" Oct 30 22:09:18.461240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.453668 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b3bd36-51f3-4b46-bed6-94703557bfdf-istio-envoy\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728\" (UID: \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" Oct 30 22:09:18.461240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.453695 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b3bd36-51f3-4b46-bed6-94703557bfdf-credential-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728\" (UID: \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" Oct 30 22:09:18.461240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.456539 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b3bd36-51f3-4b46-bed6-94703557bfdf-istio-data\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728\" (UID: \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" Oct 30 22:09:18.461240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.456793 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b3bd36-51f3-4b46-bed6-94703557bfdf-workload-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728\" (UID: \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" Oct 30 22:09:18.461240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.457003 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b3bd36-51f3-4b46-bed6-94703557bfdf-istio-podinfo\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728\" (UID: \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" Oct 30 22:09:18.461240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.459258 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b3bd36-51f3-4b46-bed6-94703557bfdf-istiod-ca-cert\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728\" (UID: \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" Oct 30 22:09:18.461240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.459543 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b3bd36-51f3-4b46-bed6-94703557bfdf-credential-socket\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728\" (UID: \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" Oct 30 22:09:18.472808 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.472515 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b3bd36-51f3-4b46-bed6-94703557bfdf-workload-certs\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728\" (UID: \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" Oct 30 22:09:18.475543 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.475505 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b3bd36-51f3-4b46-bed6-94703557bfdf-istio-envoy\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728\" (UID: \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" Oct 30 22:09:18.519156 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.519121 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b3bd36-51f3-4b46-bed6-94703557bfdf-istio-token\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728\" (UID: \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" Oct 30 22:09:18.519363 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.519338 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbprg\" (UniqueName: \"kubernetes.io/projected/f9b3bd36-51f3-4b46-bed6-94703557bfdf-kube-api-access-hbprg\") pod \"e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728\" (UID: \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\") " pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" Oct 30 22:09:18.645548 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:18.645507 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" Oct 30 22:09:19.038117 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:19.038077 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728"] Oct 30 22:09:19.038377 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:09:19.038351 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b3bd36_51f3_4b46_bed6_94703557bfdf.slice/crio-7c70994f716be35bf28cb7b0a3c927255a41351340dfc0a5c19ac6c5f9272f82 WatchSource:0}: Error finding container 7c70994f716be35bf28cb7b0a3c927255a41351340dfc0a5c19ac6c5f9272f82: Status 404 returned error can't find the container with id 7c70994f716be35bf28cb7b0a3c927255a41351340dfc0a5c19ac6c5f9272f82 Oct 30 22:09:19.041988 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:19.041949 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:09:19.042096 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:19.042037 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:09:19.042096 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:19.042079 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:09:19.181281 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:19.180272 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" event={"ID":"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0","Type":"ContainerStarted","Data":"396d223632dcdf588956ff14da792b0bbf4375eb40cec10beaa8b9be86b0dec1"} Oct 30 22:09:19.183570 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:19.183281 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" event={"ID":"f9b3bd36-51f3-4b46-bed6-94703557bfdf","Type":"ContainerStarted","Data":"cb8aa5375f6f2c8f37b4bb60f755723baefbb297d9d5c7d55537a276b57ae445"} Oct 30 22:09:19.183570 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:19.183333 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" event={"ID":"f9b3bd36-51f3-4b46-bed6-94703557bfdf","Type":"ContainerStarted","Data":"7c70994f716be35bf28cb7b0a3c927255a41351340dfc0a5c19ac6c5f9272f82"} Oct 30 22:09:19.186966 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:19.186387 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" event={"ID":"d68726e2-79a3-4c7d-9c64-c06d73bb3077","Type":"ContainerStarted","Data":"1e9896693033ee317c063c84f2a59a8f776bd55f8161842fb7df2d92378c707c"} Oct 30 22:09:19.320546 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:19.320421 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" podStartSLOduration=2.320398443 podStartE2EDuration="2.320398443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:19.308720798 +0000 UTC m=+1119.579418883" watchObservedRunningTime="2026-10-30 22:09:19.320398443 +0000 UTC m=+1119.591096524" Oct 30 22:09:19.576260 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:19.576114 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" podStartSLOduration=1.576094176 podStartE2EDuration="1.576094176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:19.442395904 +0000 UTC m=+1119.713093988" watchObservedRunningTime="2026-10-30 22:09:19.576094176 +0000 UTC m=+1119.846792262" Oct 30 22:09:19.577299 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:19.577247 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" podStartSLOduration=2.577204949 podStartE2EDuration="2.577204949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:19.571827251 +0000 UTC m=+1119.842525335" watchObservedRunningTime="2026-10-30 22:09:19.577204949 +0000 UTC m=+1119.847903034" Oct 30 22:09:19.649251 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:19.647787 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" Oct 30 22:09:19.666725 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:19.666373 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" Oct 30 22:09:19.667522 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:19.667499 5918 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 22:09:19.667610 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:19.667544 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" podUID="e2801c3c-999f-41aa-900e-5a8b1c9fd8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 22:09:19.736731 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:19.736696 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" Oct 30 22:09:19.737146 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:19.737126 5918 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 22:09:19.737345 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:19.737318 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" podUID="d68726e2-79a3-4c7d-9c64-c06d73bb3077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 22:09:20.659240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:20.658336 5918 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 22:09:20.659240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:20.658402 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" podUID="f9b3bd36-51f3-4b46-bed6-94703557bfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 22:09:20.675240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:20.670640 5918 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 22:09:20.675240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:20.670700 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" podUID="e2801c3c-999f-41aa-900e-5a8b1c9fd8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 22:09:20.745243 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:20.742669 5918 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 22:09:20.745243 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:20.742732 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" podUID="d68726e2-79a3-4c7d-9c64-c06d73bb3077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 22:09:21.369475 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:09:21.369430 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:09:21.649726 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:21.649630 5918 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 22:09:21.649993 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:21.649960 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" podUID="f9b3bd36-51f3-4b46-bed6-94703557bfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 22:09:21.666239 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:21.666053 5918 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 22:09:21.666239 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:21.666118 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" podUID="e2801c3c-999f-41aa-900e-5a8b1c9fd8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 22:09:21.729639 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:21.729601 5918 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 22:09:21.729954 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:21.729915 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" podUID="d68726e2-79a3-4c7d-9c64-c06d73bb3077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 22:09:22.647240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:22.646772 5918 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 22:09:22.647240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:22.646859 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" podUID="f9b3bd36-51f3-4b46-bed6-94703557bfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 22:09:22.667239 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:22.666797 5918 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 22:09:22.667239 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:22.666864 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" podUID="e2801c3c-999f-41aa-900e-5a8b1c9fd8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 22:09:22.745239 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:22.743345 5918 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 22:09:22.745239 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:22.743405 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" podUID="d68726e2-79a3-4c7d-9c64-c06d73bb3077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 22:09:23.646376 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:23.646338 5918 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 22:09:23.646574 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:23.646417 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" podUID="f9b3bd36-51f3-4b46-bed6-94703557bfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 22:09:23.670798 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:23.670763 5918 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 22:09:23.671445 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:23.671391 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" podUID="e2801c3c-999f-41aa-900e-5a8b1c9fd8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 22:09:23.744637 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:23.744598 5918 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 22:09:23.745891 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:23.745860 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" podUID="d68726e2-79a3-4c7d-9c64-c06d73bb3077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 22:09:24.653242 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:24.646601 5918 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 22:09:24.653242 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:24.646663 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" podUID="f9b3bd36-51f3-4b46-bed6-94703557bfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 22:09:24.666533 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:24.666293 5918 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 22:09:24.666533 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:24.666374 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" podUID="e2801c3c-999f-41aa-900e-5a8b1c9fd8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 22:09:24.732241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:24.731707 5918 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 22:09:24.732241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:24.731783 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" podUID="d68726e2-79a3-4c7d-9c64-c06d73bb3077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 22:09:25.646040 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:25.646001 5918 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 22:09:25.646265 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:25.646061 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" podUID="f9b3bd36-51f3-4b46-bed6-94703557bfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 22:09:25.666476 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:25.666445 5918 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 22:09:25.666618 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:25.666497 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" podUID="e2801c3c-999f-41aa-900e-5a8b1c9fd8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 22:09:25.728965 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:25.728802 5918 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 22:09:25.728965 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:25.728868 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" podUID="d68726e2-79a3-4c7d-9c64-c06d73bb3077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 22:09:26.646137 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:26.646108 5918 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 22:09:26.646571 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:26.646156 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" podUID="f9b3bd36-51f3-4b46-bed6-94703557bfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 22:09:26.671250 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:26.671179 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" Oct 30 22:09:26.671654 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:26.671634 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" Oct 30 22:09:26.672443 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:26.672418 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" Oct 30 22:09:26.733723 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:26.733692 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" Oct 30 22:09:26.734009 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:26.733988 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" Oct 30 22:09:26.734886 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:26.734863 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" Oct 30 22:09:27.649995 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:27.649969 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" Oct 30 22:09:27.650394 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:27.650244 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" Oct 30 22:09:27.650981 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:27.650962 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" Oct 30 22:09:31.042680 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:31.042321 5918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" podUID="dc20dd97-eb75-42c5-8e50-6b0b5cb54b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:31.059407 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:31.059300 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" podUID="dc20dd97-eb75-42c5-8e50-6b0b5cb54b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:31.085414 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:31.084626 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" podUID="e63fad41-0e42-41ee-ae20-919f15fffe55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 22:09:31.204389 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:31.204308 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" podUID="05ab563b-c4c2-4fa7-85e4-423141587664" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:31.260404 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:31.260346 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" podUID="f5ae2434-f161-4a5c-8086-2f0711678bb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:31.715241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:31.712302 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" podUID="395b4d7d-4aeb-4576-b616-67638bb1a2c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:09:47.315352 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:47.315324 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4"] Oct 30 22:09:47.320798 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:47.320774 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" Oct 30 22:09:47.422165 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:47.422129 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/438b8728-4021-49e9-8c37-fbdd405008f1-istio-data\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4\" (UID: \"438b8728-4021-49e9-8c37-fbdd405008f1\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" Oct 30 22:09:47.422165 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:47.422168 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/438b8728-4021-49e9-8c37-fbdd405008f1-istio-podinfo\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4\" (UID: \"438b8728-4021-49e9-8c37-fbdd405008f1\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" Oct 30 22:09:47.422480 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:47.422448 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/438b8728-4021-49e9-8c37-fbdd405008f1-workload-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4\" (UID: \"438b8728-4021-49e9-8c37-fbdd405008f1\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" Oct 30 22:09:47.422612 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:47.422500 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/438b8728-4021-49e9-8c37-fbdd405008f1-istio-token\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4\" (UID: \"438b8728-4021-49e9-8c37-fbdd405008f1\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" Oct 30 22:09:47.422612 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:47.422586 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/438b8728-4021-49e9-8c37-fbdd405008f1-istio-envoy\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4\" (UID: \"438b8728-4021-49e9-8c37-fbdd405008f1\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" Oct 30 22:09:47.422696 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:47.422626 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql9cl\" (UniqueName: \"kubernetes.io/projected/438b8728-4021-49e9-8c37-fbdd405008f1-kube-api-access-ql9cl\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4\" (UID: \"438b8728-4021-49e9-8c37-fbdd405008f1\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" Oct 30 22:09:47.422696 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:47.422665 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/438b8728-4021-49e9-8c37-fbdd405008f1-workload-certs\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4\" (UID: \"438b8728-4021-49e9-8c37-fbdd405008f1\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" Oct 30 22:09:47.422772 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:47.422741 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/438b8728-4021-49e9-8c37-fbdd405008f1-istiod-ca-cert\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4\" (UID: \"438b8728-4021-49e9-8c37-fbdd405008f1\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" Oct 30 22:09:47.422823 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:47.422806 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/438b8728-4021-49e9-8c37-fbdd405008f1-credential-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4\" (UID: \"438b8728-4021-49e9-8c37-fbdd405008f1\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" Oct 30 22:09:47.430356 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:47.430329 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4"] Oct 30 22:09:47.523766 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:47.523736 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/438b8728-4021-49e9-8c37-fbdd405008f1-workload-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4\" (UID: \"438b8728-4021-49e9-8c37-fbdd405008f1\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" Oct 30 22:09:47.523961 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:47.523774 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/438b8728-4021-49e9-8c37-fbdd405008f1-istio-token\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4\" (UID: \"438b8728-4021-49e9-8c37-fbdd405008f1\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" Oct 30 22:09:47.523961 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:47.523950 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/438b8728-4021-49e9-8c37-fbdd405008f1-istio-envoy\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4\" (UID: \"438b8728-4021-49e9-8c37-fbdd405008f1\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" Oct 30 22:09:47.524094 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:47.524000 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql9cl\" (UniqueName: \"kubernetes.io/projected/438b8728-4021-49e9-8c37-fbdd405008f1-kube-api-access-ql9cl\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4\" (UID: \"438b8728-4021-49e9-8c37-fbdd405008f1\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" Oct 30 22:09:47.524094 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:47.524059 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/438b8728-4021-49e9-8c37-fbdd405008f1-workload-certs\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4\" (UID: \"438b8728-4021-49e9-8c37-fbdd405008f1\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" Oct 30 22:09:47.524208 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:47.524093 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/438b8728-4021-49e9-8c37-fbdd405008f1-istiod-ca-cert\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4\" (UID: \"438b8728-4021-49e9-8c37-fbdd405008f1\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" Oct 30 22:09:47.524208 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:47.524150 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/438b8728-4021-49e9-8c37-fbdd405008f1-credential-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4\" (UID: \"438b8728-4021-49e9-8c37-fbdd405008f1\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" Oct 30 22:09:47.524208 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:47.524174 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/438b8728-4021-49e9-8c37-fbdd405008f1-workload-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4\" (UID: \"438b8728-4021-49e9-8c37-fbdd405008f1\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" Oct 30 22:09:47.524398 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:47.524312 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/438b8728-4021-49e9-8c37-fbdd405008f1-istio-data\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4\" (UID: \"438b8728-4021-49e9-8c37-fbdd405008f1\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" Oct 30 22:09:47.524398 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:47.524344 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/438b8728-4021-49e9-8c37-fbdd405008f1-istio-podinfo\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4\" (UID: \"438b8728-4021-49e9-8c37-fbdd405008f1\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" Oct 30 22:09:47.524502 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:47.524448 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/438b8728-4021-49e9-8c37-fbdd405008f1-workload-certs\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4\" (UID: \"438b8728-4021-49e9-8c37-fbdd405008f1\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" Oct 30 22:09:47.524554 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:47.524516 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/438b8728-4021-49e9-8c37-fbdd405008f1-credential-socket\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4\" (UID: \"438b8728-4021-49e9-8c37-fbdd405008f1\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" Oct 30 22:09:47.524792 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:47.524765 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/438b8728-4021-49e9-8c37-fbdd405008f1-istiod-ca-cert\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4\" (UID: \"438b8728-4021-49e9-8c37-fbdd405008f1\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" Oct 30 22:09:47.524887 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:47.524842 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/438b8728-4021-49e9-8c37-fbdd405008f1-istio-data\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4\" (UID: \"438b8728-4021-49e9-8c37-fbdd405008f1\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" Oct 30 22:09:47.526362 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:47.526338 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/438b8728-4021-49e9-8c37-fbdd405008f1-istio-envoy\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4\" (UID: \"438b8728-4021-49e9-8c37-fbdd405008f1\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" Oct 30 22:09:47.526839 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:47.526822 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/438b8728-4021-49e9-8c37-fbdd405008f1-istio-podinfo\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4\" (UID: \"438b8728-4021-49e9-8c37-fbdd405008f1\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" Oct 30 22:09:47.588039 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:47.587975 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/438b8728-4021-49e9-8c37-fbdd405008f1-istio-token\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4\" (UID: \"438b8728-4021-49e9-8c37-fbdd405008f1\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" Oct 30 22:09:47.590189 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:47.590163 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql9cl\" (UniqueName: \"kubernetes.io/projected/438b8728-4021-49e9-8c37-fbdd405008f1-kube-api-access-ql9cl\") pod \"e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4\" (UID: \"438b8728-4021-49e9-8c37-fbdd405008f1\") " pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" Oct 30 22:09:47.635584 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:47.635556 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" Oct 30 22:09:48.127125 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.127095 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4"] Oct 30 22:09:48.161158 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:09:48.161126 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod438b8728_4021_49e9_8c37_fbdd405008f1.slice/crio-da6ce86bd8b378353beaeb6da69903e88e3ce564c2673dfd68f56b12a759a598 WatchSource:0}: Error finding container da6ce86bd8b378353beaeb6da69903e88e3ce564c2673dfd68f56b12a759a598: Status 404 returned error can't find the container with id da6ce86bd8b378353beaeb6da69903e88e3ce564c2673dfd68f56b12a759a598 Oct 30 22:09:48.163573 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.163534 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:09:48.163672 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.163605 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:09:48.163672 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.163636 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:09:48.315421 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.315388 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65"] Oct 30 22:09:48.320482 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.320416 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" Oct 30 22:09:48.389774 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.389689 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65"] Oct 30 22:09:48.441629 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.441592 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34335b46-c602-47cb-92f4-62df851dd01b-istio-envoy\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65\" (UID: \"34335b46-c602-47cb-92f4-62df851dd01b\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" Oct 30 22:09:48.441827 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.441789 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34335b46-c602-47cb-92f4-62df851dd01b-istiod-ca-cert\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65\" (UID: \"34335b46-c602-47cb-92f4-62df851dd01b\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" Oct 30 22:09:48.441900 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.441866 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34335b46-c602-47cb-92f4-62df851dd01b-workload-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65\" (UID: \"34335b46-c602-47cb-92f4-62df851dd01b\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" Oct 30 22:09:48.441960 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.441908 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34335b46-c602-47cb-92f4-62df851dd01b-credential-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65\" (UID: \"34335b46-c602-47cb-92f4-62df851dd01b\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" Oct 30 22:09:48.441960 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.441952 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34335b46-c602-47cb-92f4-62df851dd01b-istio-podinfo\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65\" (UID: \"34335b46-c602-47cb-92f4-62df851dd01b\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" Oct 30 22:09:48.442141 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.441976 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n62t\" (UniqueName: \"kubernetes.io/projected/34335b46-c602-47cb-92f4-62df851dd01b-kube-api-access-6n62t\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65\" (UID: \"34335b46-c602-47cb-92f4-62df851dd01b\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" Oct 30 22:09:48.442141 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.442016 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34335b46-c602-47cb-92f4-62df851dd01b-workload-certs\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65\" (UID: \"34335b46-c602-47cb-92f4-62df851dd01b\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" Oct 30 22:09:48.442141 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.442059 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34335b46-c602-47cb-92f4-62df851dd01b-istio-data\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65\" (UID: \"34335b46-c602-47cb-92f4-62df851dd01b\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" Oct 30 22:09:48.442141 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.442111 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34335b46-c602-47cb-92f4-62df851dd01b-istio-token\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65\" (UID: \"34335b46-c602-47cb-92f4-62df851dd01b\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" Oct 30 22:09:48.543359 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.543318 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34335b46-c602-47cb-92f4-62df851dd01b-istio-envoy\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65\" (UID: \"34335b46-c602-47cb-92f4-62df851dd01b\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" Oct 30 22:09:48.543532 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.543442 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34335b46-c602-47cb-92f4-62df851dd01b-istiod-ca-cert\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65\" (UID: \"34335b46-c602-47cb-92f4-62df851dd01b\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" Oct 30 22:09:48.543532 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.543476 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34335b46-c602-47cb-92f4-62df851dd01b-workload-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65\" (UID: \"34335b46-c602-47cb-92f4-62df851dd01b\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" Oct 30 22:09:48.543532 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.543506 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34335b46-c602-47cb-92f4-62df851dd01b-credential-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65\" (UID: \"34335b46-c602-47cb-92f4-62df851dd01b\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" Oct 30 22:09:48.543703 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.543547 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34335b46-c602-47cb-92f4-62df851dd01b-istio-podinfo\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65\" (UID: \"34335b46-c602-47cb-92f4-62df851dd01b\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" Oct 30 22:09:48.543703 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.543572 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n62t\" (UniqueName: \"kubernetes.io/projected/34335b46-c602-47cb-92f4-62df851dd01b-kube-api-access-6n62t\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65\" (UID: \"34335b46-c602-47cb-92f4-62df851dd01b\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" Oct 30 22:09:48.543703 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.543614 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34335b46-c602-47cb-92f4-62df851dd01b-workload-certs\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65\" (UID: \"34335b46-c602-47cb-92f4-62df851dd01b\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" Oct 30 22:09:48.543703 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.543663 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34335b46-c602-47cb-92f4-62df851dd01b-istio-data\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65\" (UID: \"34335b46-c602-47cb-92f4-62df851dd01b\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" Oct 30 22:09:48.543892 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.543711 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34335b46-c602-47cb-92f4-62df851dd01b-istio-token\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65\" (UID: \"34335b46-c602-47cb-92f4-62df851dd01b\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" Oct 30 22:09:48.543892 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.543865 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34335b46-c602-47cb-92f4-62df851dd01b-workload-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65\" (UID: \"34335b46-c602-47cb-92f4-62df851dd01b\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" Oct 30 22:09:48.543994 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.543939 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34335b46-c602-47cb-92f4-62df851dd01b-credential-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65\" (UID: \"34335b46-c602-47cb-92f4-62df851dd01b\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" Oct 30 22:09:48.544172 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.544152 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34335b46-c602-47cb-92f4-62df851dd01b-workload-certs\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65\" (UID: \"34335b46-c602-47cb-92f4-62df851dd01b\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" Oct 30 22:09:48.544296 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.544172 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34335b46-c602-47cb-92f4-62df851dd01b-istiod-ca-cert\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65\" (UID: \"34335b46-c602-47cb-92f4-62df851dd01b\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" Oct 30 22:09:48.544355 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.544296 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34335b46-c602-47cb-92f4-62df851dd01b-istio-data\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65\" (UID: \"34335b46-c602-47cb-92f4-62df851dd01b\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" Oct 30 22:09:48.546322 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.546298 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34335b46-c602-47cb-92f4-62df851dd01b-istio-envoy\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65\" (UID: \"34335b46-c602-47cb-92f4-62df851dd01b\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" Oct 30 22:09:48.546421 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.546396 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34335b46-c602-47cb-92f4-62df851dd01b-istio-podinfo\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65\" (UID: \"34335b46-c602-47cb-92f4-62df851dd01b\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" Oct 30 22:09:48.566634 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.566608 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" event={"ID":"438b8728-4021-49e9-8c37-fbdd405008f1","Type":"ContainerStarted","Data":"c6557f6ee7fe7d192bc4813b384e3552d9108556e2720820dd3ab0d0f961c6c8"} Oct 30 22:09:48.566743 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.566660 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" event={"ID":"438b8728-4021-49e9-8c37-fbdd405008f1","Type":"ContainerStarted","Data":"da6ce86bd8b378353beaeb6da69903e88e3ce564c2673dfd68f56b12a759a598"} Oct 30 22:09:48.607310 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.607283 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n62t\" (UniqueName: \"kubernetes.io/projected/34335b46-c602-47cb-92f4-62df851dd01b-kube-api-access-6n62t\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65\" (UID: \"34335b46-c602-47cb-92f4-62df851dd01b\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" Oct 30 22:09:48.607310 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.607302 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34335b46-c602-47cb-92f4-62df851dd01b-istio-token\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65\" (UID: \"34335b46-c602-47cb-92f4-62df851dd01b\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" Oct 30 22:09:48.636511 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.636484 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" Oct 30 22:09:48.659195 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.659118 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" Oct 30 22:09:48.714251 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:48.701884 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" podStartSLOduration=1.701863788 podStartE2EDuration="1.701863788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:48.690934519 +0000 UTC m=+1148.961632614" watchObservedRunningTime="2026-10-30 22:09:48.701863788 +0000 UTC m=+1148.972561875" Oct 30 22:09:49.124983 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:49.124938 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65"] Oct 30 22:09:49.131420 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:09:49.131390 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34335b46_c602_47cb_92f4_62df851dd01b.slice/crio-b80df22b7936f81d5f6e9064e9c51c004511054c66c7ed0e2c1bae0ccd717df9 WatchSource:0}: Error finding container b80df22b7936f81d5f6e9064e9c51c004511054c66c7ed0e2c1bae0ccd717df9: Status 404 returned error can't find the container with id b80df22b7936f81d5f6e9064e9c51c004511054c66c7ed0e2c1bae0ccd717df9 Oct 30 22:09:49.134466 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:49.134423 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:09:49.134541 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:49.134513 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:09:49.134583 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:49.134554 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:09:49.576704 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:49.576655 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" event={"ID":"34335b46-c602-47cb-92f4-62df851dd01b","Type":"ContainerStarted","Data":"ae38f8be1ccacf3468e262e8f536b94d15fa44931a083dc3b81a38dd79e58406"} Oct 30 22:09:49.577146 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:49.576772 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" event={"ID":"34335b46-c602-47cb-92f4-62df851dd01b","Type":"ContainerStarted","Data":"b80df22b7936f81d5f6e9064e9c51c004511054c66c7ed0e2c1bae0ccd717df9"} Oct 30 22:09:49.639851 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:49.639814 5918 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 22:09:49.640037 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:49.639878 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" podUID="438b8728-4021-49e9-8c37-fbdd405008f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 22:09:49.660057 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:49.660021 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" Oct 30 22:09:50.639689 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:50.639653 5918 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 22:09:50.640294 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:50.640266 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" podUID="438b8728-4021-49e9-8c37-fbdd405008f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 22:09:50.661439 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:50.661408 5918 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:09:50.661568 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:50.661480 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" podUID="34335b46-c602-47cb-92f4-62df851dd01b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 22:09:51.276002 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:09:51.275962 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:09:51.640410 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:51.640321 5918 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 22:09:51.641377 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:51.640951 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" podUID="438b8728-4021-49e9-8c37-fbdd405008f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 22:09:51.660136 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:51.660117 5918 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:09:51.660311 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:51.660293 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" podUID="34335b46-c602-47cb-92f4-62df851dd01b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 22:09:52.638706 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:52.638515 5918 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 22:09:52.638706 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:52.638570 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" podUID="438b8728-4021-49e9-8c37-fbdd405008f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 22:09:52.659767 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:52.659734 5918 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:09:52.660148 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:52.659804 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" podUID="34335b46-c602-47cb-92f4-62df851dd01b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 22:09:53.639241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:53.637895 5918 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 22:09:53.639241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:53.637954 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" podUID="438b8728-4021-49e9-8c37-fbdd405008f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 22:09:53.660474 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:53.660447 5918 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:09:53.660839 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:53.660494 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" podUID="34335b46-c602-47cb-92f4-62df851dd01b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 22:09:54.636416 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:54.636383 5918 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 22:09:54.636684 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:54.636659 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" podUID="438b8728-4021-49e9-8c37-fbdd405008f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 22:09:54.660260 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:54.660049 5918 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:09:54.660260 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:54.660111 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" podUID="34335b46-c602-47cb-92f4-62df851dd01b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 22:09:55.659602 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:55.659563 5918 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:09:55.660037 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:55.659641 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" podUID="34335b46-c602-47cb-92f4-62df851dd01b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 22:09:55.696657 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:55.694014 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" Oct 30 22:09:55.696657 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:55.695577 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" Oct 30 22:09:55.696657 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:55.696612 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" Oct 30 22:09:55.839762 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:55.839712 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" podStartSLOduration=7.839699696 podStartE2EDuration="7.839699696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:49.718644066 +0000 UTC m=+1149.989342161" watchObservedRunningTime="2026-10-30 22:09:55.839699696 +0000 UTC m=+1156.110397781" Oct 30 22:09:56.745764 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:56.745733 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" Oct 30 22:09:56.746313 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:56.746266 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" Oct 30 22:09:56.747383 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:09:56.747362 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" Oct 30 22:10:03.410179 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:03.410144 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx"] Oct 30 22:10:03.414638 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:03.414614 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" Oct 30 22:10:03.513206 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:03.513163 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-workload-certs\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx\" (UID: \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" Oct 30 22:10:03.513395 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:03.513243 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-istio-envoy\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx\" (UID: \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" Oct 30 22:10:03.513395 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:03.513370 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-credential-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx\" (UID: \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" Oct 30 22:10:03.513509 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:03.513405 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-istiod-ca-cert\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx\" (UID: \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" Oct 30 22:10:03.513509 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:03.513430 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-istio-podinfo\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx\" (UID: \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" Oct 30 22:10:03.513509 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:03.513494 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-istio-data\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx\" (UID: \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" Oct 30 22:10:03.513662 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:03.513640 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctmmv\" (UniqueName: \"kubernetes.io/projected/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-kube-api-access-ctmmv\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx\" (UID: \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" Oct 30 22:10:03.513720 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:03.513662 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-workload-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx\" (UID: \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" Oct 30 22:10:03.513833 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:03.513816 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-istio-token\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx\" (UID: \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" Oct 30 22:10:03.523700 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:03.523655 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx"] Oct 30 22:10:03.615309 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:03.615277 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctmmv\" (UniqueName: \"kubernetes.io/projected/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-kube-api-access-ctmmv\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx\" (UID: \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" Oct 30 22:10:03.615309 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:03.615311 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-workload-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx\" (UID: \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" Oct 30 22:10:03.615628 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:03.615591 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-istio-token\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx\" (UID: \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" Oct 30 22:10:03.615702 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:03.615686 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-workload-certs\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx\" (UID: \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" Oct 30 22:10:03.615743 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:03.615725 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-istio-envoy\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx\" (UID: \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" Oct 30 22:10:03.615795 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:03.615741 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-workload-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx\" (UID: \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" Oct 30 22:10:03.615795 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:03.615784 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-credential-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx\" (UID: \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" Oct 30 22:10:03.615908 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:03.615817 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-istiod-ca-cert\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx\" (UID: \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" Oct 30 22:10:03.615908 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:03.615847 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-istio-podinfo\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx\" (UID: \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" Oct 30 22:10:03.615908 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:03.615886 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-istio-data\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx\" (UID: \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" Oct 30 22:10:03.616120 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:03.616093 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-workload-certs\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx\" (UID: \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" Oct 30 22:10:03.616185 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:03.616156 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-credential-socket\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx\" (UID: \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" Oct 30 22:10:03.616324 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:03.616301 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-istio-data\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx\" (UID: \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" Oct 30 22:10:03.616508 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:03.616487 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-istiod-ca-cert\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx\" (UID: \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" Oct 30 22:10:03.618116 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:03.618093 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-istio-envoy\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx\" (UID: \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" Oct 30 22:10:03.618557 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:03.618526 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-istio-podinfo\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx\" (UID: \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" Oct 30 22:10:03.678837 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:03.678814 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctmmv\" (UniqueName: \"kubernetes.io/projected/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-kube-api-access-ctmmv\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx\" (UID: \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" Oct 30 22:10:03.680054 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:03.680032 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-istio-token\") pod \"e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx\" (UID: \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\") " pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" Oct 30 22:10:03.729814 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:03.729791 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" Oct 30 22:10:04.161605 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:04.161569 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx"] Oct 30 22:10:04.162474 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:10:04.162448 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a4486c4_7472_4dae_bd5c_2c56cd386fcb.slice/crio-8699ff23ff5e33b999dee0171b95546ba20ab189c67f61e7ace4ddd3a6fd5524 WatchSource:0}: Error finding container 8699ff23ff5e33b999dee0171b95546ba20ab189c67f61e7ace4ddd3a6fd5524: Status 404 returned error can't find the container with id 8699ff23ff5e33b999dee0171b95546ba20ab189c67f61e7ace4ddd3a6fd5524 Oct 30 22:10:04.165059 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:04.165025 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:10:04.165136 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:04.165095 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:10:04.165136 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:04.165125 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:10:04.733743 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:04.733704 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" event={"ID":"2a4486c4-7472-4dae-bd5c-2c56cd386fcb","Type":"ContainerStarted","Data":"3bb6bad07e2be4b71bcd28c20ce968861761a835c469ee4c7b6a2c2fc3eb55ee"} Oct 30 22:10:04.734208 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:04.733764 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" event={"ID":"2a4486c4-7472-4dae-bd5c-2c56cd386fcb","Type":"ContainerStarted","Data":"8699ff23ff5e33b999dee0171b95546ba20ab189c67f61e7ace4ddd3a6fd5524"} Oct 30 22:10:04.865340 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:04.865289 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" podStartSLOduration=1.865276205 podStartE2EDuration="1.865276205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:04.864783289 +0000 UTC m=+1165.135481373" watchObservedRunningTime="2026-10-30 22:10:04.865276205 +0000 UTC m=+1165.135974288" Oct 30 22:10:05.738243 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:05.738183 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" Oct 30 22:10:05.767277 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:05.766328 5918 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 22:10:05.767277 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:05.766388 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" podUID="2a4486c4-7472-4dae-bd5c-2c56cd386fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 22:10:06.732606 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:06.732573 5918 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 22:10:06.732880 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:06.732855 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" podUID="2a4486c4-7472-4dae-bd5c-2c56cd386fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 22:10:06.930347 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:06.930300 5918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" podUID="05ab563b-c4c2-4fa7-85e4-423141587664" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:07.007413 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:07.007306 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" podUID="6536df0a-a317-4b24-aaa9-c4315d2f7dd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:07.063599 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:07.063552 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" podUID="dc20dd97-eb75-42c5-8e50-6b0b5cb54b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:07.092383 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:07.092345 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" podUID="e63fad41-0e42-41ee-ae20-919f15fffe55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:07.093544 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:07.093506 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" podUID="ab957b47-90d6-497a-8334-1d5d4541dc7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:07.202338 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:07.202292 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" podUID="05ab563b-c4c2-4fa7-85e4-423141587664" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:07.258296 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:07.257362 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" podUID="f5ae2434-f161-4a5c-8086-2f0711678bb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:07.384015 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:07.383968 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" podUID="21ef7a75-4005-4d56-9b35-783dc4d9cd2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:07.444241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:07.438328 5918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" podUID="ab957b47-90d6-497a-8334-1d5d4541dc7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:07.710242 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:07.709288 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" podUID="395b4d7d-4aeb-4576-b616-67638bb1a2c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:07.731683 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:07.731646 5918 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 22:10:07.731985 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:07.731954 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" podUID="2a4486c4-7472-4dae-bd5c-2c56cd386fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 22:10:08.121381 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:08.121030 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" podUID="ab957b47-90d6-497a-8334-1d5d4541dc7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:08.195238 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:08.133961 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" podUID="6536df0a-a317-4b24-aaa9-c4315d2f7dd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 22:10:08.195238 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:08.135157 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" podUID="e63fad41-0e42-41ee-ae20-919f15fffe55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 22:10:08.195496 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:08.135184 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" podUID="dc20dd97-eb75-42c5-8e50-6b0b5cb54b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 22:10:08.208323 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:08.208284 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" podUID="05ab563b-c4c2-4fa7-85e4-423141587664" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:08.285738 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:08.285690 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" podUID="f5ae2434-f161-4a5c-8086-2f0711678bb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:08.288945 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:08.288922 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" Oct 30 22:10:08.289146 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:08.289132 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" Oct 30 22:10:08.289304 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:08.289291 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" Oct 30 22:10:08.289452 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:08.289440 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" Oct 30 22:10:08.289614 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:08.289601 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" Oct 30 22:10:08.301062 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:08.301035 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" Oct 30 22:10:08.346857 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:08.346810 5918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" podUID="f5ae2434-f161-4a5c-8086-2f0711678bb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:08.446408 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:08.446381 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" Oct 30 22:10:08.461347 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:08.461306 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" podUID="21ef7a75-4005-4d56-9b35-783dc4d9cd2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 22:10:08.486088 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:08.477239 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" Oct 30 22:10:08.724304 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:08.707736 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" Oct 30 22:10:08.724304 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:08.711068 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" podUID="395b4d7d-4aeb-4576-b616-67638bb1a2c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:08.763266 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:08.711028 5918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" podUID="e63fad41-0e42-41ee-ae20-919f15fffe55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:08.763266 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:08.755255 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" Oct 30 22:10:08.765591 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:08.764539 5918 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 22:10:08.765591 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:08.764598 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" podUID="2a4486c4-7472-4dae-bd5c-2c56cd386fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 22:10:08.765591 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:08.764720 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" Oct 30 22:10:09.233460 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:09.212050 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" podUID="6536df0a-a317-4b24-aaa9-c4315d2f7dd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:09.288284 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:09.240544 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" podUID="dc20dd97-eb75-42c5-8e50-6b0b5cb54b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 22:10:09.305880 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:09.240574 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" podUID="e63fad41-0e42-41ee-ae20-919f15fffe55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 22:10:09.375883 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:09.375858 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" Oct 30 22:10:09.395323 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:09.395300 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" Oct 30 22:10:09.443274 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:09.443234 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" podUID="05ab563b-c4c2-4fa7-85e4-423141587664" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:09.574887 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:09.574803 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" Oct 30 22:10:09.668884 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:09.668857 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" Oct 30 22:10:09.669153 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:09.669137 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" Oct 30 22:10:09.731425 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:09.731389 5918 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 22:10:09.731594 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:09.731435 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" podUID="2a4486c4-7472-4dae-bd5c-2c56cd386fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 22:10:10.730892 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:10.730865 5918 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 22:10:10.731381 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:10.731355 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" podUID="2a4486c4-7472-4dae-bd5c-2c56cd386fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 22:10:11.731084 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:11.731052 5918 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 22:10:11.731508 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:11.731104 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" podUID="2a4486c4-7472-4dae-bd5c-2c56cd386fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 22:10:12.832062 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:12.832030 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" Oct 30 22:10:12.832938 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:12.832918 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" Oct 30 22:10:12.833648 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:12.833626 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" Oct 30 22:10:18.138107 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:18.138071 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf"] Oct 30 22:10:18.142501 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:18.142484 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" Oct 30 22:10:18.197264 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:18.197203 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-8ca52a-openshift-default-dockercfg-fdgr2\"" Oct 30 22:10:18.220938 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:18.220895 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5489ae6-f93c-4bcf-955d-54637b0399e3-istio-data\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf\" (UID: \"f5489ae6-f93c-4bcf-955d-54637b0399e3\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" Oct 30 22:10:18.221122 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:18.220955 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5489ae6-f93c-4bcf-955d-54637b0399e3-istiod-ca-cert\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf\" (UID: \"f5489ae6-f93c-4bcf-955d-54637b0399e3\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" Oct 30 22:10:18.221231 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:18.221182 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5489ae6-f93c-4bcf-955d-54637b0399e3-istio-envoy\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf\" (UID: \"f5489ae6-f93c-4bcf-955d-54637b0399e3\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" Oct 30 22:10:18.221305 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:18.221293 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5489ae6-f93c-4bcf-955d-54637b0399e3-istio-podinfo\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf\" (UID: \"f5489ae6-f93c-4bcf-955d-54637b0399e3\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" Oct 30 22:10:18.221382 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:18.221365 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rhtz\" (UniqueName: \"kubernetes.io/projected/f5489ae6-f93c-4bcf-955d-54637b0399e3-kube-api-access-7rhtz\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf\" (UID: \"f5489ae6-f93c-4bcf-955d-54637b0399e3\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" Oct 30 22:10:18.221450 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:18.221435 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5489ae6-f93c-4bcf-955d-54637b0399e3-credential-socket\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf\" (UID: \"f5489ae6-f93c-4bcf-955d-54637b0399e3\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" Oct 30 22:10:18.221511 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:18.221466 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5489ae6-f93c-4bcf-955d-54637b0399e3-workload-socket\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf\" (UID: \"f5489ae6-f93c-4bcf-955d-54637b0399e3\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" Oct 30 22:10:18.221568 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:18.221552 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5489ae6-f93c-4bcf-955d-54637b0399e3-istio-token\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf\" (UID: \"f5489ae6-f93c-4bcf-955d-54637b0399e3\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" Oct 30 22:10:18.221620 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:18.221579 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5489ae6-f93c-4bcf-955d-54637b0399e3-workload-certs\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf\" (UID: \"f5489ae6-f93c-4bcf-955d-54637b0399e3\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" Oct 30 22:10:18.250662 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:18.250627 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf"] Oct 30 22:10:18.322893 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:18.322863 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rhtz\" (UniqueName: \"kubernetes.io/projected/f5489ae6-f93c-4bcf-955d-54637b0399e3-kube-api-access-7rhtz\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf\" (UID: \"f5489ae6-f93c-4bcf-955d-54637b0399e3\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" Oct 30 22:10:18.323075 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:18.322924 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5489ae6-f93c-4bcf-955d-54637b0399e3-credential-socket\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf\" (UID: \"f5489ae6-f93c-4bcf-955d-54637b0399e3\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" Oct 30 22:10:18.323075 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:18.322949 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5489ae6-f93c-4bcf-955d-54637b0399e3-workload-socket\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf\" (UID: \"f5489ae6-f93c-4bcf-955d-54637b0399e3\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" Oct 30 22:10:18.323075 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:18.322998 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5489ae6-f93c-4bcf-955d-54637b0399e3-istio-token\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf\" (UID: \"f5489ae6-f93c-4bcf-955d-54637b0399e3\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" Oct 30 22:10:18.323075 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:18.323025 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5489ae6-f93c-4bcf-955d-54637b0399e3-workload-certs\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf\" (UID: \"f5489ae6-f93c-4bcf-955d-54637b0399e3\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" Oct 30 22:10:18.323362 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:18.323135 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5489ae6-f93c-4bcf-955d-54637b0399e3-istio-data\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf\" (UID: \"f5489ae6-f93c-4bcf-955d-54637b0399e3\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" Oct 30 22:10:18.323362 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:18.323288 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5489ae6-f93c-4bcf-955d-54637b0399e3-istiod-ca-cert\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf\" (UID: \"f5489ae6-f93c-4bcf-955d-54637b0399e3\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" Oct 30 22:10:18.323485 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:18.323462 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5489ae6-f93c-4bcf-955d-54637b0399e3-istio-envoy\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf\" (UID: \"f5489ae6-f93c-4bcf-955d-54637b0399e3\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" Oct 30 22:10:18.323539 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:18.323485 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5489ae6-f93c-4bcf-955d-54637b0399e3-workload-socket\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf\" (UID: \"f5489ae6-f93c-4bcf-955d-54637b0399e3\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" Oct 30 22:10:18.323539 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:18.323521 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5489ae6-f93c-4bcf-955d-54637b0399e3-istio-podinfo\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf\" (UID: \"f5489ae6-f93c-4bcf-955d-54637b0399e3\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" Oct 30 22:10:18.323633 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:18.323556 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5489ae6-f93c-4bcf-955d-54637b0399e3-workload-certs\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf\" (UID: \"f5489ae6-f93c-4bcf-955d-54637b0399e3\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" Oct 30 22:10:18.323633 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:18.323573 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5489ae6-f93c-4bcf-955d-54637b0399e3-istio-data\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf\" (UID: \"f5489ae6-f93c-4bcf-955d-54637b0399e3\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" Oct 30 22:10:18.323708 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:18.323652 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5489ae6-f93c-4bcf-955d-54637b0399e3-credential-socket\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf\" (UID: \"f5489ae6-f93c-4bcf-955d-54637b0399e3\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" Oct 30 22:10:18.324064 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:18.324037 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5489ae6-f93c-4bcf-955d-54637b0399e3-istiod-ca-cert\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf\" (UID: \"f5489ae6-f93c-4bcf-955d-54637b0399e3\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" Oct 30 22:10:18.325772 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:18.325745 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5489ae6-f93c-4bcf-955d-54637b0399e3-istio-envoy\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf\" (UID: \"f5489ae6-f93c-4bcf-955d-54637b0399e3\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" Oct 30 22:10:18.326173 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:18.326155 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5489ae6-f93c-4bcf-955d-54637b0399e3-istio-podinfo\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf\" (UID: \"f5489ae6-f93c-4bcf-955d-54637b0399e3\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" Oct 30 22:10:18.386415 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:18.386386 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5489ae6-f93c-4bcf-955d-54637b0399e3-istio-token\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf\" (UID: \"f5489ae6-f93c-4bcf-955d-54637b0399e3\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" Oct 30 22:10:18.386586 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:18.386568 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rhtz\" (UniqueName: \"kubernetes.io/projected/f5489ae6-f93c-4bcf-955d-54637b0399e3-kube-api-access-7rhtz\") pod \"e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf\" (UID: \"f5489ae6-f93c-4bcf-955d-54637b0399e3\") " pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" Oct 30 22:10:18.457420 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:18.457396 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" Oct 30 22:10:18.911541 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:10:18.911510 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5489ae6_f93c_4bcf_955d_54637b0399e3.slice/crio-530ba836e0e4d24af10b355cd5810e51e994e5a71f5dc8e1085c66234d681886 WatchSource:0}: Error finding container 530ba836e0e4d24af10b355cd5810e51e994e5a71f5dc8e1085c66234d681886: Status 404 returned error can't find the container with id 530ba836e0e4d24af10b355cd5810e51e994e5a71f5dc8e1085c66234d681886 Oct 30 22:10:18.911759 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:18.911735 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf"] Oct 30 22:10:18.914471 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:18.914427 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:10:18.914555 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:18.914518 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:10:18.914600 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:18.914562 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:10:19.110624 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:19.110578 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" event={"ID":"f5489ae6-f93c-4bcf-955d-54637b0399e3","Type":"ContainerStarted","Data":"ef1c4ffe9b50423a97b75be3a17c1df34e3e39103d88409226705d49c5028320"} Oct 30 22:10:19.110799 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:19.110652 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" event={"ID":"f5489ae6-f93c-4bcf-955d-54637b0399e3","Type":"ContainerStarted","Data":"530ba836e0e4d24af10b355cd5810e51e994e5a71f5dc8e1085c66234d681886"} Oct 30 22:10:19.239965 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:19.239913 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" podStartSLOduration=1.23989931 podStartE2EDuration="1.23989931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:19.238415579 +0000 UTC m=+1179.509113665" watchObservedRunningTime="2026-10-30 22:10:19.23989931 +0000 UTC m=+1179.510597394" Oct 30 22:10:19.457882 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:19.457854 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" Oct 30 22:10:19.459157 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:19.459135 5918 patch_prober.go:29] interesting pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:10:19.459252 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:19.459181 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" podUID="f5489ae6-f93c-4bcf-955d-54637b0399e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 22:10:20.458279 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:20.458247 5918 patch_prober.go:29] interesting pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:10:20.458692 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:20.458298 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" podUID="f5489ae6-f93c-4bcf-955d-54637b0399e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 22:10:21.276652 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:10:21.276614 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:10:21.458360 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:21.458326 5918 patch_prober.go:29] interesting pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:10:21.458778 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:21.458388 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" podUID="f5489ae6-f93c-4bcf-955d-54637b0399e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 22:10:22.458446 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:22.458416 5918 patch_prober.go:29] interesting pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:10:22.458818 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:22.458466 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" podUID="f5489ae6-f93c-4bcf-955d-54637b0399e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 22:10:23.421691 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:23.421654 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989"] Oct 30 22:10:23.421902 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:23.421833 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" podUID="5fb62082-3ac2-4985-854f-9a63194f65e1" containerName="istio-proxy" containerID="cri-o://663f531ae3de7c53bf3a5f4ab0340b921d2e083202e15a6d3c67da244b456012" gracePeriod=30 Oct 30 22:10:23.458402 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:23.458343 5918 patch_prober.go:29] interesting pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:10:23.458670 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:23.458643 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" podUID="f5489ae6-f93c-4bcf-955d-54637b0399e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 22:10:24.458514 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:24.458478 5918 patch_prober.go:29] interesting pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:10:24.458718 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:24.458543 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" podUID="f5489ae6-f93c-4bcf-955d-54637b0399e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 22:10:25.458368 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:25.458336 5918 patch_prober.go:29] interesting pod/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:10:25.458538 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:25.458386 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" podUID="f5489ae6-f93c-4bcf-955d-54637b0399e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 22:10:26.524600 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:26.524559 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" Oct 30 22:10:26.525589 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:26.525563 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" Oct 30 22:10:26.526754 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:26.526729 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" Oct 30 22:10:28.583676 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:28.582981 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" podUID="395b4d7d-4aeb-4576-b616-67638bb1a2c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:28.618363 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:28.618075 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" podUID="1a1ae368-0e8c-408d-bfcd-8f4a8194464f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 22:10:28.653312 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:28.630876 5918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" podUID="e63fad41-0e42-41ee-ae20-919f15fffe55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:28.676985 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:28.676941 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" podUID="e63fad41-0e42-41ee-ae20-919f15fffe55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:28.695488 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:28.695333 5918 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:10:28.695488 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:28.695417 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" podUID="f9b3bd36-51f3-4b46-bed6-94703557bfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:28.710548 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:28.710391 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" podUID="ab957b47-90d6-497a-8334-1d5d4541dc7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 22:10:28.762089 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:28.761313 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" podUID="f5ae2434-f161-4a5c-8086-2f0711678bb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:29.232242 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:29.228826 5918 generic.go:356] "Generic (PLEG): container finished" podID="5fb62082-3ac2-4985-854f-9a63194f65e1" containerID="663f531ae3de7c53bf3a5f4ab0340b921d2e083202e15a6d3c67da244b456012" exitCode=0 Oct 30 22:10:29.232242 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:29.229045 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" event={"ID":"5fb62082-3ac2-4985-854f-9a63194f65e1","Type":"ContainerDied","Data":"663f531ae3de7c53bf3a5f4ab0340b921d2e083202e15a6d3c67da244b456012"} Oct 30 22:10:29.547914 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:29.528705 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" podUID="dc20dd97-eb75-42c5-8e50-6b0b5cb54b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:29.564763 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:29.556887 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" podUID="05ab563b-c4c2-4fa7-85e4-423141587664" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:29.609251 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:29.596322 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" podUID="395b4d7d-4aeb-4576-b616-67638bb1a2c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:29.609251 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:29.596785 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" Oct 30 22:10:29.619407 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:29.619356 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" podUID="1a1ae368-0e8c-408d-bfcd-8f4a8194464f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:29.620119 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:29.620091 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" Oct 30 22:10:29.722269 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:29.677565 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" podUID="e63fad41-0e42-41ee-ae20-919f15fffe55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:29.722269 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:29.699234 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" Oct 30 22:10:29.722269 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:29.716300 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" podUID="ab957b47-90d6-497a-8334-1d5d4541dc7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 22:10:29.725035 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:29.723666 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" Oct 30 22:10:29.862323 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:29.768563 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" podUID="f5ae2434-f161-4a5c-8086-2f0711678bb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:29.862323 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:29.805196 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" Oct 30 22:10:30.169627 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.169548 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" Oct 30 22:10:30.313245 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.313075 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" event={"ID":"5fb62082-3ac2-4985-854f-9a63194f65e1","Type":"ContainerDied","Data":"6104f54022db0a1c50ee371e66edf79899a76fe12619ed44f1ea592561039219"} Oct 30 22:10:30.313245 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.313154 5918 scope.go:122] "RemoveContainer" containerID="663f531ae3de7c53bf3a5f4ab0340b921d2e083202e15a6d3c67da244b456012" Oct 30 22:10:30.313519 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.313395 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989" Oct 30 22:10:30.374268 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.369760 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fb62082-3ac2-4985-854f-9a63194f65e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb62082-3ac2-4985-854f-9a63194f65e1-istio-envoy\") pod \"5fb62082-3ac2-4985-854f-9a63194f65e1\" (UID: \"5fb62082-3ac2-4985-854f-9a63194f65e1\") " Oct 30 22:10:30.374268 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.369883 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fb62082-3ac2-4985-854f-9a63194f65e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb62082-3ac2-4985-854f-9a63194f65e1-istiod-ca-cert\") pod \"5fb62082-3ac2-4985-854f-9a63194f65e1\" (UID: \"5fb62082-3ac2-4985-854f-9a63194f65e1\") " Oct 30 22:10:30.374268 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.369980 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fb62082-3ac2-4985-854f-9a63194f65e1-kube-api-access-tkrp7\" (UniqueName: \"kubernetes.io/projected/5fb62082-3ac2-4985-854f-9a63194f65e1-kube-api-access-tkrp7\") pod \"5fb62082-3ac2-4985-854f-9a63194f65e1\" (UID: \"5fb62082-3ac2-4985-854f-9a63194f65e1\") " Oct 30 22:10:30.374268 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.370016 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5fb62082-3ac2-4985-854f-9a63194f65e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb62082-3ac2-4985-854f-9a63194f65e1-istio-podinfo\") pod \"5fb62082-3ac2-4985-854f-9a63194f65e1\" (UID: \"5fb62082-3ac2-4985-854f-9a63194f65e1\") " Oct 30 22:10:30.374268 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.370101 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fb62082-3ac2-4985-854f-9a63194f65e1-istio-token\" (UniqueName: \"kubernetes.io/projected/5fb62082-3ac2-4985-854f-9a63194f65e1-istio-token\") pod \"5fb62082-3ac2-4985-854f-9a63194f65e1\" (UID: \"5fb62082-3ac2-4985-854f-9a63194f65e1\") " Oct 30 22:10:30.374268 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.370144 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fb62082-3ac2-4985-854f-9a63194f65e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb62082-3ac2-4985-854f-9a63194f65e1-credential-socket\") pod \"5fb62082-3ac2-4985-854f-9a63194f65e1\" (UID: \"5fb62082-3ac2-4985-854f-9a63194f65e1\") " Oct 30 22:10:30.374268 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.370177 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fb62082-3ac2-4985-854f-9a63194f65e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb62082-3ac2-4985-854f-9a63194f65e1-workload-socket\") pod \"5fb62082-3ac2-4985-854f-9a63194f65e1\" (UID: \"5fb62082-3ac2-4985-854f-9a63194f65e1\") " Oct 30 22:10:30.374268 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.370282 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fb62082-3ac2-4985-854f-9a63194f65e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb62082-3ac2-4985-854f-9a63194f65e1-istio-data\") pod \"5fb62082-3ac2-4985-854f-9a63194f65e1\" (UID: \"5fb62082-3ac2-4985-854f-9a63194f65e1\") " Oct 30 22:10:30.374268 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.370380 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fb62082-3ac2-4985-854f-9a63194f65e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb62082-3ac2-4985-854f-9a63194f65e1-workload-certs\") pod \"5fb62082-3ac2-4985-854f-9a63194f65e1\" (UID: \"5fb62082-3ac2-4985-854f-9a63194f65e1\") " Oct 30 22:10:30.397369 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.397334 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fb62082-3ac2-4985-854f-9a63194f65e1-istiod-ca-cert" pod "5fb62082-3ac2-4985-854f-9a63194f65e1" (UID: "5fb62082-3ac2-4985-854f-9a63194f65e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:30.397931 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.397905 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb62082-3ac2-4985-854f-9a63194f65e1-workload-socket" pod "5fb62082-3ac2-4985-854f-9a63194f65e1" (UID: "5fb62082-3ac2-4985-854f-9a63194f65e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:30.398409 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.398387 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb62082-3ac2-4985-854f-9a63194f65e1-credential-socket" pod "5fb62082-3ac2-4985-854f-9a63194f65e1" (UID: "5fb62082-3ac2-4985-854f-9a63194f65e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:30.410802 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.409628 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb62082-3ac2-4985-854f-9a63194f65e1-istio-data" pod "5fb62082-3ac2-4985-854f-9a63194f65e1" (UID: "5fb62082-3ac2-4985-854f-9a63194f65e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:30.417128 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.417098 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb62082-3ac2-4985-854f-9a63194f65e1-workload-certs" pod "5fb62082-3ac2-4985-854f-9a63194f65e1" (UID: "5fb62082-3ac2-4985-854f-9a63194f65e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:30.500706 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.500663 5918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb62082-3ac2-4985-854f-9a63194f65e1-istiod-ca-cert\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:30.500954 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.500935 5918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb62082-3ac2-4985-854f-9a63194f65e1-credential-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:30.501375 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.501066 5918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb62082-3ac2-4985-854f-9a63194f65e1-workload-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:30.533297 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.533268 5918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb62082-3ac2-4985-854f-9a63194f65e1-istio-data\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:30.557246 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.550047 5918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb62082-3ac2-4985-854f-9a63194f65e1-workload-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:30.557457 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.527418 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fb62082-3ac2-4985-854f-9a63194f65e1-kube-api-access-tkrp7" pod "5fb62082-3ac2-4985-854f-9a63194f65e1" (UID: "5fb62082-3ac2-4985-854f-9a63194f65e1"). InnerVolumeSpecName "kube-api-access-tkrp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:30.557555 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.539291 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" podUID="6536df0a-a317-4b24-aaa9-c4315d2f7dd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:30.558943 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.532462 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" podUID="21ef7a75-4005-4d56-9b35-783dc4d9cd2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:30.577043 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.555348 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" podUID="dc20dd97-eb75-42c5-8e50-6b0b5cb54b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:30.577043 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.570494 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" Oct 30 22:10:30.577043 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.571445 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" podUID="05ab563b-c4c2-4fa7-85e4-423141587664" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:30.577043 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.572342 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" Oct 30 22:10:30.601245 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.590282 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fb62082-3ac2-4985-854f-9a63194f65e1-istio-podinfo" pod "5fb62082-3ac2-4985-854f-9a63194f65e1" (UID: "5fb62082-3ac2-4985-854f-9a63194f65e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:10:30.601245 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.596629 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" podUID="395b4d7d-4aeb-4576-b616-67638bb1a2c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:30.629940 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.629320 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" podUID="1a1ae368-0e8c-408d-bfcd-8f4a8194464f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:30.671246 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.670890 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkrp7\" (UniqueName: \"kubernetes.io/projected/5fb62082-3ac2-4985-854f-9a63194f65e1-kube-api-access-tkrp7\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:30.671246 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.670927 5918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb62082-3ac2-4985-854f-9a63194f65e1-istio-podinfo\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:30.693393 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.683293 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" podUID="e63fad41-0e42-41ee-ae20-919f15fffe55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 22:10:30.693393 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.692292 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb62082-3ac2-4985-854f-9a63194f65e1-istio-envoy" pod "5fb62082-3ac2-4985-854f-9a63194f65e1" (UID: "5fb62082-3ac2-4985-854f-9a63194f65e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:30.693393 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.692397 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fb62082-3ac2-4985-854f-9a63194f65e1-istio-token" pod "5fb62082-3ac2-4985-854f-9a63194f65e1" (UID: "5fb62082-3ac2-4985-854f-9a63194f65e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:30.696784 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.696763 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" Oct 30 22:10:30.789164 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.789074 5918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb62082-3ac2-4985-854f-9a63194f65e1-istio-envoy\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:30.791703 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.791682 5918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb62082-3ac2-4985-854f-9a63194f65e1-istio-token\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:30.791869 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.790036 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" podUID="f5ae2434-f161-4a5c-8086-2f0711678bb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:30.977109 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:30.977080 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" Oct 30 22:10:31.097299 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:31.089608 5918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" podUID="dc20dd97-eb75-42c5-8e50-6b0b5cb54b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:31.414291 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:31.409573 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" Oct 30 22:10:31.504244 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:31.488153 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" Oct 30 22:10:31.615475 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:31.615448 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" Oct 30 22:10:31.650242 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:31.626752 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989"] Oct 30 22:10:31.650845 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:31.650819 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9q989"] Oct 30 22:10:31.650978 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:10:31.633838 5918 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fb62082_3ac2_4985_854f_9a63194f65e1.slice/crio-6104f54022db0a1c50ee371e66edf79899a76fe12619ed44f1ea592561039219\": RecentStats: unable to find data in memory cache]" Oct 30 22:10:31.651264 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:31.630297 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" podUID="1a1ae368-0e8c-408d-bfcd-8f4a8194464f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:31.658238 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:31.655812 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" podUID="05ab563b-c4c2-4fa7-85e4-423141587664" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:31.757276 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:31.727161 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" Oct 30 22:10:31.848673 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:31.818746 5918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" podUID="6536df0a-a317-4b24-aaa9-c4315d2f7dd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:32.029573 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:32.028933 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fb62082-3ac2-4985-854f-9a63194f65e1" path="/var/lib/kubelet/pods/5fb62082-3ac2-4985-854f-9a63194f65e1/volumes" Oct 30 22:10:32.030428 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:32.029784 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" Oct 30 22:10:49.362126 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:49.362089 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc"] Oct 30 22:10:49.363163 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:49.363143 5918 cpu_manager.go:414] "containerMap: removing container" podUID="5fb62082-3ac2-4985-854f-9a63194f65e1" containerName="istio-proxy" Oct 30 22:10:49.363222 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:49.363168 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fb62082-3ac2-4985-854f-9a63194f65e1" containerName="istio-proxy" Oct 30 22:10:49.363365 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:49.363353 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fb62082-3ac2-4985-854f-9a63194f65e1" containerName="istio-proxy" Oct 30 22:10:49.369449 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:49.369429 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" Oct 30 22:10:49.476813 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:49.476770 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7df7a416-e18e-4713-89b1-5b3f21e9251d-workload-certs\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc\" (UID: \"7df7a416-e18e-4713-89b1-5b3f21e9251d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" Oct 30 22:10:49.476813 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:49.476915 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7df7a416-e18e-4713-89b1-5b3f21e9251d-istio-podinfo\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc\" (UID: \"7df7a416-e18e-4713-89b1-5b3f21e9251d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" Oct 30 22:10:49.476813 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:49.476952 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df7a416-e18e-4713-89b1-5b3f21e9251d-credential-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc\" (UID: \"7df7a416-e18e-4713-89b1-5b3f21e9251d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" Oct 30 22:10:49.476813 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:49.477026 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7df7a416-e18e-4713-89b1-5b3f21e9251d-istio-data\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc\" (UID: \"7df7a416-e18e-4713-89b1-5b3f21e9251d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" Oct 30 22:10:49.476813 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:49.477142 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7df7a416-e18e-4713-89b1-5b3f21e9251d-istiod-ca-cert\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc\" (UID: \"7df7a416-e18e-4713-89b1-5b3f21e9251d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" Oct 30 22:10:49.478242 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:49.477599 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbm5p\" (UniqueName: \"kubernetes.io/projected/7df7a416-e18e-4713-89b1-5b3f21e9251d-kube-api-access-vbm5p\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc\" (UID: \"7df7a416-e18e-4713-89b1-5b3f21e9251d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" Oct 30 22:10:49.478242 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:49.477713 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df7a416-e18e-4713-89b1-5b3f21e9251d-workload-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc\" (UID: \"7df7a416-e18e-4713-89b1-5b3f21e9251d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" Oct 30 22:10:49.478242 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:49.477858 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7df7a416-e18e-4713-89b1-5b3f21e9251d-istio-envoy\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc\" (UID: \"7df7a416-e18e-4713-89b1-5b3f21e9251d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" Oct 30 22:10:49.478242 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:49.478010 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7df7a416-e18e-4713-89b1-5b3f21e9251d-istio-token\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc\" (UID: \"7df7a416-e18e-4713-89b1-5b3f21e9251d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" Oct 30 22:10:49.580678 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:49.580630 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7df7a416-e18e-4713-89b1-5b3f21e9251d-istio-envoy\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc\" (UID: \"7df7a416-e18e-4713-89b1-5b3f21e9251d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" Oct 30 22:10:49.580954 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:49.580735 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7df7a416-e18e-4713-89b1-5b3f21e9251d-istio-token\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc\" (UID: \"7df7a416-e18e-4713-89b1-5b3f21e9251d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" Oct 30 22:10:49.580954 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:49.580891 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7df7a416-e18e-4713-89b1-5b3f21e9251d-workload-certs\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc\" (UID: \"7df7a416-e18e-4713-89b1-5b3f21e9251d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" Oct 30 22:10:49.580954 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:49.580916 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7df7a416-e18e-4713-89b1-5b3f21e9251d-istio-podinfo\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc\" (UID: \"7df7a416-e18e-4713-89b1-5b3f21e9251d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" Oct 30 22:10:49.581112 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:49.580964 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df7a416-e18e-4713-89b1-5b3f21e9251d-credential-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc\" (UID: \"7df7a416-e18e-4713-89b1-5b3f21e9251d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" Oct 30 22:10:49.581112 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:49.580982 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7df7a416-e18e-4713-89b1-5b3f21e9251d-istio-data\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc\" (UID: \"7df7a416-e18e-4713-89b1-5b3f21e9251d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" Oct 30 22:10:49.581112 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:49.581024 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7df7a416-e18e-4713-89b1-5b3f21e9251d-istiod-ca-cert\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc\" (UID: \"7df7a416-e18e-4713-89b1-5b3f21e9251d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" Oct 30 22:10:49.581112 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:49.581054 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbm5p\" (UniqueName: \"kubernetes.io/projected/7df7a416-e18e-4713-89b1-5b3f21e9251d-kube-api-access-vbm5p\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc\" (UID: \"7df7a416-e18e-4713-89b1-5b3f21e9251d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" Oct 30 22:10:49.581112 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:49.581102 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df7a416-e18e-4713-89b1-5b3f21e9251d-workload-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc\" (UID: \"7df7a416-e18e-4713-89b1-5b3f21e9251d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" Oct 30 22:10:49.582832 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:49.581972 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7df7a416-e18e-4713-89b1-5b3f21e9251d-workload-certs\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc\" (UID: \"7df7a416-e18e-4713-89b1-5b3f21e9251d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" Oct 30 22:10:49.583389 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:49.583367 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df7a416-e18e-4713-89b1-5b3f21e9251d-workload-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc\" (UID: \"7df7a416-e18e-4713-89b1-5b3f21e9251d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" Oct 30 22:10:49.584264 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:49.583592 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df7a416-e18e-4713-89b1-5b3f21e9251d-credential-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc\" (UID: \"7df7a416-e18e-4713-89b1-5b3f21e9251d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" Oct 30 22:10:49.584400 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:49.584194 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7df7a416-e18e-4713-89b1-5b3f21e9251d-istiod-ca-cert\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc\" (UID: \"7df7a416-e18e-4713-89b1-5b3f21e9251d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" Oct 30 22:10:49.584753 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:49.584710 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7df7a416-e18e-4713-89b1-5b3f21e9251d-istio-envoy\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc\" (UID: \"7df7a416-e18e-4713-89b1-5b3f21e9251d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" Oct 30 22:10:49.584900 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:49.584880 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7df7a416-e18e-4713-89b1-5b3f21e9251d-istio-data\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc\" (UID: \"7df7a416-e18e-4713-89b1-5b3f21e9251d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" Oct 30 22:10:49.586261 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:49.585836 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7df7a416-e18e-4713-89b1-5b3f21e9251d-istio-podinfo\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc\" (UID: \"7df7a416-e18e-4713-89b1-5b3f21e9251d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" Oct 30 22:10:49.587684 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:49.587662 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc"] Oct 30 22:10:49.646799 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:49.646717 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbm5p\" (UniqueName: \"kubernetes.io/projected/7df7a416-e18e-4713-89b1-5b3f21e9251d-kube-api-access-vbm5p\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc\" (UID: \"7df7a416-e18e-4713-89b1-5b3f21e9251d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" Oct 30 22:10:49.646928 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:49.646820 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7df7a416-e18e-4713-89b1-5b3f21e9251d-istio-token\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc\" (UID: \"7df7a416-e18e-4713-89b1-5b3f21e9251d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" Oct 30 22:10:49.685181 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:49.685153 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" Oct 30 22:10:50.119336 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:50.119309 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc"] Oct 30 22:10:50.119721 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:10:50.119695 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df7a416_e18e_4713_89b1_5b3f21e9251d.slice/crio-bb25eb334ee24e8d05371dda12ed51b11748bb5965f4b63b26bd4fec794af7e1 WatchSource:0}: Error finding container bb25eb334ee24e8d05371dda12ed51b11748bb5965f4b63b26bd4fec794af7e1: Status 404 returned error can't find the container with id bb25eb334ee24e8d05371dda12ed51b11748bb5965f4b63b26bd4fec794af7e1 Oct 30 22:10:50.122061 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:50.122030 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:10:50.122133 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:50.122101 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:10:50.122237 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:50.122130 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:10:50.753292 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:50.753259 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" event={"ID":"7df7a416-e18e-4713-89b1-5b3f21e9251d","Type":"ContainerStarted","Data":"d351f8766665b4e44b2ef7bb2ea09fa5e2d582a074df19124121eef746a39bda"} Oct 30 22:10:50.753765 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:50.753316 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" event={"ID":"7df7a416-e18e-4713-89b1-5b3f21e9251d","Type":"ContainerStarted","Data":"bb25eb334ee24e8d05371dda12ed51b11748bb5965f4b63b26bd4fec794af7e1"} Oct 30 22:10:51.011366 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:51.010546 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" podStartSLOduration=2.010519734 podStartE2EDuration="2.010519734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:50.94576698 +0000 UTC m=+1211.216465067" watchObservedRunningTime="2026-10-30 22:10:51.010519734 +0000 UTC m=+1211.281217819" Oct 30 22:10:51.275191 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:10:51.275104 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:10:51.707239 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:51.686164 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" Oct 30 22:10:51.711301 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:51.708345 5918 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:10:51.711301 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:51.708411 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" podUID="7df7a416-e18e-4713-89b1-5b3f21e9251d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 22:10:52.685656 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:52.685626 5918 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:10:52.686030 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:52.685677 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" podUID="7df7a416-e18e-4713-89b1-5b3f21e9251d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 22:10:53.686142 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:53.686110 5918 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:10:53.686651 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:53.686165 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" podUID="7df7a416-e18e-4713-89b1-5b3f21e9251d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 22:10:54.690241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:54.686417 5918 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:10:54.690241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:54.686470 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" podUID="7df7a416-e18e-4713-89b1-5b3f21e9251d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 22:10:55.685836 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:55.685798 5918 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:10:55.686046 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:55.685857 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" podUID="7df7a416-e18e-4713-89b1-5b3f21e9251d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 22:10:56.686353 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:56.686315 5918 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:10:56.686881 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:56.686375 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" podUID="7df7a416-e18e-4713-89b1-5b3f21e9251d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 22:10:57.689758 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:57.689729 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" Oct 30 22:10:57.690181 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:57.690158 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" Oct 30 22:10:57.690980 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:10:57.690965 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" Oct 30 22:11:04.591240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:04.590278 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst"] Oct 30 22:11:04.614565 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:04.614532 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" Oct 30 22:11:04.705685 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:04.705648 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06181307-c42a-411e-8216-6c8f4d9a4a9d-istio-data\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst\" (UID: \"06181307-c42a-411e-8216-6c8f4d9a4a9d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" Oct 30 22:11:04.733235 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:04.730372 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06181307-c42a-411e-8216-6c8f4d9a4a9d-istio-token\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst\" (UID: \"06181307-c42a-411e-8216-6c8f4d9a4a9d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" Oct 30 22:11:04.733235 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:04.730553 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06181307-c42a-411e-8216-6c8f4d9a4a9d-istio-envoy\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst\" (UID: \"06181307-c42a-411e-8216-6c8f4d9a4a9d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" Oct 30 22:11:04.733235 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:04.730744 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06181307-c42a-411e-8216-6c8f4d9a4a9d-istiod-ca-cert\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst\" (UID: \"06181307-c42a-411e-8216-6c8f4d9a4a9d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" Oct 30 22:11:04.733235 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:04.730895 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz8x2\" (UniqueName: \"kubernetes.io/projected/06181307-c42a-411e-8216-6c8f4d9a4a9d-kube-api-access-kz8x2\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst\" (UID: \"06181307-c42a-411e-8216-6c8f4d9a4a9d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" Oct 30 22:11:04.733235 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:04.730992 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06181307-c42a-411e-8216-6c8f4d9a4a9d-workload-certs\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst\" (UID: \"06181307-c42a-411e-8216-6c8f4d9a4a9d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" Oct 30 22:11:04.733235 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:04.731157 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06181307-c42a-411e-8216-6c8f4d9a4a9d-istio-podinfo\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst\" (UID: \"06181307-c42a-411e-8216-6c8f4d9a4a9d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" Oct 30 22:11:04.733235 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:04.731281 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06181307-c42a-411e-8216-6c8f4d9a4a9d-workload-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst\" (UID: \"06181307-c42a-411e-8216-6c8f4d9a4a9d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" Oct 30 22:11:04.733235 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:04.731354 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06181307-c42a-411e-8216-6c8f4d9a4a9d-credential-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst\" (UID: \"06181307-c42a-411e-8216-6c8f4d9a4a9d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" Oct 30 22:11:04.733799 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:04.733350 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst"] Oct 30 22:11:04.999239 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:04.993317 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" podUID="395b4d7d-4aeb-4576-b616-67638bb1a2c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:04.999239 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:04.996512 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06181307-c42a-411e-8216-6c8f4d9a4a9d-istio-podinfo\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst\" (UID: \"06181307-c42a-411e-8216-6c8f4d9a4a9d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" Oct 30 22:11:04.999239 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:04.996605 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06181307-c42a-411e-8216-6c8f4d9a4a9d-workload-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst\" (UID: \"06181307-c42a-411e-8216-6c8f4d9a4a9d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" Oct 30 22:11:04.999239 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:04.996643 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06181307-c42a-411e-8216-6c8f4d9a4a9d-credential-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst\" (UID: \"06181307-c42a-411e-8216-6c8f4d9a4a9d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" Oct 30 22:11:04.999239 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:04.996860 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06181307-c42a-411e-8216-6c8f4d9a4a9d-istio-data\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst\" (UID: \"06181307-c42a-411e-8216-6c8f4d9a4a9d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" Oct 30 22:11:04.999239 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:04.996904 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06181307-c42a-411e-8216-6c8f4d9a4a9d-istio-token\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst\" (UID: \"06181307-c42a-411e-8216-6c8f4d9a4a9d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" Oct 30 22:11:04.999239 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:04.997117 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06181307-c42a-411e-8216-6c8f4d9a4a9d-istio-envoy\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst\" (UID: \"06181307-c42a-411e-8216-6c8f4d9a4a9d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" Oct 30 22:11:05.065242 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:05.064534 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" podUID="dc20dd97-eb75-42c5-8e50-6b0b5cb54b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:05.315329 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:05.282094 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" podUID="6536df0a-a317-4b24-aaa9-c4315d2f7dd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:05.393580 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:05.393514 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" podUID="21ef7a75-4005-4d56-9b35-783dc4d9cd2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:05.444286 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:05.415100 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" podUID="f5ae2434-f161-4a5c-8086-2f0711678bb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:05.446268 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:05.446207 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06181307-c42a-411e-8216-6c8f4d9a4a9d-istiod-ca-cert\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst\" (UID: \"06181307-c42a-411e-8216-6c8f4d9a4a9d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" Oct 30 22:11:05.456412 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:05.456378 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz8x2\" (UniqueName: \"kubernetes.io/projected/06181307-c42a-411e-8216-6c8f4d9a4a9d-kube-api-access-kz8x2\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst\" (UID: \"06181307-c42a-411e-8216-6c8f4d9a4a9d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" Oct 30 22:11:05.460817 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:05.460788 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06181307-c42a-411e-8216-6c8f4d9a4a9d-workload-certs\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst\" (UID: \"06181307-c42a-411e-8216-6c8f4d9a4a9d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" Oct 30 22:11:05.483275 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:05.451395 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06181307-c42a-411e-8216-6c8f4d9a4a9d-workload-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst\" (UID: \"06181307-c42a-411e-8216-6c8f4d9a4a9d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" Oct 30 22:11:05.483911 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:05.452800 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06181307-c42a-411e-8216-6c8f4d9a4a9d-istiod-ca-cert\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst\" (UID: \"06181307-c42a-411e-8216-6c8f4d9a4a9d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" Oct 30 22:11:05.484123 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:05.446848 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06181307-c42a-411e-8216-6c8f4d9a4a9d-istio-data\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst\" (UID: \"06181307-c42a-411e-8216-6c8f4d9a4a9d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" Oct 30 22:11:05.484356 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:05.447096 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06181307-c42a-411e-8216-6c8f4d9a4a9d-credential-socket\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst\" (UID: \"06181307-c42a-411e-8216-6c8f4d9a4a9d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" Oct 30 22:11:05.484542 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:05.461361 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06181307-c42a-411e-8216-6c8f4d9a4a9d-workload-certs\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst\" (UID: \"06181307-c42a-411e-8216-6c8f4d9a4a9d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" Oct 30 22:11:05.490615 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:05.490576 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06181307-c42a-411e-8216-6c8f4d9a4a9d-istio-envoy\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst\" (UID: \"06181307-c42a-411e-8216-6c8f4d9a4a9d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" Oct 30 22:11:05.502429 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:05.502386 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" podUID="e63fad41-0e42-41ee-ae20-919f15fffe55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 22:11:05.572985 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:05.572901 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06181307-c42a-411e-8216-6c8f4d9a4a9d-istio-podinfo\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst\" (UID: \"06181307-c42a-411e-8216-6c8f4d9a4a9d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" Oct 30 22:11:05.613245 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:05.594078 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz8x2\" (UniqueName: \"kubernetes.io/projected/06181307-c42a-411e-8216-6c8f4d9a4a9d-kube-api-access-kz8x2\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst\" (UID: \"06181307-c42a-411e-8216-6c8f4d9a4a9d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" Oct 30 22:11:05.665243 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:05.661318 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" podUID="05ab563b-c4c2-4fa7-85e4-423141587664" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:05.723243 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:05.700389 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" podUID="1a1ae368-0e8c-408d-bfcd-8f4a8194464f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 22:11:05.887294 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:05.886190 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" podUID="ab957b47-90d6-497a-8334-1d5d4541dc7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:05.900170 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:05.900135 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06181307-c42a-411e-8216-6c8f4d9a4a9d-istio-token\") pod \"e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst\" (UID: \"06181307-c42a-411e-8216-6c8f4d9a4a9d\") " pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" Oct 30 22:11:05.962244 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:05.936933 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" Oct 30 22:11:06.002992 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:06.002949 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" podUID="395b4d7d-4aeb-4576-b616-67638bb1a2c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 22:11:06.007675 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:06.007644 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" Oct 30 22:11:06.087250 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:06.087198 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" podUID="dc20dd97-eb75-42c5-8e50-6b0b5cb54b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 22:11:06.089343 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:06.089318 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" Oct 30 22:11:06.314438 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:06.314397 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" podUID="6536df0a-a317-4b24-aaa9-c4315d2f7dd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:06.315477 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:06.315456 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" Oct 30 22:11:06.411093 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:06.404336 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" podUID="21ef7a75-4005-4d56-9b35-783dc4d9cd2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:06.430159 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:06.430130 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" Oct 30 22:11:06.432704 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:06.432671 5918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" podUID="1a1ae368-0e8c-408d-bfcd-8f4a8194464f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:06.444954 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:06.444903 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" podUID="f5ae2434-f161-4a5c-8086-2f0711678bb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:06.464284 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:06.446182 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" Oct 30 22:11:06.713890 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:06.704911 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" podUID="05ab563b-c4c2-4fa7-85e4-423141587664" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 22:11:06.738241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:06.719404 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" Oct 30 22:11:06.738241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:06.720107 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" podUID="e63fad41-0e42-41ee-ae20-919f15fffe55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 22:11:06.744875 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:06.739878 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" Oct 30 22:11:06.761837 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:06.760752 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" podUID="1a1ae368-0e8c-408d-bfcd-8f4a8194464f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 22:11:06.767206 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:06.762106 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" Oct 30 22:11:06.888436 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:06.888326 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" podUID="ab957b47-90d6-497a-8334-1d5d4541dc7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:06.889314 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:06.889292 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" Oct 30 22:11:07.006532 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:07.005157 5918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" podUID="05ab563b-c4c2-4fa7-85e4-423141587664" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:07.008319 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:07.007695 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" podUID="395b4d7d-4aeb-4576-b616-67638bb1a2c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:07.182633 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:07.173891 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" podUID="dc20dd97-eb75-42c5-8e50-6b0b5cb54b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 22:11:07.379242 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:07.373873 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" podUID="6536df0a-a317-4b24-aaa9-c4315d2f7dd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 22:11:07.450316 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:07.450195 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" podUID="f5ae2434-f161-4a5c-8086-2f0711678bb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:07.483256 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:07.465274 5918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" podUID="ab957b47-90d6-497a-8334-1d5d4541dc7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:07.504958 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:07.504919 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" podUID="21ef7a75-4005-4d56-9b35-783dc4d9cd2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:07.727161 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:07.719348 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" podUID="05ab563b-c4c2-4fa7-85e4-423141587664" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:07.727161 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:07.725420 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" podUID="e63fad41-0e42-41ee-ae20-919f15fffe55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:07.779357 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:07.779295 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" podUID="1a1ae368-0e8c-408d-bfcd-8f4a8194464f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 22:11:07.907052 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:07.903306 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" podUID="ab957b47-90d6-497a-8334-1d5d4541dc7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:07.972517 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:07.972484 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" Oct 30 22:11:08.036461 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:08.021912 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" podUID="395b4d7d-4aeb-4576-b616-67638bb1a2c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:08.214474 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:08.200735 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" Oct 30 22:11:08.388489 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:08.388382 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" podUID="6536df0a-a317-4b24-aaa9-c4315d2f7dd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 22:11:08.450863 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:08.398673 5918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" podUID="f5ae2434-f161-4a5c-8086-2f0711678bb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:08.467504 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:08.467401 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" podUID="f5ae2434-f161-4a5c-8086-2f0711678bb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 22:11:08.727241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:08.726235 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" podUID="21ef7a75-4005-4d56-9b35-783dc4d9cd2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 22:11:08.751003 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:08.750974 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" Oct 30 22:11:08.751625 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:08.751590 5918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" podUID="e63fad41-0e42-41ee-ae20-919f15fffe55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:08.752879 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:08.752831 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" podUID="05ab563b-c4c2-4fa7-85e4-423141587664" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 22:11:08.754101 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:08.754077 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" podUID="e63fad41-0e42-41ee-ae20-919f15fffe55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:08.783437 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:08.780994 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" podUID="1a1ae368-0e8c-408d-bfcd-8f4a8194464f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:09.174343 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:09.174299 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" Oct 30 22:11:09.190878 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:09.190839 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" podUID="dc20dd97-eb75-42c5-8e50-6b0b5cb54b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:09.252850 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:09.210769 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" podUID="395b4d7d-4aeb-4576-b616-67638bb1a2c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:09.342965 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:09.342943 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" Oct 30 22:11:09.345427 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:09.345394 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:11:09.345671 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:09.345654 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:11:09.345820 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:09.345805 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:11:09.364138 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:09.364092 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" Oct 30 22:11:09.391618 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:09.390429 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" podUID="6536df0a-a317-4b24-aaa9-c4315d2f7dd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 22:11:09.406390 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:09.406295 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst"] Oct 30 22:11:09.410253 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:09.409766 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" Oct 30 22:11:09.423753 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:09.423679 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" Oct 30 22:11:09.507086 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:09.507049 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" event={"ID":"06181307-c42a-411e-8216-6c8f4d9a4a9d","Type":"ContainerStarted","Data":"1436c08eb43a76ecbc45f2d87b5fcd90286123a34b1ab36c1c3b7b577454daec"} Oct 30 22:11:09.582726 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:09.582700 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" Oct 30 22:11:10.518423 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:10.518386 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" event={"ID":"06181307-c42a-411e-8216-6c8f4d9a4a9d","Type":"ContainerStarted","Data":"fd6b13aff9e3de14e6768bb2b3a0e11c59b264ea89ebecab3f0a5506484ab353"} Oct 30 22:11:10.645883 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:10.645817 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" podStartSLOduration=6.645803476 podStartE2EDuration="6.645803476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:10.644958164 +0000 UTC m=+1230.915656247" watchObservedRunningTime="2026-10-30 22:11:10.645803476 +0000 UTC m=+1230.916501561" Oct 30 22:11:10.942441 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:10.942414 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" Oct 30 22:11:10.943796 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:10.943774 5918 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 22:11:10.943858 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:10.943825 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" podUID="06181307-c42a-411e-8216-6c8f4d9a4a9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 22:11:11.942313 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:11.942279 5918 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 22:11:11.942823 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:11.942330 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" podUID="06181307-c42a-411e-8216-6c8f4d9a4a9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 22:11:12.942488 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:12.942440 5918 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 22:11:12.942981 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:12.942496 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" podUID="06181307-c42a-411e-8216-6c8f4d9a4a9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 22:11:13.942677 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:13.942636 5918 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 22:11:13.943194 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:13.942710 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" podUID="06181307-c42a-411e-8216-6c8f4d9a4a9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 22:11:14.942550 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:14.942512 5918 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 22:11:14.942752 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:14.942598 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" podUID="06181307-c42a-411e-8216-6c8f4d9a4a9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 22:11:15.942494 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:15.942453 5918 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 22:11:15.942672 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:15.942511 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" podUID="06181307-c42a-411e-8216-6c8f4d9a4a9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 22:11:16.989070 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:16.989034 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" Oct 30 22:11:16.990051 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:16.990017 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" Oct 30 22:11:16.990831 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:16.990807 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" Oct 30 22:11:21.276794 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:11:21.276752 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:11:37.763351 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:37.761405 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" podUID="05ab563b-c4c2-4fa7-85e4-423141587664" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 22:11:37.763351 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:37.762678 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" podUID="e63fad41-0e42-41ee-ae20-919f15fffe55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:37.796676 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:37.796614 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" podUID="1a1ae368-0e8c-408d-bfcd-8f4a8194464f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:37.907269 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:37.904401 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" podUID="ab957b47-90d6-497a-8334-1d5d4541dc7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:38.201251 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:38.200385 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" podUID="dc20dd97-eb75-42c5-8e50-6b0b5cb54b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:38.305324 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:38.305265 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" podUID="395b4d7d-4aeb-4576-b616-67638bb1a2c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:38.404137 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:38.404087 5918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" podUID="f5ae2434-f161-4a5c-8086-2f0711678bb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:38.432244 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:38.406699 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" podUID="6536df0a-a317-4b24-aaa9-c4315d2f7dd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:38.432244 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:38.427074 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" podUID="f5ae2434-f161-4a5c-8086-2f0711678bb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:38.601322 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:38.580328 5918 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:11:38.601322 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:38.580392 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" podUID="a77bc09f-2593-4ed7-89fe-21fffab26adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:38.636513 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:38.636394 5918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" podUID="e63fad41-0e42-41ee-ae20-919f15fffe55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:38.733236 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:38.733182 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" podUID="21ef7a75-4005-4d56-9b35-783dc4d9cd2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:38.776242 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:38.770129 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" podUID="e63fad41-0e42-41ee-ae20-919f15fffe55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:38.814245 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:38.777560 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" podUID="05ab563b-c4c2-4fa7-85e4-423141587664" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:38.815591 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:38.791377 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" Oct 30 22:11:38.815819 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:38.815804 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" Oct 30 22:11:38.840255 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:38.822281 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" podUID="1a1ae368-0e8c-408d-bfcd-8f4a8194464f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:38.840255 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:38.822672 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" Oct 30 22:11:38.951237 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:38.934542 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" podUID="ab957b47-90d6-497a-8334-1d5d4541dc7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:39.210177 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:39.210085 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" Oct 30 22:11:39.214500 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:39.214464 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" podUID="dc20dd97-eb75-42c5-8e50-6b0b5cb54b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:39.215747 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:39.215724 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" Oct 30 22:11:39.355556 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:39.321735 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" podUID="395b4d7d-4aeb-4576-b616-67638bb1a2c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:39.355556 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:39.326190 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" Oct 30 22:11:39.424770 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:39.424725 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" podUID="6536df0a-a317-4b24-aaa9-c4315d2f7dd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:39.436511 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:39.434353 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" podUID="f5ae2434-f161-4a5c-8086-2f0711678bb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:39.436975 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:39.436952 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" Oct 30 22:11:39.454243 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:39.437134 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" Oct 30 22:11:39.473444 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:39.473352 5918 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:11:39.473714 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:39.473684 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" podUID="f83c719c-83ad-4b5e-be59-a3242cd7ecaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:39.617148 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:39.617119 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" Oct 30 22:11:39.762007 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:39.761907 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" podUID="21ef7a75-4005-4d56-9b35-783dc4d9cd2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:39.763521 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:39.763498 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" Oct 30 22:11:39.839256 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:39.834512 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" podUID="1a1ae368-0e8c-408d-bfcd-8f4a8194464f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 22:11:39.977003 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:39.976961 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" podUID="05ab563b-c4c2-4fa7-85e4-423141587664" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:39.991242 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:39.987403 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" podUID="ab957b47-90d6-497a-8334-1d5d4541dc7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 22:11:40.429239 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:40.414122 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" podUID="395b4d7d-4aeb-4576-b616-67638bb1a2c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 22:11:40.481241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:40.478180 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" podUID="f5ae2434-f161-4a5c-8086-2f0711678bb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:40.493131 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:40.493075 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" podUID="6536df0a-a317-4b24-aaa9-c4315d2f7dd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:40.522247 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:40.514945 5918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" podUID="395b4d7d-4aeb-4576-b616-67638bb1a2c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded" Oct 30 22:11:41.109071 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:40.895878 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" podUID="1a1ae368-0e8c-408d-bfcd-8f4a8194464f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:41.109071 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:40.920987 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" podUID="21ef7a75-4005-4d56-9b35-783dc4d9cd2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 22:11:41.109071 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:41.045371 5918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" podUID="dc20dd97-eb75-42c5-8e50-6b0b5cb54b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:41.109071 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:41.045509 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" podUID="dc20dd97-eb75-42c5-8e50-6b0b5cb54b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:41.159674 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:11:41.134343 5918 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.262s" Oct 30 22:11:41.159674 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:41.134526 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" Oct 30 22:11:41.159674 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:41.134587 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" Oct 30 22:11:41.159674 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:41.134653 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" Oct 30 22:11:41.159674 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:41.139024 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" Oct 30 22:11:41.336952 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:41.336553 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" podUID="ab957b47-90d6-497a-8334-1d5d4541dc7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:41.416817 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:41.406150 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" Oct 30 22:11:41.416817 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:41.406339 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" Oct 30 22:11:41.450245 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:41.445749 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" Oct 30 22:11:41.450245 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:41.449919 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" podUID="395b4d7d-4aeb-4576-b616-67638bb1a2c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 22:11:41.467889 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:41.467860 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" Oct 30 22:11:46.379359 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:46.379324 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7"] Oct 30 22:11:46.383774 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:46.383754 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7" Oct 30 22:11:46.427741 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:46.427711 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/583871c1-8b4a-4145-a350-f3413686ad38-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgvh7\" (UID: \"583871c1-8b4a-4145-a350-f3413686ad38\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7" Oct 30 22:11:46.427907 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:46.427764 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/583871c1-8b4a-4145-a350-f3413686ad38-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgvh7\" (UID: \"583871c1-8b4a-4145-a350-f3413686ad38\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7" Oct 30 22:11:46.427980 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:46.427909 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/583871c1-8b4a-4145-a350-f3413686ad38-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgvh7\" (UID: \"583871c1-8b4a-4145-a350-f3413686ad38\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7" Oct 30 22:11:46.428023 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:46.428001 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkqcd\" (UniqueName: \"kubernetes.io/projected/583871c1-8b4a-4145-a350-f3413686ad38-kube-api-access-kkqcd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgvh7\" (UID: \"583871c1-8b4a-4145-a350-f3413686ad38\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7" Oct 30 22:11:46.428074 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:46.428029 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/583871c1-8b4a-4145-a350-f3413686ad38-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgvh7\" (UID: \"583871c1-8b4a-4145-a350-f3413686ad38\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7" Oct 30 22:11:46.428130 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:46.428092 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/583871c1-8b4a-4145-a350-f3413686ad38-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgvh7\" (UID: \"583871c1-8b4a-4145-a350-f3413686ad38\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7" Oct 30 22:11:46.428356 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:46.428340 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/583871c1-8b4a-4145-a350-f3413686ad38-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgvh7\" (UID: \"583871c1-8b4a-4145-a350-f3413686ad38\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7" Oct 30 22:11:46.438425 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:46.438401 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 22:11:46.438425 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:46.438415 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 22:11:46.438577 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:46.438446 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 22:11:46.438577 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:46.438470 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6s5w5\"" Oct 30 22:11:46.491739 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:46.491709 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7"] Oct 30 22:11:46.529231 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:46.529186 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/583871c1-8b4a-4145-a350-f3413686ad38-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgvh7\" (UID: \"583871c1-8b4a-4145-a350-f3413686ad38\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7" Oct 30 22:11:46.529411 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:46.529291 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/583871c1-8b4a-4145-a350-f3413686ad38-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgvh7\" (UID: \"583871c1-8b4a-4145-a350-f3413686ad38\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7" Oct 30 22:11:46.529411 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:46.529340 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/583871c1-8b4a-4145-a350-f3413686ad38-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgvh7\" (UID: \"583871c1-8b4a-4145-a350-f3413686ad38\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7" Oct 30 22:11:46.529528 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:46.529416 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/583871c1-8b4a-4145-a350-f3413686ad38-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgvh7\" (UID: \"583871c1-8b4a-4145-a350-f3413686ad38\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7" Oct 30 22:11:46.529528 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:46.529481 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkqcd\" (UniqueName: \"kubernetes.io/projected/583871c1-8b4a-4145-a350-f3413686ad38-kube-api-access-kkqcd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgvh7\" (UID: \"583871c1-8b4a-4145-a350-f3413686ad38\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7" Oct 30 22:11:46.529528 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:46.529511 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/583871c1-8b4a-4145-a350-f3413686ad38-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgvh7\" (UID: \"583871c1-8b4a-4145-a350-f3413686ad38\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7" Oct 30 22:11:46.529686 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:46.529559 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/583871c1-8b4a-4145-a350-f3413686ad38-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgvh7\" (UID: \"583871c1-8b4a-4145-a350-f3413686ad38\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7" Oct 30 22:11:46.530127 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:46.530011 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/583871c1-8b4a-4145-a350-f3413686ad38-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgvh7\" (UID: \"583871c1-8b4a-4145-a350-f3413686ad38\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7" Oct 30 22:11:46.532391 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:46.532366 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/583871c1-8b4a-4145-a350-f3413686ad38-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgvh7\" (UID: \"583871c1-8b4a-4145-a350-f3413686ad38\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7" Oct 30 22:11:46.532663 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:46.532641 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/583871c1-8b4a-4145-a350-f3413686ad38-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgvh7\" (UID: \"583871c1-8b4a-4145-a350-f3413686ad38\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7" Oct 30 22:11:46.532742 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:46.532721 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/583871c1-8b4a-4145-a350-f3413686ad38-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgvh7\" (UID: \"583871c1-8b4a-4145-a350-f3413686ad38\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7" Oct 30 22:11:46.532857 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:46.532836 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/583871c1-8b4a-4145-a350-f3413686ad38-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgvh7\" (UID: \"583871c1-8b4a-4145-a350-f3413686ad38\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7" Oct 30 22:11:46.593178 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:46.593149 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/583871c1-8b4a-4145-a350-f3413686ad38-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgvh7\" (UID: \"583871c1-8b4a-4145-a350-f3413686ad38\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7" Oct 30 22:11:46.593360 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:46.593176 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkqcd\" (UniqueName: \"kubernetes.io/projected/583871c1-8b4a-4145-a350-f3413686ad38-kube-api-access-kkqcd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-pgvh7\" (UID: \"583871c1-8b4a-4145-a350-f3413686ad38\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7" Oct 30 22:11:46.699078 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:46.699047 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7" Oct 30 22:11:47.140785 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:47.140701 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7"] Oct 30 22:11:47.171644 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:11:47.171617 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod583871c1_8b4a_4145_a350_f3413686ad38.slice/crio-0d48b1d8ad28d68457ccbbb4eafe15fceba0d84fa4185aca8bbe7472f0534de7 WatchSource:0}: Error finding container 0d48b1d8ad28d68457ccbbb4eafe15fceba0d84fa4185aca8bbe7472f0534de7: Status 404 returned error can't find the container with id 0d48b1d8ad28d68457ccbbb4eafe15fceba0d84fa4185aca8bbe7472f0534de7 Oct 30 22:11:47.173764 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:47.173746 5918 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:11:47.462925 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:47.462866 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7" event={"ID":"583871c1-8b4a-4145-a350-f3413686ad38","Type":"ContainerStarted","Data":"0d48b1d8ad28d68457ccbbb4eafe15fceba0d84fa4185aca8bbe7472f0534de7"} Oct 30 22:11:51.189404 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:51.189371 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:11:51.189689 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:51.189436 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:11:51.276763 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:11:51.276722 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:11:51.493921 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:51.493825 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7" event={"ID":"583871c1-8b4a-4145-a350-f3413686ad38","Type":"ContainerStarted","Data":"84123bd6ca67e1721791fd49d2d218313167be4fc78df6035b024e41de75688d"} Oct 30 22:11:51.493921 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:51.493910 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7" Oct 30 22:11:51.721028 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:51.720960 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7" podStartSLOduration=1.706708861 podStartE2EDuration="5.720945628s" totalImagesPullingTime="4.014236767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:51.667924848 +0000 UTC m=+1271.938622937" watchObservedRunningTime="2026-10-30 22:11:51.720945628 +0000 UTC m=+1271.991643712" Oct 30 22:11:52.507427 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:52.507392 5918 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-pgvh7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:11:52.507908 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:52.507448 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7" podUID="583871c1-8b4a-4145-a350-f3413686ad38" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:11:54.372081 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:54.372041 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc"] Oct 30 22:11:54.372635 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:54.372343 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" podUID="74d3f33a-98ec-4b95-88d4-6e836fe59fd5" containerName="istio-proxy" containerID="cri-o://a418702ab277f4d16865f6fa931acc75dd3ce2a386637c4a38ce7d4311380f3f" gracePeriod=30 Oct 30 22:11:55.505847 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:55.505819 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7" Oct 30 22:11:59.557488 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:59.556332 5918 generic.go:356] "Generic (PLEG): container finished" podID="74d3f33a-98ec-4b95-88d4-6e836fe59fd5" containerID="a418702ab277f4d16865f6fa931acc75dd3ce2a386637c4a38ce7d4311380f3f" exitCode=0 Oct 30 22:11:59.557893 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:59.557865 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" event={"ID":"74d3f33a-98ec-4b95-88d4-6e836fe59fd5","Type":"ContainerDied","Data":"a418702ab277f4d16865f6fa931acc75dd3ce2a386637c4a38ce7d4311380f3f"} Oct 30 22:11:59.653313 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:59.653207 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" Oct 30 22:11:59.745752 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:59.745720 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-istiod-ca-cert\") pod \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\" (UID: \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\") " Oct 30 22:11:59.745911 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:59.745825 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-istio-envoy\") pod \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\" (UID: \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\") " Oct 30 22:11:59.745911 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:59.745854 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-kube-api-access-cpm57\" (UniqueName: \"kubernetes.io/projected/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-kube-api-access-cpm57\") pod \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\" (UID: \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\") " Oct 30 22:11:59.745911 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:59.745889 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-workload-certs\") pod \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\" (UID: \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\") " Oct 30 22:11:59.746048 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:59.745948 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-credential-socket\") pod \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\" (UID: \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\") " Oct 30 22:11:59.746048 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:59.745992 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-istio-data\") pod \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\" (UID: \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\") " Oct 30 22:11:59.746048 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:59.746032 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-istio-token\" (UniqueName: \"kubernetes.io/projected/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-istio-token\") pod \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\" (UID: \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\") " Oct 30 22:11:59.746207 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:59.746142 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-istiod-ca-cert" pod "74d3f33a-98ec-4b95-88d4-6e836fe59fd5" (UID: "74d3f33a-98ec-4b95-88d4-6e836fe59fd5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:59.746207 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:59.746176 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-istio-podinfo\") pod \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\" (UID: \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\") " Oct 30 22:11:59.746367 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:59.746354 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-workload-socket\") pod \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\" (UID: \"74d3f33a-98ec-4b95-88d4-6e836fe59fd5\") " Oct 30 22:11:59.746423 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:59.746370 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-workload-certs" pod "74d3f33a-98ec-4b95-88d4-6e836fe59fd5" (UID: "74d3f33a-98ec-4b95-88d4-6e836fe59fd5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:59.746423 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:59.746391 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-credential-socket" pod "74d3f33a-98ec-4b95-88d4-6e836fe59fd5" (UID: "74d3f33a-98ec-4b95-88d4-6e836fe59fd5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:59.746763 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:59.746741 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-workload-socket" pod "74d3f33a-98ec-4b95-88d4-6e836fe59fd5" (UID: "74d3f33a-98ec-4b95-88d4-6e836fe59fd5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:59.747022 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:59.746757 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-istio-data" pod "74d3f33a-98ec-4b95-88d4-6e836fe59fd5" (UID: "74d3f33a-98ec-4b95-88d4-6e836fe59fd5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:59.747383 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:59.747356 5918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-workload-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:59.747475 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:59.747388 5918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-istiod-ca-cert\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:59.747475 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:59.747403 5918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-workload-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:59.747475 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:59.747417 5918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-credential-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:59.747475 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:59.747432 5918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-istio-data\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:59.748631 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:59.748608 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-istio-token" pod "74d3f33a-98ec-4b95-88d4-6e836fe59fd5" (UID: "74d3f33a-98ec-4b95-88d4-6e836fe59fd5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:59.748787 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:59.748769 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-istio-envoy" pod "74d3f33a-98ec-4b95-88d4-6e836fe59fd5" (UID: "74d3f33a-98ec-4b95-88d4-6e836fe59fd5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:59.748973 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:59.748956 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-istio-podinfo" pod "74d3f33a-98ec-4b95-88d4-6e836fe59fd5" (UID: "74d3f33a-98ec-4b95-88d4-6e836fe59fd5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:11:59.749044 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:59.748979 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-kube-api-access-cpm57" pod "74d3f33a-98ec-4b95-88d4-6e836fe59fd5" (UID: "74d3f33a-98ec-4b95-88d4-6e836fe59fd5"). InnerVolumeSpecName "kube-api-access-cpm57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:59.848948 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:59.848895 5918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-istio-envoy\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:59.849133 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:59.848953 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpm57\" (UniqueName: \"kubernetes.io/projected/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-kube-api-access-cpm57\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:59.849133 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:59.849122 5918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-istio-token\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:59.849287 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:11:59.849149 5918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74d3f33a-98ec-4b95-88d4-6e836fe59fd5-istio-podinfo\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:59.971854 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:11:59.971823 5918 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74d3f33a_98ec_4b95_88d4_6e836fe59fd5.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74d3f33a_98ec_4b95_88d4_6e836fe59fd5.slice/crio-793ec52634ae1090a7ecf2bff251328fdc208f18036f2d2d20662520180d1da0\": RecentStats: unable to find data in memory cache]" Oct 30 22:12:00.563843 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:00.563815 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" Oct 30 22:12:00.564321 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:00.563850 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc" event={"ID":"74d3f33a-98ec-4b95-88d4-6e836fe59fd5","Type":"ContainerDied","Data":"793ec52634ae1090a7ecf2bff251328fdc208f18036f2d2d20662520180d1da0"} Oct 30 22:12:00.564321 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:00.563912 5918 scope.go:122] "RemoveContainer" containerID="a418702ab277f4d16865f6fa931acc75dd3ce2a386637c4a38ce7d4311380f3f" Oct 30 22:12:00.786140 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:00.786108 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc"] Oct 30 22:12:00.838757 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:00.838669 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-plwzc"] Oct 30 22:12:01.852192 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:01.852159 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74d3f33a-98ec-4b95-88d4-6e836fe59fd5" path="/var/lib/kubelet/pods/74d3f33a-98ec-4b95-88d4-6e836fe59fd5/volumes" Oct 30 22:12:21.275973 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:12:21.275939 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:12:24.673434 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:24.673400 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm"] Oct 30 22:12:24.674029 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:24.674014 5918 cpu_manager.go:414] "containerMap: removing container" podUID="74d3f33a-98ec-4b95-88d4-6e836fe59fd5" containerName="istio-proxy" Oct 30 22:12:24.674085 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:24.674030 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74d3f33a-98ec-4b95-88d4-6e836fe59fd5" containerName="istio-proxy" Oct 30 22:12:24.674143 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:24.674134 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="74d3f33a-98ec-4b95-88d4-6e836fe59fd5" containerName="istio-proxy" Oct 30 22:12:24.682162 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:24.682139 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" Oct 30 22:12:24.801567 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:24.801527 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ada89063-30f1-4149-bd94-688bd43ed3dc-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-ws8qm\" (UID: \"ada89063-30f1-4149-bd94-688bd43ed3dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" Oct 30 22:12:24.801740 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:24.801617 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ada89063-30f1-4149-bd94-688bd43ed3dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-ws8qm\" (UID: \"ada89063-30f1-4149-bd94-688bd43ed3dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" Oct 30 22:12:24.801740 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:24.801660 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rvmq\" (UniqueName: \"kubernetes.io/projected/ada89063-30f1-4149-bd94-688bd43ed3dc-kube-api-access-8rvmq\") pod \"maas-default-gateway-openshift-default-67bf986f99-ws8qm\" (UID: \"ada89063-30f1-4149-bd94-688bd43ed3dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" Oct 30 22:12:24.801834 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:24.801733 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ada89063-30f1-4149-bd94-688bd43ed3dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-ws8qm\" (UID: \"ada89063-30f1-4149-bd94-688bd43ed3dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" Oct 30 22:12:24.801888 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:24.801830 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada89063-30f1-4149-bd94-688bd43ed3dc-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-ws8qm\" (UID: \"ada89063-30f1-4149-bd94-688bd43ed3dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" Oct 30 22:12:24.801888 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:24.801866 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ada89063-30f1-4149-bd94-688bd43ed3dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-ws8qm\" (UID: \"ada89063-30f1-4149-bd94-688bd43ed3dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" Oct 30 22:12:24.802028 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:24.801985 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada89063-30f1-4149-bd94-688bd43ed3dc-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-ws8qm\" (UID: \"ada89063-30f1-4149-bd94-688bd43ed3dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" Oct 30 22:12:24.802148 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:24.802072 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ada89063-30f1-4149-bd94-688bd43ed3dc-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-ws8qm\" (UID: \"ada89063-30f1-4149-bd94-688bd43ed3dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" Oct 30 22:12:24.802148 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:24.802100 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ada89063-30f1-4149-bd94-688bd43ed3dc-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-ws8qm\" (UID: \"ada89063-30f1-4149-bd94-688bd43ed3dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" Oct 30 22:12:24.802662 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:24.802641 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm"] Oct 30 22:12:24.903690 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:24.903657 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rvmq\" (UniqueName: \"kubernetes.io/projected/ada89063-30f1-4149-bd94-688bd43ed3dc-kube-api-access-8rvmq\") pod \"maas-default-gateway-openshift-default-67bf986f99-ws8qm\" (UID: \"ada89063-30f1-4149-bd94-688bd43ed3dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" Oct 30 22:12:24.903877 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:24.903708 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ada89063-30f1-4149-bd94-688bd43ed3dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-ws8qm\" (UID: \"ada89063-30f1-4149-bd94-688bd43ed3dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" Oct 30 22:12:24.903877 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:24.903760 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada89063-30f1-4149-bd94-688bd43ed3dc-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-ws8qm\" (UID: \"ada89063-30f1-4149-bd94-688bd43ed3dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" Oct 30 22:12:24.903877 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:24.903786 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ada89063-30f1-4149-bd94-688bd43ed3dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-ws8qm\" (UID: \"ada89063-30f1-4149-bd94-688bd43ed3dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" Oct 30 22:12:24.903877 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:24.903823 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada89063-30f1-4149-bd94-688bd43ed3dc-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-ws8qm\" (UID: \"ada89063-30f1-4149-bd94-688bd43ed3dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" Oct 30 22:12:24.904123 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:24.903918 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ada89063-30f1-4149-bd94-688bd43ed3dc-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-ws8qm\" (UID: \"ada89063-30f1-4149-bd94-688bd43ed3dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" Oct 30 22:12:24.904123 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:24.903944 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ada89063-30f1-4149-bd94-688bd43ed3dc-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-ws8qm\" (UID: \"ada89063-30f1-4149-bd94-688bd43ed3dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" Oct 30 22:12:24.904271 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:24.904122 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ada89063-30f1-4149-bd94-688bd43ed3dc-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-ws8qm\" (UID: \"ada89063-30f1-4149-bd94-688bd43ed3dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" Oct 30 22:12:24.904271 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:24.904187 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ada89063-30f1-4149-bd94-688bd43ed3dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-ws8qm\" (UID: \"ada89063-30f1-4149-bd94-688bd43ed3dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" Oct 30 22:12:24.904271 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:24.904202 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada89063-30f1-4149-bd94-688bd43ed3dc-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-ws8qm\" (UID: \"ada89063-30f1-4149-bd94-688bd43ed3dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" Oct 30 22:12:24.904271 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:24.904245 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada89063-30f1-4149-bd94-688bd43ed3dc-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-ws8qm\" (UID: \"ada89063-30f1-4149-bd94-688bd43ed3dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" Oct 30 22:12:24.904498 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:24.904303 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ada89063-30f1-4149-bd94-688bd43ed3dc-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-ws8qm\" (UID: \"ada89063-30f1-4149-bd94-688bd43ed3dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" Oct 30 22:12:24.904568 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:24.904550 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ada89063-30f1-4149-bd94-688bd43ed3dc-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-ws8qm\" (UID: \"ada89063-30f1-4149-bd94-688bd43ed3dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" Oct 30 22:12:24.905075 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:24.905036 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ada89063-30f1-4149-bd94-688bd43ed3dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-ws8qm\" (UID: \"ada89063-30f1-4149-bd94-688bd43ed3dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" Oct 30 22:12:24.906559 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:24.906535 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ada89063-30f1-4149-bd94-688bd43ed3dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-ws8qm\" (UID: \"ada89063-30f1-4149-bd94-688bd43ed3dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" Oct 30 22:12:24.906727 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:24.906711 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ada89063-30f1-4149-bd94-688bd43ed3dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-ws8qm\" (UID: \"ada89063-30f1-4149-bd94-688bd43ed3dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" Oct 30 22:12:24.969817 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:24.969788 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ada89063-30f1-4149-bd94-688bd43ed3dc-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-ws8qm\" (UID: \"ada89063-30f1-4149-bd94-688bd43ed3dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" Oct 30 22:12:24.969968 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:24.969948 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rvmq\" (UniqueName: \"kubernetes.io/projected/ada89063-30f1-4149-bd94-688bd43ed3dc-kube-api-access-8rvmq\") pod \"maas-default-gateway-openshift-default-67bf986f99-ws8qm\" (UID: \"ada89063-30f1-4149-bd94-688bd43ed3dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" Oct 30 22:12:24.996361 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:24.996341 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" Oct 30 22:12:25.433771 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:25.433744 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm"] Oct 30 22:12:25.434841 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:12:25.434809 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podada89063_30f1_4149_bd94_688bd43ed3dc.slice/crio-ab8790630fe6c13a9f0e4a49844bab1f0c48b3541fbe530f349c86c077282406 WatchSource:0}: Error finding container ab8790630fe6c13a9f0e4a49844bab1f0c48b3541fbe530f349c86c077282406: Status 404 returned error can't find the container with id ab8790630fe6c13a9f0e4a49844bab1f0c48b3541fbe530f349c86c077282406 Oct 30 22:12:25.437379 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:25.437341 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:12:25.437492 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:25.437422 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:12:25.437492 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:25.437468 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:12:25.734562 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:25.734479 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" event={"ID":"ada89063-30f1-4149-bd94-688bd43ed3dc","Type":"ContainerStarted","Data":"9f232437abd50869fae45c238591fb400ef615409e862d5f03ce1b1568f23ce5"} Oct 30 22:12:25.734562 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:25.734528 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" event={"ID":"ada89063-30f1-4149-bd94-688bd43ed3dc","Type":"ContainerStarted","Data":"ab8790630fe6c13a9f0e4a49844bab1f0c48b3541fbe530f349c86c077282406"} Oct 30 22:12:25.860938 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:25.860890 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" podStartSLOduration=1.86087607 podStartE2EDuration="1.86087607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:25.859966451 +0000 UTC m=+1306.130664536" watchObservedRunningTime="2026-10-30 22:12:25.86087607 +0000 UTC m=+1306.131574153" Oct 30 22:12:25.997036 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:25.996951 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" Oct 30 22:12:27.013406 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:27.013370 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-ws8qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 22:12:27.014015 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:27.013435 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" podUID="ada89063-30f1-4149-bd94-688bd43ed3dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 22:12:28.000987 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:28.000558 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-ws8qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 22:12:28.000987 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:28.000619 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" podUID="ada89063-30f1-4149-bd94-688bd43ed3dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 22:12:29.007240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:29.006612 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-ws8qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 22:12:29.007240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:29.006671 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" podUID="ada89063-30f1-4149-bd94-688bd43ed3dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 22:12:30.014501 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:30.014464 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-ws8qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 22:12:30.015080 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:30.015049 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" podUID="ada89063-30f1-4149-bd94-688bd43ed3dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 22:12:30.998974 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:30.998937 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-ws8qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 22:12:30.999269 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:30.999237 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" podUID="ada89063-30f1-4149-bd94-688bd43ed3dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 22:12:31.465245 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:31.464652 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" podUID="21ef7a75-4005-4d56-9b35-783dc4d9cd2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:12:31.471505 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:31.471468 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" podUID="395b4d7d-4aeb-4576-b616-67638bb1a2c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:12:31.627617 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:31.627568 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" podUID="e63fad41-0e42-41ee-ae20-919f15fffe55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 22:12:31.852494 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:31.846846 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" podUID="1a1ae368-0e8c-408d-bfcd-8f4a8194464f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:12:32.003454 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:32.000050 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" podUID="05ab563b-c4c2-4fa7-85e4-423141587664" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:12:32.054239 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:32.038555 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-ws8qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 22:12:32.054239 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:32.038626 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" podUID="ada89063-30f1-4149-bd94-688bd43ed3dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 22:12:32.997057 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:32.997020 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-ws8qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 22:12:32.997658 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:32.997098 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" podUID="ada89063-30f1-4149-bd94-688bd43ed3dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 22:12:34.713596 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:34.713562 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" Oct 30 22:12:34.714685 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:34.714023 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" Oct 30 22:12:34.715123 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:34.715104 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" Oct 30 22:12:39.854662 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:39.854630 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk"] Oct 30 22:12:39.858697 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:39.858677 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" Oct 30 22:12:39.956940 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:39.956911 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk"] Oct 30 22:12:39.965562 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:39.965536 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a3edcb-2c75-49eb-a134-a5d98f105534-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-sflfk\" (UID: \"44a3edcb-2c75-49eb-a134-a5d98f105534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" Oct 30 22:12:39.965730 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:39.965613 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a3edcb-2c75-49eb-a134-a5d98f105534-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-sflfk\" (UID: \"44a3edcb-2c75-49eb-a134-a5d98f105534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" Oct 30 22:12:39.965730 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:39.965638 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a3edcb-2c75-49eb-a134-a5d98f105534-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-sflfk\" (UID: \"44a3edcb-2c75-49eb-a134-a5d98f105534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" Oct 30 22:12:39.965849 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:39.965760 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2chv\" (UniqueName: \"kubernetes.io/projected/44a3edcb-2c75-49eb-a134-a5d98f105534-kube-api-access-h2chv\") pod \"maas-default-gateway-openshift-default-67bf986f99-sflfk\" (UID: \"44a3edcb-2c75-49eb-a134-a5d98f105534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" Oct 30 22:12:39.965849 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:39.965837 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a3edcb-2c75-49eb-a134-a5d98f105534-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-sflfk\" (UID: \"44a3edcb-2c75-49eb-a134-a5d98f105534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" Oct 30 22:12:39.965961 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:39.965874 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a3edcb-2c75-49eb-a134-a5d98f105534-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-sflfk\" (UID: \"44a3edcb-2c75-49eb-a134-a5d98f105534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" Oct 30 22:12:39.966060 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:39.966025 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a3edcb-2c75-49eb-a134-a5d98f105534-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-sflfk\" (UID: \"44a3edcb-2c75-49eb-a134-a5d98f105534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" Oct 30 22:12:39.966197 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:39.966084 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a3edcb-2c75-49eb-a134-a5d98f105534-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-sflfk\" (UID: \"44a3edcb-2c75-49eb-a134-a5d98f105534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" Oct 30 22:12:39.966311 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:39.966197 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a3edcb-2c75-49eb-a134-a5d98f105534-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-sflfk\" (UID: \"44a3edcb-2c75-49eb-a134-a5d98f105534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" Oct 30 22:12:40.067999 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:40.067968 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a3edcb-2c75-49eb-a134-a5d98f105534-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-sflfk\" (UID: \"44a3edcb-2c75-49eb-a134-a5d98f105534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" Oct 30 22:12:40.068253 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:40.068010 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a3edcb-2c75-49eb-a134-a5d98f105534-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-sflfk\" (UID: \"44a3edcb-2c75-49eb-a134-a5d98f105534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" Oct 30 22:12:40.068253 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:40.068072 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a3edcb-2c75-49eb-a134-a5d98f105534-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-sflfk\" (UID: \"44a3edcb-2c75-49eb-a134-a5d98f105534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" Oct 30 22:12:40.068253 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:40.068097 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a3edcb-2c75-49eb-a134-a5d98f105534-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-sflfk\" (UID: \"44a3edcb-2c75-49eb-a134-a5d98f105534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" Oct 30 22:12:40.068253 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:40.068128 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a3edcb-2c75-49eb-a134-a5d98f105534-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-sflfk\" (UID: \"44a3edcb-2c75-49eb-a134-a5d98f105534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" Oct 30 22:12:40.068547 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:40.068276 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a3edcb-2c75-49eb-a134-a5d98f105534-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-sflfk\" (UID: \"44a3edcb-2c75-49eb-a134-a5d98f105534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" Oct 30 22:12:40.068547 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:40.068373 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a3edcb-2c75-49eb-a134-a5d98f105534-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-sflfk\" (UID: \"44a3edcb-2c75-49eb-a134-a5d98f105534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" Oct 30 22:12:40.068547 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:40.068402 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a3edcb-2c75-49eb-a134-a5d98f105534-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-sflfk\" (UID: \"44a3edcb-2c75-49eb-a134-a5d98f105534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" Oct 30 22:12:40.068547 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:40.068409 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a3edcb-2c75-49eb-a134-a5d98f105534-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-sflfk\" (UID: \"44a3edcb-2c75-49eb-a134-a5d98f105534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" Oct 30 22:12:40.068764 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:40.068546 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2chv\" (UniqueName: \"kubernetes.io/projected/44a3edcb-2c75-49eb-a134-a5d98f105534-kube-api-access-h2chv\") pod \"maas-default-gateway-openshift-default-67bf986f99-sflfk\" (UID: \"44a3edcb-2c75-49eb-a134-a5d98f105534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" Oct 30 22:12:40.068764 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:40.068482 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a3edcb-2c75-49eb-a134-a5d98f105534-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-sflfk\" (UID: \"44a3edcb-2c75-49eb-a134-a5d98f105534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" Oct 30 22:12:40.068764 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:40.068596 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a3edcb-2c75-49eb-a134-a5d98f105534-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-sflfk\" (UID: \"44a3edcb-2c75-49eb-a134-a5d98f105534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" Oct 30 22:12:40.068764 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:40.068696 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a3edcb-2c75-49eb-a134-a5d98f105534-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-sflfk\" (UID: \"44a3edcb-2c75-49eb-a134-a5d98f105534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" Oct 30 22:12:40.069149 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:40.069125 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a3edcb-2c75-49eb-a134-a5d98f105534-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-sflfk\" (UID: \"44a3edcb-2c75-49eb-a134-a5d98f105534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" Oct 30 22:12:40.070801 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:40.070777 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a3edcb-2c75-49eb-a134-a5d98f105534-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-sflfk\" (UID: \"44a3edcb-2c75-49eb-a134-a5d98f105534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" Oct 30 22:12:40.070878 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:40.070816 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a3edcb-2c75-49eb-a134-a5d98f105534-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-sflfk\" (UID: \"44a3edcb-2c75-49eb-a134-a5d98f105534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" Oct 30 22:12:40.132024 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:40.131957 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a3edcb-2c75-49eb-a134-a5d98f105534-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-sflfk\" (UID: \"44a3edcb-2c75-49eb-a134-a5d98f105534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" Oct 30 22:12:40.132192 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:40.132167 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2chv\" (UniqueName: \"kubernetes.io/projected/44a3edcb-2c75-49eb-a134-a5d98f105534-kube-api-access-h2chv\") pod \"maas-default-gateway-openshift-default-67bf986f99-sflfk\" (UID: \"44a3edcb-2c75-49eb-a134-a5d98f105534\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" Oct 30 22:12:40.173605 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:40.173577 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" Oct 30 22:12:40.606605 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:40.606567 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk"] Oct 30 22:12:40.611166 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:40.611128 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:12:40.611308 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:40.611198 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:12:40.611308 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:40.611247 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:12:40.934759 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:40.934728 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" event={"ID":"44a3edcb-2c75-49eb-a134-a5d98f105534","Type":"ContainerStarted","Data":"5637de566e41b2ffe03f1e87e514f794fb8873b51e321d47f56d1de6efe58635"} Oct 30 22:12:40.935157 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:40.934789 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" event={"ID":"44a3edcb-2c75-49eb-a134-a5d98f105534","Type":"ContainerStarted","Data":"d04c5d827bf00a93ca76027b303efdd8729ddc581fa91669342e286980c9ce1e"} Oct 30 22:12:41.074736 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:41.074677 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" podStartSLOduration=2.074658114 podStartE2EDuration="2.074658114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:41.074374685 +0000 UTC m=+1321.345072782" watchObservedRunningTime="2026-10-30 22:12:41.074658114 +0000 UTC m=+1321.345356199" Oct 30 22:12:41.173703 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:41.173670 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" Oct 30 22:12:41.175114 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:41.175093 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-sflfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 22:12:41.175178 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:41.175141 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" podUID="44a3edcb-2c75-49eb-a134-a5d98f105534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 22:12:42.174440 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:42.174400 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-sflfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 22:12:42.174927 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:42.174461 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" podUID="44a3edcb-2c75-49eb-a134-a5d98f105534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 22:12:43.174487 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:43.174451 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-sflfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 22:12:43.174891 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:43.174526 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" podUID="44a3edcb-2c75-49eb-a134-a5d98f105534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 22:12:44.174610 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:44.174566 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-sflfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 22:12:44.175072 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:44.174633 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" podUID="44a3edcb-2c75-49eb-a134-a5d98f105534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 22:12:45.174557 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:45.174518 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-sflfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 22:12:45.174752 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:45.174580 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" podUID="44a3edcb-2c75-49eb-a134-a5d98f105534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 22:12:46.174629 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:46.174595 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-sflfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 22:12:46.174827 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:46.174684 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" podUID="44a3edcb-2c75-49eb-a134-a5d98f105534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 22:12:47.174583 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:47.174552 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-sflfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 22:12:47.174776 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:47.174609 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" podUID="44a3edcb-2c75-49eb-a134-a5d98f105534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 22:12:48.173984 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:48.173939 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-sflfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 22:12:48.174529 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:48.174020 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" podUID="44a3edcb-2c75-49eb-a134-a5d98f105534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 22:12:49.244657 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:49.244625 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" Oct 30 22:12:49.245847 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:49.245015 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" Oct 30 22:12:49.246470 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:12:49.246451 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" Oct 30 22:12:51.276757 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:12:51.276720 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:13:21.274761 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:13:21.274678 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:13:51.280000 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:13:51.279952 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:14:21.283264 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:14:21.283207 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:14:51.277132 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:14:51.277041 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:15:21.275822 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:15:21.275782 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:15:35.737053 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:35.737020 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf"] Oct 30 22:15:35.737645 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:35.737282 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" podUID="f5489ae6-f93c-4bcf-955d-54637b0399e3" containerName="istio-proxy" containerID="cri-o://ef1c4ffe9b50423a97b75be3a17c1df34e3e39103d88409226705d49c5028320" gracePeriod=30 Oct 30 22:15:41.027678 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:41.027652 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" Oct 30 22:15:41.047667 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:41.047638 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5489ae6-f93c-4bcf-955d-54637b0399e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5489ae6-f93c-4bcf-955d-54637b0399e3-workload-socket\") pod \"f5489ae6-f93c-4bcf-955d-54637b0399e3\" (UID: \"f5489ae6-f93c-4bcf-955d-54637b0399e3\") " Oct 30 22:15:41.047829 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:41.047722 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5489ae6-f93c-4bcf-955d-54637b0399e3-kube-api-access-7rhtz\" (UniqueName: \"kubernetes.io/projected/f5489ae6-f93c-4bcf-955d-54637b0399e3-kube-api-access-7rhtz\") pod \"f5489ae6-f93c-4bcf-955d-54637b0399e3\" (UID: \"f5489ae6-f93c-4bcf-955d-54637b0399e3\") " Oct 30 22:15:41.047829 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:41.047754 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5489ae6-f93c-4bcf-955d-54637b0399e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5489ae6-f93c-4bcf-955d-54637b0399e3-credential-socket\") pod \"f5489ae6-f93c-4bcf-955d-54637b0399e3\" (UID: \"f5489ae6-f93c-4bcf-955d-54637b0399e3\") " Oct 30 22:15:41.047829 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:41.047780 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5489ae6-f93c-4bcf-955d-54637b0399e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5489ae6-f93c-4bcf-955d-54637b0399e3-istiod-ca-cert\") pod \"f5489ae6-f93c-4bcf-955d-54637b0399e3\" (UID: \"f5489ae6-f93c-4bcf-955d-54637b0399e3\") " Oct 30 22:15:41.048000 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:41.047897 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5489ae6-f93c-4bcf-955d-54637b0399e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5489ae6-f93c-4bcf-955d-54637b0399e3-workload-certs\") pod \"f5489ae6-f93c-4bcf-955d-54637b0399e3\" (UID: \"f5489ae6-f93c-4bcf-955d-54637b0399e3\") " Oct 30 22:15:41.048000 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:41.047937 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5489ae6-f93c-4bcf-955d-54637b0399e3-workload-socket" pod "f5489ae6-f93c-4bcf-955d-54637b0399e3" (UID: "f5489ae6-f93c-4bcf-955d-54637b0399e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:41.048000 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:41.047969 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5489ae6-f93c-4bcf-955d-54637b0399e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5489ae6-f93c-4bcf-955d-54637b0399e3-istio-envoy\") pod \"f5489ae6-f93c-4bcf-955d-54637b0399e3\" (UID: \"f5489ae6-f93c-4bcf-955d-54637b0399e3\") " Oct 30 22:15:41.048169 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:41.048020 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5489ae6-f93c-4bcf-955d-54637b0399e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5489ae6-f93c-4bcf-955d-54637b0399e3-istio-podinfo\") pod \"f5489ae6-f93c-4bcf-955d-54637b0399e3\" (UID: \"f5489ae6-f93c-4bcf-955d-54637b0399e3\") " Oct 30 22:15:41.048169 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:41.048059 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5489ae6-f93c-4bcf-955d-54637b0399e3-credential-socket" pod "f5489ae6-f93c-4bcf-955d-54637b0399e3" (UID: "f5489ae6-f93c-4bcf-955d-54637b0399e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:41.048169 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:41.048092 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5489ae6-f93c-4bcf-955d-54637b0399e3-istio-token\" (UniqueName: \"kubernetes.io/projected/f5489ae6-f93c-4bcf-955d-54637b0399e3-istio-token\") pod \"f5489ae6-f93c-4bcf-955d-54637b0399e3\" (UID: \"f5489ae6-f93c-4bcf-955d-54637b0399e3\") " Oct 30 22:15:41.048169 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:41.048124 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5489ae6-f93c-4bcf-955d-54637b0399e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5489ae6-f93c-4bcf-955d-54637b0399e3-istio-data\") pod \"f5489ae6-f93c-4bcf-955d-54637b0399e3\" (UID: \"f5489ae6-f93c-4bcf-955d-54637b0399e3\") " Oct 30 22:15:41.048169 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:41.048130 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5489ae6-f93c-4bcf-955d-54637b0399e3-workload-certs" pod "f5489ae6-f93c-4bcf-955d-54637b0399e3" (UID: "f5489ae6-f93c-4bcf-955d-54637b0399e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:41.048436 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:41.048236 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5489ae6-f93c-4bcf-955d-54637b0399e3-istiod-ca-cert" pod "f5489ae6-f93c-4bcf-955d-54637b0399e3" (UID: "f5489ae6-f93c-4bcf-955d-54637b0399e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:41.048436 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:41.048399 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5489ae6-f93c-4bcf-955d-54637b0399e3-istio-data" pod "f5489ae6-f93c-4bcf-955d-54637b0399e3" (UID: "f5489ae6-f93c-4bcf-955d-54637b0399e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:41.049126 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:41.049060 5918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5489ae6-f93c-4bcf-955d-54637b0399e3-istio-data\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:41.049126 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:41.049088 5918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5489ae6-f93c-4bcf-955d-54637b0399e3-workload-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:41.049126 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:41.049107 5918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5489ae6-f93c-4bcf-955d-54637b0399e3-credential-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:41.049126 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:41.049122 5918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5489ae6-f93c-4bcf-955d-54637b0399e3-istiod-ca-cert\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:41.049126 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:41.049137 5918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5489ae6-f93c-4bcf-955d-54637b0399e3-workload-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:41.050155 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:41.050131 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5489ae6-f93c-4bcf-955d-54637b0399e3-kube-api-access-7rhtz" pod "f5489ae6-f93c-4bcf-955d-54637b0399e3" (UID: "f5489ae6-f93c-4bcf-955d-54637b0399e3"). InnerVolumeSpecName "kube-api-access-7rhtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:41.050317 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:41.050298 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5489ae6-f93c-4bcf-955d-54637b0399e3-istio-envoy" pod "f5489ae6-f93c-4bcf-955d-54637b0399e3" (UID: "f5489ae6-f93c-4bcf-955d-54637b0399e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:41.050424 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:41.050405 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5489ae6-f93c-4bcf-955d-54637b0399e3-istio-podinfo" pod "f5489ae6-f93c-4bcf-955d-54637b0399e3" (UID: "f5489ae6-f93c-4bcf-955d-54637b0399e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:15:41.050635 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:41.050620 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5489ae6-f93c-4bcf-955d-54637b0399e3-istio-token" pod "f5489ae6-f93c-4bcf-955d-54637b0399e3" (UID: "f5489ae6-f93c-4bcf-955d-54637b0399e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:41.151066 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:41.151032 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rhtz\" (UniqueName: \"kubernetes.io/projected/f5489ae6-f93c-4bcf-955d-54637b0399e3-kube-api-access-7rhtz\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:41.151066 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:41.151066 5918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5489ae6-f93c-4bcf-955d-54637b0399e3-istio-envoy\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:41.151306 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:41.151079 5918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5489ae6-f93c-4bcf-955d-54637b0399e3-istio-podinfo\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:41.151306 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:41.151091 5918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5489ae6-f93c-4bcf-955d-54637b0399e3-istio-token\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:41.467885 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:41.467852 5918 generic.go:356] "Generic (PLEG): container finished" podID="f5489ae6-f93c-4bcf-955d-54637b0399e3" containerID="ef1c4ffe9b50423a97b75be3a17c1df34e3e39103d88409226705d49c5028320" exitCode=0 Oct 30 22:15:41.468058 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:41.467887 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" event={"ID":"f5489ae6-f93c-4bcf-955d-54637b0399e3","Type":"ContainerDied","Data":"ef1c4ffe9b50423a97b75be3a17c1df34e3e39103d88409226705d49c5028320"} Oct 30 22:15:41.468058 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:41.467910 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" Oct 30 22:15:41.468058 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:41.467955 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf" event={"ID":"f5489ae6-f93c-4bcf-955d-54637b0399e3","Type":"ContainerDied","Data":"530ba836e0e4d24af10b355cd5810e51e994e5a71f5dc8e1085c66234d681886"} Oct 30 22:15:41.468058 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:41.467986 5918 scope.go:122] "RemoveContainer" containerID="ef1c4ffe9b50423a97b75be3a17c1df34e3e39103d88409226705d49c5028320" Oct 30 22:15:41.482533 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:41.482510 5918 scope.go:122] "RemoveContainer" containerID="ef1c4ffe9b50423a97b75be3a17c1df34e3e39103d88409226705d49c5028320" Oct 30 22:15:41.482852 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:15:41.482831 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef1c4ffe9b50423a97b75be3a17c1df34e3e39103d88409226705d49c5028320\": container with ID starting with ef1c4ffe9b50423a97b75be3a17c1df34e3e39103d88409226705d49c5028320 not found: ID does not exist" containerID="ef1c4ffe9b50423a97b75be3a17c1df34e3e39103d88409226705d49c5028320" Oct 30 22:15:41.482927 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:41.482864 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef1c4ffe9b50423a97b75be3a17c1df34e3e39103d88409226705d49c5028320"} err="failed to get container status \"ef1c4ffe9b50423a97b75be3a17c1df34e3e39103d88409226705d49c5028320\": rpc error: code = NotFound desc = could not find container \"ef1c4ffe9b50423a97b75be3a17c1df34e3e39103d88409226705d49c5028320\": container with ID starting with ef1c4ffe9b50423a97b75be3a17c1df34e3e39103d88409226705d49c5028320 not found: ID does not exist" Oct 30 22:15:41.689574 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:41.689539 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf"] Oct 30 22:15:41.791068 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:41.790981 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8ca52a-openshift-default-6fb94bf447-t97qf"] Oct 30 22:15:41.852659 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:15:41.852625 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5489ae6-f93c-4bcf-955d-54637b0399e3" path="/var/lib/kubelet/pods/f5489ae6-f93c-4bcf-955d-54637b0399e3/volumes" Oct 30 22:15:51.275786 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:15:51.275749 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:16:18.484663 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.484631 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt"] Oct 30 22:16:18.487233 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.485276 5918 cpu_manager.go:414] "containerMap: removing container" podUID="f5489ae6-f93c-4bcf-955d-54637b0399e3" containerName="istio-proxy" Oct 30 22:16:18.487233 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.485288 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5489ae6-f93c-4bcf-955d-54637b0399e3" containerName="istio-proxy" Oct 30 22:16:18.487233 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.485385 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5489ae6-f93c-4bcf-955d-54637b0399e3" containerName="istio-proxy" Oct 30 22:16:18.488678 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.488664 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" Oct 30 22:16:18.534462 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.534430 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l"] Oct 30 22:16:18.538764 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.538747 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" Oct 30 22:16:18.565000 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.564977 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8d9b\" (UniqueName: \"kubernetes.io/projected/44ed0322-9b5b-4751-8f7f-5f0c58493345-kube-api-access-k8d9b\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt\" (UID: \"44ed0322-9b5b-4751-8f7f-5f0c58493345\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" Oct 30 22:16:18.565123 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.565009 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ed0322-9b5b-4751-8f7f-5f0c58493345-credential-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt\" (UID: \"44ed0322-9b5b-4751-8f7f-5f0c58493345\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" Oct 30 22:16:18.565164 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.565129 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ed0322-9b5b-4751-8f7f-5f0c58493345-workload-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt\" (UID: \"44ed0322-9b5b-4751-8f7f-5f0c58493345\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" Oct 30 22:16:18.565347 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.565329 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44ed0322-9b5b-4751-8f7f-5f0c58493345-istio-envoy\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt\" (UID: \"44ed0322-9b5b-4751-8f7f-5f0c58493345\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" Oct 30 22:16:18.565407 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.565393 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44ed0322-9b5b-4751-8f7f-5f0c58493345-workload-certs\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt\" (UID: \"44ed0322-9b5b-4751-8f7f-5f0c58493345\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" Oct 30 22:16:18.565456 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.565444 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44ed0322-9b5b-4751-8f7f-5f0c58493345-istiod-ca-cert\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt\" (UID: \"44ed0322-9b5b-4751-8f7f-5f0c58493345\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" Oct 30 22:16:18.565520 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.565504 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44ed0322-9b5b-4751-8f7f-5f0c58493345-istio-token\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt\" (UID: \"44ed0322-9b5b-4751-8f7f-5f0c58493345\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" Oct 30 22:16:18.565564 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.565552 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44ed0322-9b5b-4751-8f7f-5f0c58493345-istio-data\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt\" (UID: \"44ed0322-9b5b-4751-8f7f-5f0c58493345\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" Oct 30 22:16:18.565636 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.565624 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44ed0322-9b5b-4751-8f7f-5f0c58493345-istio-podinfo\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt\" (UID: \"44ed0322-9b5b-4751-8f7f-5f0c58493345\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" Oct 30 22:16:18.581625 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.581609 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3ef8140e-openshift-default-dockercfg-24rvz\"" Oct 30 22:16:18.600034 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.600007 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt"] Oct 30 22:16:18.668014 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.667974 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg4st\" (UniqueName: \"kubernetes.io/projected/0b4e06dd-69d5-485e-acac-62dddf7f51cb-kube-api-access-mg4st\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l\" (UID: \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" Oct 30 22:16:18.668271 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.668051 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4e06dd-69d5-485e-acac-62dddf7f51cb-credential-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l\" (UID: \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" Oct 30 22:16:18.668271 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.668175 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44ed0322-9b5b-4751-8f7f-5f0c58493345-istio-envoy\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt\" (UID: \"44ed0322-9b5b-4751-8f7f-5f0c58493345\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" Oct 30 22:16:18.668456 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.668317 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b4e06dd-69d5-485e-acac-62dddf7f51cb-workload-certs\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l\" (UID: \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" Oct 30 22:16:18.668456 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.668351 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44ed0322-9b5b-4751-8f7f-5f0c58493345-workload-certs\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt\" (UID: \"44ed0322-9b5b-4751-8f7f-5f0c58493345\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" Oct 30 22:16:18.668456 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.668421 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44ed0322-9b5b-4751-8f7f-5f0c58493345-istiod-ca-cert\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt\" (UID: \"44ed0322-9b5b-4751-8f7f-5f0c58493345\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" Oct 30 22:16:18.668616 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.668451 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b4e06dd-69d5-485e-acac-62dddf7f51cb-istio-token\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l\" (UID: \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" Oct 30 22:16:18.668616 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.668481 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b4e06dd-69d5-485e-acac-62dddf7f51cb-istio-envoy\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l\" (UID: \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" Oct 30 22:16:18.668616 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.668510 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44ed0322-9b5b-4751-8f7f-5f0c58493345-istio-token\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt\" (UID: \"44ed0322-9b5b-4751-8f7f-5f0c58493345\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" Oct 30 22:16:18.668616 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.668573 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44ed0322-9b5b-4751-8f7f-5f0c58493345-istio-data\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt\" (UID: \"44ed0322-9b5b-4751-8f7f-5f0c58493345\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" Oct 30 22:16:18.668823 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.668625 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44ed0322-9b5b-4751-8f7f-5f0c58493345-istio-podinfo\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt\" (UID: \"44ed0322-9b5b-4751-8f7f-5f0c58493345\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" Oct 30 22:16:18.668823 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.668651 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b4e06dd-69d5-485e-acac-62dddf7f51cb-istio-podinfo\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l\" (UID: \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" Oct 30 22:16:18.668823 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.668705 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8d9b\" (UniqueName: \"kubernetes.io/projected/44ed0322-9b5b-4751-8f7f-5f0c58493345-kube-api-access-k8d9b\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt\" (UID: \"44ed0322-9b5b-4751-8f7f-5f0c58493345\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" Oct 30 22:16:18.668823 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.668732 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ed0322-9b5b-4751-8f7f-5f0c58493345-credential-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt\" (UID: \"44ed0322-9b5b-4751-8f7f-5f0c58493345\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" Oct 30 22:16:18.668823 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.668814 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4e06dd-69d5-485e-acac-62dddf7f51cb-workload-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l\" (UID: \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" Oct 30 22:16:18.669088 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.668837 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b4e06dd-69d5-485e-acac-62dddf7f51cb-istio-data\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l\" (UID: \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" Oct 30 22:16:18.669088 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.668886 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ed0322-9b5b-4751-8f7f-5f0c58493345-workload-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt\" (UID: \"44ed0322-9b5b-4751-8f7f-5f0c58493345\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" Oct 30 22:16:18.669088 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.668908 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b4e06dd-69d5-485e-acac-62dddf7f51cb-istiod-ca-cert\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l\" (UID: \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" Oct 30 22:16:18.670066 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.670015 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44ed0322-9b5b-4751-8f7f-5f0c58493345-istio-data\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt\" (UID: \"44ed0322-9b5b-4751-8f7f-5f0c58493345\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" Oct 30 22:16:18.670663 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.670617 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44ed0322-9b5b-4751-8f7f-5f0c58493345-istiod-ca-cert\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt\" (UID: \"44ed0322-9b5b-4751-8f7f-5f0c58493345\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" Oct 30 22:16:18.670784 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.670745 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ed0322-9b5b-4751-8f7f-5f0c58493345-workload-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt\" (UID: \"44ed0322-9b5b-4751-8f7f-5f0c58493345\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" Oct 30 22:16:18.670855 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.670778 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44ed0322-9b5b-4751-8f7f-5f0c58493345-workload-certs\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt\" (UID: \"44ed0322-9b5b-4751-8f7f-5f0c58493345\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" Oct 30 22:16:18.671061 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.671038 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ed0322-9b5b-4751-8f7f-5f0c58493345-credential-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt\" (UID: \"44ed0322-9b5b-4751-8f7f-5f0c58493345\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" Oct 30 22:16:18.672686 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.671986 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44ed0322-9b5b-4751-8f7f-5f0c58493345-istio-envoy\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt\" (UID: \"44ed0322-9b5b-4751-8f7f-5f0c58493345\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" Oct 30 22:16:18.673696 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.673670 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44ed0322-9b5b-4751-8f7f-5f0c58493345-istio-podinfo\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt\" (UID: \"44ed0322-9b5b-4751-8f7f-5f0c58493345\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" Oct 30 22:16:18.703688 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.703659 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l"] Oct 30 22:16:18.734502 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.734476 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44ed0322-9b5b-4751-8f7f-5f0c58493345-istio-token\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt\" (UID: \"44ed0322-9b5b-4751-8f7f-5f0c58493345\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" Oct 30 22:16:18.758845 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.758783 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8d9b\" (UniqueName: \"kubernetes.io/projected/44ed0322-9b5b-4751-8f7f-5f0c58493345-kube-api-access-k8d9b\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt\" (UID: \"44ed0322-9b5b-4751-8f7f-5f0c58493345\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" Oct 30 22:16:18.770747 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.770723 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg4st\" (UniqueName: \"kubernetes.io/projected/0b4e06dd-69d5-485e-acac-62dddf7f51cb-kube-api-access-mg4st\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l\" (UID: \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" Oct 30 22:16:18.770863 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.770766 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4e06dd-69d5-485e-acac-62dddf7f51cb-credential-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l\" (UID: \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" Oct 30 22:16:18.770904 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.770868 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b4e06dd-69d5-485e-acac-62dddf7f51cb-workload-certs\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l\" (UID: \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" Oct 30 22:16:18.770957 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.770940 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b4e06dd-69d5-485e-acac-62dddf7f51cb-istio-token\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l\" (UID: \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" Oct 30 22:16:18.770996 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.770971 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b4e06dd-69d5-485e-acac-62dddf7f51cb-istio-envoy\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l\" (UID: \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" Oct 30 22:16:18.771080 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.771065 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b4e06dd-69d5-485e-acac-62dddf7f51cb-istio-podinfo\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l\" (UID: \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" Oct 30 22:16:18.771189 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.771174 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4e06dd-69d5-485e-acac-62dddf7f51cb-workload-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l\" (UID: \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" Oct 30 22:16:18.771252 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.771203 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b4e06dd-69d5-485e-acac-62dddf7f51cb-istio-data\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l\" (UID: \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" Oct 30 22:16:18.771299 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.771271 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b4e06dd-69d5-485e-acac-62dddf7f51cb-workload-certs\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l\" (UID: \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" Oct 30 22:16:18.771299 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.771289 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b4e06dd-69d5-485e-acac-62dddf7f51cb-istiod-ca-cert\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l\" (UID: \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" Oct 30 22:16:18.771399 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.771204 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4e06dd-69d5-485e-acac-62dddf7f51cb-credential-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l\" (UID: \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" Oct 30 22:16:18.771489 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.771470 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b4e06dd-69d5-485e-acac-62dddf7f51cb-istio-data\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l\" (UID: \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" Oct 30 22:16:18.771533 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.771517 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4e06dd-69d5-485e-acac-62dddf7f51cb-workload-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l\" (UID: \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" Oct 30 22:16:18.771805 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.771784 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b4e06dd-69d5-485e-acac-62dddf7f51cb-istiod-ca-cert\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l\" (UID: \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" Oct 30 22:16:18.773246 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.773202 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b4e06dd-69d5-485e-acac-62dddf7f51cb-istio-envoy\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l\" (UID: \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" Oct 30 22:16:18.773525 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.773506 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b4e06dd-69d5-485e-acac-62dddf7f51cb-istio-podinfo\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l\" (UID: \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" Oct 30 22:16:18.802468 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.802443 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" Oct 30 22:16:18.883168 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.883135 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b4e06dd-69d5-485e-acac-62dddf7f51cb-istio-token\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l\" (UID: \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" Oct 30 22:16:18.884063 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:18.884033 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg4st\" (UniqueName: \"kubernetes.io/projected/0b4e06dd-69d5-485e-acac-62dddf7f51cb-kube-api-access-mg4st\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l\" (UID: \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" Oct 30 22:16:19.152731 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:19.152650 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" Oct 30 22:16:19.242791 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:16:19.242748 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44ed0322_9b5b_4751_8f7f_5f0c58493345.slice/crio-f1a0c0a252e47c4ae252c56b505e9dfad43f659e5235ee323c61d89cfaec8e87 WatchSource:0}: Error finding container f1a0c0a252e47c4ae252c56b505e9dfad43f659e5235ee323c61d89cfaec8e87: Status 404 returned error can't find the container with id f1a0c0a252e47c4ae252c56b505e9dfad43f659e5235ee323c61d89cfaec8e87 Oct 30 22:16:19.243103 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:19.243077 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt"] Oct 30 22:16:19.245969 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:19.245930 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:16:19.246083 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:19.246011 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:16:19.246083 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:19.246052 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:16:19.552291 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:19.552259 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l"] Oct 30 22:16:19.555405 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:16:19.555373 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b4e06dd_69d5_485e_acac_62dddf7f51cb.slice/crio-7994a68f55645987b5caf249568f8ed676692bdf0884a8e85e975b656b1544bd WatchSource:0}: Error finding container 7994a68f55645987b5caf249568f8ed676692bdf0884a8e85e975b656b1544bd: Status 404 returned error can't find the container with id 7994a68f55645987b5caf249568f8ed676692bdf0884a8e85e975b656b1544bd Oct 30 22:16:19.558389 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:19.558350 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:16:19.558454 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:19.558441 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:16:19.558504 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:19.558488 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:16:19.827531 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:19.827426 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" event={"ID":"44ed0322-9b5b-4751-8f7f-5f0c58493345","Type":"ContainerStarted","Data":"285b949241428a7d5792f0ad39c044b35b5776a0ef693c8df4c05fe24db47cee"} Oct 30 22:16:19.827531 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:19.827494 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" event={"ID":"44ed0322-9b5b-4751-8f7f-5f0c58493345","Type":"ContainerStarted","Data":"f1a0c0a252e47c4ae252c56b505e9dfad43f659e5235ee323c61d89cfaec8e87"} Oct 30 22:16:19.829346 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:19.829319 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" event={"ID":"0b4e06dd-69d5-485e-acac-62dddf7f51cb","Type":"ContainerStarted","Data":"772a26e1786f359420389b4c9ac35a88a19b648c28d75d3c53eb326ac36338f3"} Oct 30 22:16:19.829489 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:19.829361 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" event={"ID":"0b4e06dd-69d5-485e-acac-62dddf7f51cb","Type":"ContainerStarted","Data":"7994a68f55645987b5caf249568f8ed676692bdf0884a8e85e975b656b1544bd"} Oct 30 22:16:20.015238 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:20.011388 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" podStartSLOduration=2.011368971 podStartE2EDuration="2.011368971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:19.959721125 +0000 UTC m=+1540.230419226" watchObservedRunningTime="2026-10-30 22:16:20.011368971 +0000 UTC m=+1540.282067056" Oct 30 22:16:20.149836 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:20.148658 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" podStartSLOduration=2.148636702 podStartE2EDuration="2.148636702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:20.099018245 +0000 UTC m=+1540.369716341" watchObservedRunningTime="2026-10-30 22:16:20.148636702 +0000 UTC m=+1540.419334787" Oct 30 22:16:20.156640 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:20.156256 5918 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:16:20.156640 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:20.156325 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" podUID="0b4e06dd-69d5-485e-acac-62dddf7f51cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 22:16:20.157760 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:20.157733 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" Oct 30 22:16:20.806029 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:20.805995 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" Oct 30 22:16:20.806828 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:20.806711 5918 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 22:16:20.806991 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:20.806965 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" podUID="44ed0322-9b5b-4751-8f7f-5f0c58493345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 22:16:21.153888 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:21.153798 5918 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:16:21.153888 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:21.153865 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" podUID="0b4e06dd-69d5-485e-acac-62dddf7f51cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 22:16:21.279123 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:16:21.279085 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:16:21.815496 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:21.815459 5918 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 22:16:21.816065 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:21.816034 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" podUID="44ed0322-9b5b-4751-8f7f-5f0c58493345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 22:16:22.160533 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:22.160438 5918 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:16:22.160793 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:22.160764 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" podUID="0b4e06dd-69d5-485e-acac-62dddf7f51cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 22:16:22.811239 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:22.810606 5918 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 22:16:22.811239 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:22.810664 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" podUID="44ed0322-9b5b-4751-8f7f-5f0c58493345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 22:16:23.154299 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:23.153622 5918 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:16:23.154299 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:23.153692 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" podUID="0b4e06dd-69d5-485e-acac-62dddf7f51cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 22:16:23.805240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:23.804751 5918 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 22:16:23.805240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:23.804830 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" podUID="44ed0322-9b5b-4751-8f7f-5f0c58493345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 22:16:24.156323 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:24.153684 5918 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:16:24.156323 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:24.153742 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" podUID="0b4e06dd-69d5-485e-acac-62dddf7f51cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 22:16:24.805667 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:24.805627 5918 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 22:16:24.806419 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:24.806384 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" podUID="44ed0322-9b5b-4751-8f7f-5f0c58493345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 22:16:25.153128 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:25.153037 5918 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:16:25.153128 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:25.153099 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" podUID="0b4e06dd-69d5-485e-acac-62dddf7f51cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 22:16:25.809241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:25.804696 5918 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 22:16:25.809241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:25.804775 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" podUID="44ed0322-9b5b-4751-8f7f-5f0c58493345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 22:16:26.154005 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:26.153921 5918 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:16:26.154005 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:26.153975 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" podUID="0b4e06dd-69d5-485e-acac-62dddf7f51cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 22:16:26.809180 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:26.809150 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" Oct 30 22:16:26.809506 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:26.809475 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" Oct 30 22:16:26.810406 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:26.810385 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" Oct 30 22:16:27.173967 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:27.173847 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" Oct 30 22:16:27.176246 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:27.176203 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" Oct 30 22:16:27.177871 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:27.177843 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" Oct 30 22:16:28.641066 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:28.641037 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv"] Oct 30 22:16:28.645601 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:28.645571 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv" Oct 30 22:16:28.712328 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:28.712297 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv"] Oct 30 22:16:28.755493 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:28.755466 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-422c1029-dockercfg-8gd98\"" Oct 30 22:16:28.755772 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:28.755751 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-422c1029\"" Oct 30 22:16:28.806320 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:28.806267 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/471d388d-0ab5-4bec-ac57-93d6f4df1bd6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv\" (UID: \"471d388d-0ab5-4bec-ac57-93d6f4df1bd6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv" Oct 30 22:16:28.806726 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:28.806706 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrf97\" (UniqueName: \"kubernetes.io/projected/471d388d-0ab5-4bec-ac57-93d6f4df1bd6-kube-api-access-xrf97\") pod \"payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv\" (UID: \"471d388d-0ab5-4bec-ac57-93d6f4df1bd6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv" Oct 30 22:16:28.908573 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:28.908472 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/471d388d-0ab5-4bec-ac57-93d6f4df1bd6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv\" (UID: \"471d388d-0ab5-4bec-ac57-93d6f4df1bd6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv" Oct 30 22:16:28.908753 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:28.908582 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrf97\" (UniqueName: \"kubernetes.io/projected/471d388d-0ab5-4bec-ac57-93d6f4df1bd6-kube-api-access-xrf97\") pod \"payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv\" (UID: \"471d388d-0ab5-4bec-ac57-93d6f4df1bd6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv" Oct 30 22:16:28.909397 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:28.909372 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/471d388d-0ab5-4bec-ac57-93d6f4df1bd6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv\" (UID: \"471d388d-0ab5-4bec-ac57-93d6f4df1bd6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv" Oct 30 22:16:28.974450 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:28.974419 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrf97\" (UniqueName: \"kubernetes.io/projected/471d388d-0ab5-4bec-ac57-93d6f4df1bd6-kube-api-access-xrf97\") pod \"payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv\" (UID: \"471d388d-0ab5-4bec-ac57-93d6f4df1bd6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv" Oct 30 22:16:29.261023 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:29.260994 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv" Oct 30 22:16:29.678079 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:29.678048 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv"] Oct 30 22:16:29.679583 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:16:29.679559 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod471d388d_0ab5_4bec_ac57_93d6f4df1bd6.slice/crio-53b9f2ee05a265d28238c0f605b6fbf39257df46e1543e93ceb74d142a54f220 WatchSource:0}: Error finding container 53b9f2ee05a265d28238c0f605b6fbf39257df46e1543e93ceb74d142a54f220: Status 404 returned error can't find the container with id 53b9f2ee05a265d28238c0f605b6fbf39257df46e1543e93ceb74d142a54f220 Oct 30 22:16:30.253319 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:30.253277 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv" event={"ID":"471d388d-0ab5-4bec-ac57-93d6f4df1bd6","Type":"ContainerStarted","Data":"32b232dd22f404562c2aa606f0fb4967162196dccbaa8dfcce5d039ca6ad2dd1"} Oct 30 22:16:30.253519 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:30.253347 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv" event={"ID":"471d388d-0ab5-4bec-ac57-93d6f4df1bd6","Type":"ContainerStarted","Data":"53b9f2ee05a265d28238c0f605b6fbf39257df46e1543e93ceb74d142a54f220"} Oct 30 22:16:30.257849 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:30.254417 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv" Oct 30 22:16:30.381296 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:30.381233 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv" podStartSLOduration=2.381201376 podStartE2EDuration="2.381201376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:30.378802379 +0000 UTC m=+1550.649500462" watchObservedRunningTime="2026-10-30 22:16:30.381201376 +0000 UTC m=+1550.651899461" Oct 30 22:16:42.269362 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:42.269332 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv" Oct 30 22:16:50.661164 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:50.661126 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4"] Oct 30 22:16:50.661799 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:50.661360 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" podUID="438b8728-4021-49e9-8c37-fbdd405008f1" containerName="istio-proxy" containerID="cri-o://c6557f6ee7fe7d192bc4813b384e3552d9108556e2720820dd3ab0d0f961c6c8" gracePeriod=30 Oct 30 22:16:51.275830 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:16:51.275788 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:16:52.391482 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:52.391448 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst"] Oct 30 22:16:52.391889 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:52.391649 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" podUID="06181307-c42a-411e-8216-6c8f4d9a4a9d" containerName="istio-proxy" containerID="cri-o://fd6b13aff9e3de14e6768bb2b3a0e11c59b264ea89ebecab3f0a5506484ab353" gracePeriod=30 Oct 30 22:16:55.696197 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:55.696169 5918 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 22:16:55.696563 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:55.696235 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" podUID="438b8728-4021-49e9-8c37-fbdd405008f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 22:16:55.947310 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:55.947286 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" Oct 30 22:16:56.020704 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.020671 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/438b8728-4021-49e9-8c37-fbdd405008f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/438b8728-4021-49e9-8c37-fbdd405008f1-credential-socket\") pod \"438b8728-4021-49e9-8c37-fbdd405008f1\" (UID: \"438b8728-4021-49e9-8c37-fbdd405008f1\") " Oct 30 22:16:56.020704 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.020709 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/438b8728-4021-49e9-8c37-fbdd405008f1-istio-token\" (UniqueName: \"kubernetes.io/projected/438b8728-4021-49e9-8c37-fbdd405008f1-istio-token\") pod \"438b8728-4021-49e9-8c37-fbdd405008f1\" (UID: \"438b8728-4021-49e9-8c37-fbdd405008f1\") " Oct 30 22:16:56.020952 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.020812 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/438b8728-4021-49e9-8c37-fbdd405008f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/438b8728-4021-49e9-8c37-fbdd405008f1-istio-data\") pod \"438b8728-4021-49e9-8c37-fbdd405008f1\" (UID: \"438b8728-4021-49e9-8c37-fbdd405008f1\") " Oct 30 22:16:56.020952 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.020871 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/438b8728-4021-49e9-8c37-fbdd405008f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/438b8728-4021-49e9-8c37-fbdd405008f1-istiod-ca-cert\") pod \"438b8728-4021-49e9-8c37-fbdd405008f1\" (UID: \"438b8728-4021-49e9-8c37-fbdd405008f1\") " Oct 30 22:16:56.020952 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.020915 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/438b8728-4021-49e9-8c37-fbdd405008f1-kube-api-access-ql9cl\" (UniqueName: \"kubernetes.io/projected/438b8728-4021-49e9-8c37-fbdd405008f1-kube-api-access-ql9cl\") pod \"438b8728-4021-49e9-8c37-fbdd405008f1\" (UID: \"438b8728-4021-49e9-8c37-fbdd405008f1\") " Oct 30 22:16:56.021120 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.020974 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/438b8728-4021-49e9-8c37-fbdd405008f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/438b8728-4021-49e9-8c37-fbdd405008f1-workload-certs\") pod \"438b8728-4021-49e9-8c37-fbdd405008f1\" (UID: \"438b8728-4021-49e9-8c37-fbdd405008f1\") " Oct 30 22:16:56.021120 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.020983 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/438b8728-4021-49e9-8c37-fbdd405008f1-credential-socket" pod "438b8728-4021-49e9-8c37-fbdd405008f1" (UID: "438b8728-4021-49e9-8c37-fbdd405008f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:56.021120 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.021015 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/438b8728-4021-49e9-8c37-fbdd405008f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/438b8728-4021-49e9-8c37-fbdd405008f1-istio-envoy\") pod \"438b8728-4021-49e9-8c37-fbdd405008f1\" (UID: \"438b8728-4021-49e9-8c37-fbdd405008f1\") " Oct 30 22:16:56.021304 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.021133 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/438b8728-4021-49e9-8c37-fbdd405008f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/438b8728-4021-49e9-8c37-fbdd405008f1-workload-socket\") pod \"438b8728-4021-49e9-8c37-fbdd405008f1\" (UID: \"438b8728-4021-49e9-8c37-fbdd405008f1\") " Oct 30 22:16:56.021304 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.021276 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/438b8728-4021-49e9-8c37-fbdd405008f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/438b8728-4021-49e9-8c37-fbdd405008f1-istio-podinfo\") pod \"438b8728-4021-49e9-8c37-fbdd405008f1\" (UID: \"438b8728-4021-49e9-8c37-fbdd405008f1\") " Oct 30 22:16:56.021416 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.021344 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/438b8728-4021-49e9-8c37-fbdd405008f1-istiod-ca-cert" pod "438b8728-4021-49e9-8c37-fbdd405008f1" (UID: "438b8728-4021-49e9-8c37-fbdd405008f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:56.021839 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.021794 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/438b8728-4021-49e9-8c37-fbdd405008f1-workload-certs" pod "438b8728-4021-49e9-8c37-fbdd405008f1" (UID: "438b8728-4021-49e9-8c37-fbdd405008f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:56.021839 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.021828 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/438b8728-4021-49e9-8c37-fbdd405008f1-istio-data" pod "438b8728-4021-49e9-8c37-fbdd405008f1" (UID: "438b8728-4021-49e9-8c37-fbdd405008f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:56.022068 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.021965 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/438b8728-4021-49e9-8c37-fbdd405008f1-workload-socket" pod "438b8728-4021-49e9-8c37-fbdd405008f1" (UID: "438b8728-4021-49e9-8c37-fbdd405008f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:56.022289 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.022257 5918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/438b8728-4021-49e9-8c37-fbdd405008f1-credential-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:56.022289 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.022291 5918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/438b8728-4021-49e9-8c37-fbdd405008f1-istio-data\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:56.022508 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.022300 5918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/438b8728-4021-49e9-8c37-fbdd405008f1-istiod-ca-cert\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:56.022508 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.022310 5918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/438b8728-4021-49e9-8c37-fbdd405008f1-workload-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:56.022508 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.022318 5918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/438b8728-4021-49e9-8c37-fbdd405008f1-workload-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:56.023481 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.023451 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/438b8728-4021-49e9-8c37-fbdd405008f1-kube-api-access-ql9cl" pod "438b8728-4021-49e9-8c37-fbdd405008f1" (UID: "438b8728-4021-49e9-8c37-fbdd405008f1"). InnerVolumeSpecName "kube-api-access-ql9cl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:56.023942 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.023909 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/438b8728-4021-49e9-8c37-fbdd405008f1-istio-token" pod "438b8728-4021-49e9-8c37-fbdd405008f1" (UID: "438b8728-4021-49e9-8c37-fbdd405008f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:56.024026 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.023945 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/438b8728-4021-49e9-8c37-fbdd405008f1-istio-envoy" pod "438b8728-4021-49e9-8c37-fbdd405008f1" (UID: "438b8728-4021-49e9-8c37-fbdd405008f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:56.024174 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.024156 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/438b8728-4021-49e9-8c37-fbdd405008f1-istio-podinfo" pod "438b8728-4021-49e9-8c37-fbdd405008f1" (UID: "438b8728-4021-49e9-8c37-fbdd405008f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:16:56.123937 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.123905 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql9cl\" (UniqueName: \"kubernetes.io/projected/438b8728-4021-49e9-8c37-fbdd405008f1-kube-api-access-ql9cl\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:56.123937 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.123934 5918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/438b8728-4021-49e9-8c37-fbdd405008f1-istio-envoy\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:56.123937 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.123945 5918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/438b8728-4021-49e9-8c37-fbdd405008f1-istio-podinfo\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:56.124148 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.123957 5918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/438b8728-4021-49e9-8c37-fbdd405008f1-istio-token\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:56.454838 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.454803 5918 generic.go:356] "Generic (PLEG): container finished" podID="438b8728-4021-49e9-8c37-fbdd405008f1" containerID="c6557f6ee7fe7d192bc4813b384e3552d9108556e2720820dd3ab0d0f961c6c8" exitCode=0 Oct 30 22:16:56.455037 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.454864 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" Oct 30 22:16:56.455037 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.454929 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" event={"ID":"438b8728-4021-49e9-8c37-fbdd405008f1","Type":"ContainerDied","Data":"c6557f6ee7fe7d192bc4813b384e3552d9108556e2720820dd3ab0d0f961c6c8"} Oct 30 22:16:56.455037 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.454980 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4" event={"ID":"438b8728-4021-49e9-8c37-fbdd405008f1","Type":"ContainerDied","Data":"da6ce86bd8b378353beaeb6da69903e88e3ce564c2673dfd68f56b12a759a598"} Oct 30 22:16:56.455037 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.455001 5918 scope.go:122] "RemoveContainer" containerID="c6557f6ee7fe7d192bc4813b384e3552d9108556e2720820dd3ab0d0f961c6c8" Oct 30 22:16:56.466267 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.466245 5918 scope.go:122] "RemoveContainer" containerID="c6557f6ee7fe7d192bc4813b384e3552d9108556e2720820dd3ab0d0f961c6c8" Oct 30 22:16:56.466552 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:16:56.466531 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6557f6ee7fe7d192bc4813b384e3552d9108556e2720820dd3ab0d0f961c6c8\": container with ID starting with c6557f6ee7fe7d192bc4813b384e3552d9108556e2720820dd3ab0d0f961c6c8 not found: ID does not exist" containerID="c6557f6ee7fe7d192bc4813b384e3552d9108556e2720820dd3ab0d0f961c6c8" Oct 30 22:16:56.466644 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.466562 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6557f6ee7fe7d192bc4813b384e3552d9108556e2720820dd3ab0d0f961c6c8"} err="failed to get container status \"c6557f6ee7fe7d192bc4813b384e3552d9108556e2720820dd3ab0d0f961c6c8\": rpc error: code = NotFound desc = could not find container \"c6557f6ee7fe7d192bc4813b384e3552d9108556e2720820dd3ab0d0f961c6c8\": container with ID starting with c6557f6ee7fe7d192bc4813b384e3552d9108556e2720820dd3ab0d0f961c6c8 not found: ID does not exist" Oct 30 22:16:56.510489 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.510463 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh"] Oct 30 22:16:56.511146 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.511120 5918 cpu_manager.go:414] "containerMap: removing container" podUID="438b8728-4021-49e9-8c37-fbdd405008f1" containerName="istio-proxy" Oct 30 22:16:56.511146 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.511137 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="438b8728-4021-49e9-8c37-fbdd405008f1" containerName="istio-proxy" Oct 30 22:16:56.511320 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.511259 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="438b8728-4021-49e9-8c37-fbdd405008f1" containerName="istio-proxy" Oct 30 22:16:56.516141 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.516124 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh" Oct 30 22:16:56.578151 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.578127 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh"] Oct 30 22:16:56.579747 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.579730 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-516ccb-serving-cert\"" Oct 30 22:16:56.630481 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.630451 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6eadf8-b382-4d7e-a27b-9f02938df064-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh\" (UID: \"1c6eadf8-b382-4d7e-a27b-9f02938df064\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh" Oct 30 22:16:56.630659 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.630508 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c6eadf8-b382-4d7e-a27b-9f02938df064-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh\" (UID: \"1c6eadf8-b382-4d7e-a27b-9f02938df064\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh" Oct 30 22:16:56.630748 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.630720 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rp2r\" (UniqueName: \"kubernetes.io/projected/1c6eadf8-b382-4d7e-a27b-9f02938df064-kube-api-access-2rp2r\") pod \"maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh\" (UID: \"1c6eadf8-b382-4d7e-a27b-9f02938df064\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh" Oct 30 22:16:56.630810 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.630782 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c6eadf8-b382-4d7e-a27b-9f02938df064-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh\" (UID: \"1c6eadf8-b382-4d7e-a27b-9f02938df064\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh" Oct 30 22:16:56.676565 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.676533 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4"] Oct 30 22:16:56.728845 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.728768 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-th4q4"] Oct 30 22:16:56.732004 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.731982 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6eadf8-b382-4d7e-a27b-9f02938df064-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh\" (UID: \"1c6eadf8-b382-4d7e-a27b-9f02938df064\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh" Oct 30 22:16:56.732133 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.732037 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c6eadf8-b382-4d7e-a27b-9f02938df064-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh\" (UID: \"1c6eadf8-b382-4d7e-a27b-9f02938df064\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh" Oct 30 22:16:56.732194 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.732145 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rp2r\" (UniqueName: \"kubernetes.io/projected/1c6eadf8-b382-4d7e-a27b-9f02938df064-kube-api-access-2rp2r\") pod \"maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh\" (UID: \"1c6eadf8-b382-4d7e-a27b-9f02938df064\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh" Oct 30 22:16:56.732194 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.732173 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c6eadf8-b382-4d7e-a27b-9f02938df064-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh\" (UID: \"1c6eadf8-b382-4d7e-a27b-9f02938df064\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh" Oct 30 22:16:56.732394 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:16:56.732372 5918 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-516ccb-serving-cert: secret "maas-api-e2e-shared-a-w0-516ccb-serving-cert" not found Oct 30 22:16:56.732458 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:16:56.732448 5918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c6eadf8-b382-4d7e-a27b-9f02938df064-maas-api-tls podName:1c6eadf8-b382-4d7e-a27b-9f02938df064 nodeName:}" failed. No retries permitted until 2026-10-30 22:16:57.232427498 +0000 UTC m=+1577.503125561 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1c6eadf8-b382-4d7e-a27b-9f02938df064-maas-api-tls") pod "maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh" (UID: "1c6eadf8-b382-4d7e-a27b-9f02938df064") : secret "maas-api-e2e-shared-a-w0-516ccb-serving-cert" not found Oct 30 22:16:56.732776 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.732755 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c6eadf8-b382-4d7e-a27b-9f02938df064-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh\" (UID: \"1c6eadf8-b382-4d7e-a27b-9f02938df064\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh" Oct 30 22:16:56.734630 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.734606 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6eadf8-b382-4d7e-a27b-9f02938df064-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh\" (UID: \"1c6eadf8-b382-4d7e-a27b-9f02938df064\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh" Oct 30 22:16:56.796443 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:56.796406 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rp2r\" (UniqueName: \"kubernetes.io/projected/1c6eadf8-b382-4d7e-a27b-9f02938df064-kube-api-access-2rp2r\") pod \"maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh\" (UID: \"1c6eadf8-b382-4d7e-a27b-9f02938df064\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh" Oct 30 22:16:57.240139 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:57.240102 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c6eadf8-b382-4d7e-a27b-9f02938df064-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh\" (UID: \"1c6eadf8-b382-4d7e-a27b-9f02938df064\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh" Oct 30 22:16:57.242753 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:57.242730 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c6eadf8-b382-4d7e-a27b-9f02938df064-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh\" (UID: \"1c6eadf8-b382-4d7e-a27b-9f02938df064\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh" Oct 30 22:16:57.429540 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:57.429510 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh" Oct 30 22:16:57.662886 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:57.662863 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" Oct 30 22:16:57.748628 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:57.748596 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06181307-c42a-411e-8216-6c8f4d9a4a9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06181307-c42a-411e-8216-6c8f4d9a4a9d-istio-data\") pod \"06181307-c42a-411e-8216-6c8f4d9a4a9d\" (UID: \"06181307-c42a-411e-8216-6c8f4d9a4a9d\") " Oct 30 22:16:57.749022 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:57.748645 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06181307-c42a-411e-8216-6c8f4d9a4a9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06181307-c42a-411e-8216-6c8f4d9a4a9d-istio-envoy\") pod \"06181307-c42a-411e-8216-6c8f4d9a4a9d\" (UID: \"06181307-c42a-411e-8216-6c8f4d9a4a9d\") " Oct 30 22:16:57.749022 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:57.748696 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06181307-c42a-411e-8216-6c8f4d9a4a9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06181307-c42a-411e-8216-6c8f4d9a4a9d-workload-certs\") pod \"06181307-c42a-411e-8216-6c8f4d9a4a9d\" (UID: \"06181307-c42a-411e-8216-6c8f4d9a4a9d\") " Oct 30 22:16:57.749022 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:57.748744 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06181307-c42a-411e-8216-6c8f4d9a4a9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06181307-c42a-411e-8216-6c8f4d9a4a9d-istio-podinfo\") pod \"06181307-c42a-411e-8216-6c8f4d9a4a9d\" (UID: \"06181307-c42a-411e-8216-6c8f4d9a4a9d\") " Oct 30 22:16:57.749022 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:57.748877 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06181307-c42a-411e-8216-6c8f4d9a4a9d-istio-data" pod "06181307-c42a-411e-8216-6c8f4d9a4a9d" (UID: "06181307-c42a-411e-8216-6c8f4d9a4a9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:57.749022 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:57.748891 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06181307-c42a-411e-8216-6c8f4d9a4a9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06181307-c42a-411e-8216-6c8f4d9a4a9d-credential-socket\") pod \"06181307-c42a-411e-8216-6c8f4d9a4a9d\" (UID: \"06181307-c42a-411e-8216-6c8f4d9a4a9d\") " Oct 30 22:16:57.749022 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:57.748995 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06181307-c42a-411e-8216-6c8f4d9a4a9d-workload-certs" pod "06181307-c42a-411e-8216-6c8f4d9a4a9d" (UID: "06181307-c42a-411e-8216-6c8f4d9a4a9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:57.749382 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:57.749063 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06181307-c42a-411e-8216-6c8f4d9a4a9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06181307-c42a-411e-8216-6c8f4d9a4a9d-workload-socket\") pod \"06181307-c42a-411e-8216-6c8f4d9a4a9d\" (UID: \"06181307-c42a-411e-8216-6c8f4d9a4a9d\") " Oct 30 22:16:57.749382 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:57.749100 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06181307-c42a-411e-8216-6c8f4d9a4a9d-istio-token\" (UniqueName: \"kubernetes.io/projected/06181307-c42a-411e-8216-6c8f4d9a4a9d-istio-token\") pod \"06181307-c42a-411e-8216-6c8f4d9a4a9d\" (UID: \"06181307-c42a-411e-8216-6c8f4d9a4a9d\") " Oct 30 22:16:57.749382 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:57.749104 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06181307-c42a-411e-8216-6c8f4d9a4a9d-credential-socket" pod "06181307-c42a-411e-8216-6c8f4d9a4a9d" (UID: "06181307-c42a-411e-8216-6c8f4d9a4a9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:57.749382 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:57.749167 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06181307-c42a-411e-8216-6c8f4d9a4a9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06181307-c42a-411e-8216-6c8f4d9a4a9d-istiod-ca-cert\") pod \"06181307-c42a-411e-8216-6c8f4d9a4a9d\" (UID: \"06181307-c42a-411e-8216-6c8f4d9a4a9d\") " Oct 30 22:16:57.749382 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:57.749265 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06181307-c42a-411e-8216-6c8f4d9a4a9d-kube-api-access-kz8x2\" (UniqueName: \"kubernetes.io/projected/06181307-c42a-411e-8216-6c8f4d9a4a9d-kube-api-access-kz8x2\") pod \"06181307-c42a-411e-8216-6c8f4d9a4a9d\" (UID: \"06181307-c42a-411e-8216-6c8f4d9a4a9d\") " Oct 30 22:16:57.749382 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:57.749299 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06181307-c42a-411e-8216-6c8f4d9a4a9d-workload-socket" pod "06181307-c42a-411e-8216-6c8f4d9a4a9d" (UID: "06181307-c42a-411e-8216-6c8f4d9a4a9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:57.749628 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:57.749532 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06181307-c42a-411e-8216-6c8f4d9a4a9d-istiod-ca-cert" pod "06181307-c42a-411e-8216-6c8f4d9a4a9d" (UID: "06181307-c42a-411e-8216-6c8f4d9a4a9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:57.750232 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:57.750197 5918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06181307-c42a-411e-8216-6c8f4d9a4a9d-workload-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:57.750341 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:57.750243 5918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06181307-c42a-411e-8216-6c8f4d9a4a9d-credential-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:57.750341 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:57.750262 5918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06181307-c42a-411e-8216-6c8f4d9a4a9d-workload-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:57.750341 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:57.750276 5918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06181307-c42a-411e-8216-6c8f4d9a4a9d-istiod-ca-cert\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:57.750341 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:57.750289 5918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06181307-c42a-411e-8216-6c8f4d9a4a9d-istio-data\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:57.751598 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:57.751569 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06181307-c42a-411e-8216-6c8f4d9a4a9d-istio-envoy" pod "06181307-c42a-411e-8216-6c8f4d9a4a9d" (UID: "06181307-c42a-411e-8216-6c8f4d9a4a9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:57.751685 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:57.751591 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06181307-c42a-411e-8216-6c8f4d9a4a9d-istio-token" pod "06181307-c42a-411e-8216-6c8f4d9a4a9d" (UID: "06181307-c42a-411e-8216-6c8f4d9a4a9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:57.751685 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:57.751644 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06181307-c42a-411e-8216-6c8f4d9a4a9d-kube-api-access-kz8x2" pod "06181307-c42a-411e-8216-6c8f4d9a4a9d" (UID: "06181307-c42a-411e-8216-6c8f4d9a4a9d"). InnerVolumeSpecName "kube-api-access-kz8x2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:57.751685 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:57.751657 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06181307-c42a-411e-8216-6c8f4d9a4a9d-istio-podinfo" pod "06181307-c42a-411e-8216-6c8f4d9a4a9d" (UID: "06181307-c42a-411e-8216-6c8f4d9a4a9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:16:57.820400 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:57.820367 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh"] Oct 30 22:16:57.820822 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:16:57.820795 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c6eadf8_b382_4d7e_a27b_9f02938df064.slice/crio-fe76e433c2e0a9f4168b93130129879f1b43551e1e0d0340a510b6e8662163eb WatchSource:0}: Error finding container fe76e433c2e0a9f4168b93130129879f1b43551e1e0d0340a510b6e8662163eb: Status 404 returned error can't find the container with id fe76e433c2e0a9f4168b93130129879f1b43551e1e0d0340a510b6e8662163eb Oct 30 22:16:57.822666 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:57.822648 5918 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:16:57.851725 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:57.851692 5918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06181307-c42a-411e-8216-6c8f4d9a4a9d-istio-podinfo\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:57.851725 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:57.851724 5918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06181307-c42a-411e-8216-6c8f4d9a4a9d-istio-token\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:57.851965 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:57.851740 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz8x2\" (UniqueName: \"kubernetes.io/projected/06181307-c42a-411e-8216-6c8f4d9a4a9d-kube-api-access-kz8x2\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:57.851965 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:57.851755 5918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06181307-c42a-411e-8216-6c8f4d9a4a9d-istio-envoy\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:57.857046 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:57.857015 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="438b8728-4021-49e9-8c37-fbdd405008f1" path="/var/lib/kubelet/pods/438b8728-4021-49e9-8c37-fbdd405008f1/volumes" Oct 30 22:16:58.471061 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:58.471032 5918 generic.go:356] "Generic (PLEG): container finished" podID="06181307-c42a-411e-8216-6c8f4d9a4a9d" containerID="fd6b13aff9e3de14e6768bb2b3a0e11c59b264ea89ebecab3f0a5506484ab353" exitCode=0 Oct 30 22:16:58.471249 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:58.471087 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" Oct 30 22:16:58.471249 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:58.471105 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" event={"ID":"06181307-c42a-411e-8216-6c8f4d9a4a9d","Type":"ContainerDied","Data":"fd6b13aff9e3de14e6768bb2b3a0e11c59b264ea89ebecab3f0a5506484ab353"} Oct 30 22:16:58.471249 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:58.471158 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst" event={"ID":"06181307-c42a-411e-8216-6c8f4d9a4a9d","Type":"ContainerDied","Data":"1436c08eb43a76ecbc45f2d87b5fcd90286123a34b1ab36c1c3b7b577454daec"} Oct 30 22:16:58.471249 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:58.471190 5918 scope.go:122] "RemoveContainer" containerID="fd6b13aff9e3de14e6768bb2b3a0e11c59b264ea89ebecab3f0a5506484ab353" Oct 30 22:16:58.472665 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:58.472640 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh" event={"ID":"1c6eadf8-b382-4d7e-a27b-9f02938df064","Type":"ContainerStarted","Data":"fe76e433c2e0a9f4168b93130129879f1b43551e1e0d0340a510b6e8662163eb"} Oct 30 22:16:58.481751 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:58.481734 5918 scope.go:122] "RemoveContainer" containerID="fd6b13aff9e3de14e6768bb2b3a0e11c59b264ea89ebecab3f0a5506484ab353" Oct 30 22:16:58.482041 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:16:58.482024 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd6b13aff9e3de14e6768bb2b3a0e11c59b264ea89ebecab3f0a5506484ab353\": container with ID starting with fd6b13aff9e3de14e6768bb2b3a0e11c59b264ea89ebecab3f0a5506484ab353 not found: ID does not exist" containerID="fd6b13aff9e3de14e6768bb2b3a0e11c59b264ea89ebecab3f0a5506484ab353" Oct 30 22:16:58.482087 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:58.482049 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd6b13aff9e3de14e6768bb2b3a0e11c59b264ea89ebecab3f0a5506484ab353"} err="failed to get container status \"fd6b13aff9e3de14e6768bb2b3a0e11c59b264ea89ebecab3f0a5506484ab353\": rpc error: code = NotFound desc = could not find container \"fd6b13aff9e3de14e6768bb2b3a0e11c59b264ea89ebecab3f0a5506484ab353\": container with ID starting with fd6b13aff9e3de14e6768bb2b3a0e11c59b264ea89ebecab3f0a5506484ab353 not found: ID does not exist" Oct 30 22:16:58.697430 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:58.697395 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst"] Oct 30 22:16:58.748452 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:58.748379 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-96rst"] Oct 30 22:16:59.484685 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:59.484649 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh" event={"ID":"1c6eadf8-b382-4d7e-a27b-9f02938df064","Type":"ContainerStarted","Data":"c7e3037a3755c892fc83d5f69879bbacbb31e962e8ed504e310e366585951e38"} Oct 30 22:16:59.485099 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:59.484721 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh" Oct 30 22:16:59.606988 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:59.606932 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh" podStartSLOduration=2.666643167 podStartE2EDuration="3.606918504s" totalImagesPullingTime="940.275337ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:59.605936876 +0000 UTC m=+1579.876634960" watchObservedRunningTime="2026-10-30 22:16:59.606918504 +0000 UTC m=+1579.877616587" Oct 30 22:16:59.860733 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:16:59.860647 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06181307-c42a-411e-8216-6c8f4d9a4a9d" path="/var/lib/kubelet/pods/06181307-c42a-411e-8216-6c8f4d9a4a9d/volumes" Oct 30 22:17:03.886645 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:03.886612 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff"] Oct 30 22:17:03.887286 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:03.887270 5918 cpu_manager.go:414] "containerMap: removing container" podUID="06181307-c42a-411e-8216-6c8f4d9a4a9d" containerName="istio-proxy" Oct 30 22:17:03.887341 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:03.887287 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06181307-c42a-411e-8216-6c8f4d9a4a9d" containerName="istio-proxy" Oct 30 22:17:03.887405 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:03.887396 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="06181307-c42a-411e-8216-6c8f4d9a4a9d" containerName="istio-proxy" Oct 30 22:17:03.891181 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:03.891159 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" Oct 30 22:17:04.011859 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:04.011820 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff"] Oct 30 22:17:04.051154 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:04.051119 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7e83907-c1a2-4761-ab76-31fe664742a3-credential-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff\" (UID: \"b7e83907-c1a2-4761-ab76-31fe664742a3\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" Oct 30 22:17:04.051335 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:04.051260 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7e83907-c1a2-4761-ab76-31fe664742a3-workload-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff\" (UID: \"b7e83907-c1a2-4761-ab76-31fe664742a3\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" Oct 30 22:17:04.051335 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:04.051306 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwk54\" (UniqueName: \"kubernetes.io/projected/b7e83907-c1a2-4761-ab76-31fe664742a3-kube-api-access-nwk54\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff\" (UID: \"b7e83907-c1a2-4761-ab76-31fe664742a3\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" Oct 30 22:17:04.051440 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:04.051420 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7e83907-c1a2-4761-ab76-31fe664742a3-istio-envoy\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff\" (UID: \"b7e83907-c1a2-4761-ab76-31fe664742a3\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" Oct 30 22:17:04.051535 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:04.051521 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7e83907-c1a2-4761-ab76-31fe664742a3-istio-token\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff\" (UID: \"b7e83907-c1a2-4761-ab76-31fe664742a3\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" Oct 30 22:17:04.051571 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:04.051545 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7e83907-c1a2-4761-ab76-31fe664742a3-workload-certs\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff\" (UID: \"b7e83907-c1a2-4761-ab76-31fe664742a3\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" Oct 30 22:17:04.051607 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:04.051579 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7e83907-c1a2-4761-ab76-31fe664742a3-istio-data\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff\" (UID: \"b7e83907-c1a2-4761-ab76-31fe664742a3\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" Oct 30 22:17:04.051755 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:04.051736 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7e83907-c1a2-4761-ab76-31fe664742a3-istiod-ca-cert\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff\" (UID: \"b7e83907-c1a2-4761-ab76-31fe664742a3\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" Oct 30 22:17:04.051799 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:04.051787 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7e83907-c1a2-4761-ab76-31fe664742a3-istio-podinfo\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff\" (UID: \"b7e83907-c1a2-4761-ab76-31fe664742a3\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" Oct 30 22:17:04.153005 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:04.152920 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7e83907-c1a2-4761-ab76-31fe664742a3-istio-podinfo\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff\" (UID: \"b7e83907-c1a2-4761-ab76-31fe664742a3\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" Oct 30 22:17:04.153005 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:04.152976 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7e83907-c1a2-4761-ab76-31fe664742a3-credential-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff\" (UID: \"b7e83907-c1a2-4761-ab76-31fe664742a3\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" Oct 30 22:17:04.153296 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:04.153032 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7e83907-c1a2-4761-ab76-31fe664742a3-workload-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff\" (UID: \"b7e83907-c1a2-4761-ab76-31fe664742a3\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" Oct 30 22:17:04.153296 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:04.153062 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwk54\" (UniqueName: \"kubernetes.io/projected/b7e83907-c1a2-4761-ab76-31fe664742a3-kube-api-access-nwk54\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff\" (UID: \"b7e83907-c1a2-4761-ab76-31fe664742a3\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" Oct 30 22:17:04.153296 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:04.153115 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7e83907-c1a2-4761-ab76-31fe664742a3-istio-envoy\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff\" (UID: \"b7e83907-c1a2-4761-ab76-31fe664742a3\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" Oct 30 22:17:04.153296 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:04.153185 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7e83907-c1a2-4761-ab76-31fe664742a3-istio-token\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff\" (UID: \"b7e83907-c1a2-4761-ab76-31fe664742a3\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" Oct 30 22:17:04.153296 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:04.153203 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7e83907-c1a2-4761-ab76-31fe664742a3-workload-certs\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff\" (UID: \"b7e83907-c1a2-4761-ab76-31fe664742a3\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" Oct 30 22:17:04.153296 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:04.153268 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7e83907-c1a2-4761-ab76-31fe664742a3-istio-data\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff\" (UID: \"b7e83907-c1a2-4761-ab76-31fe664742a3\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" Oct 30 22:17:04.153626 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:04.153391 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7e83907-c1a2-4761-ab76-31fe664742a3-istiod-ca-cert\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff\" (UID: \"b7e83907-c1a2-4761-ab76-31fe664742a3\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" Oct 30 22:17:04.153626 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:04.153465 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7e83907-c1a2-4761-ab76-31fe664742a3-credential-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff\" (UID: \"b7e83907-c1a2-4761-ab76-31fe664742a3\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" Oct 30 22:17:04.153626 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:04.153473 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7e83907-c1a2-4761-ab76-31fe664742a3-workload-socket\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff\" (UID: \"b7e83907-c1a2-4761-ab76-31fe664742a3\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" Oct 30 22:17:04.153791 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:04.153627 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7e83907-c1a2-4761-ab76-31fe664742a3-workload-certs\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff\" (UID: \"b7e83907-c1a2-4761-ab76-31fe664742a3\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" Oct 30 22:17:04.153851 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:04.153807 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7e83907-c1a2-4761-ab76-31fe664742a3-istio-data\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff\" (UID: \"b7e83907-c1a2-4761-ab76-31fe664742a3\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" Oct 30 22:17:04.154075 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:04.154057 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7e83907-c1a2-4761-ab76-31fe664742a3-istiod-ca-cert\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff\" (UID: \"b7e83907-c1a2-4761-ab76-31fe664742a3\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" Oct 30 22:17:04.155751 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:04.155732 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7e83907-c1a2-4761-ab76-31fe664742a3-istio-podinfo\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff\" (UID: \"b7e83907-c1a2-4761-ab76-31fe664742a3\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" Oct 30 22:17:04.155903 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:04.155884 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7e83907-c1a2-4761-ab76-31fe664742a3-istio-envoy\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff\" (UID: \"b7e83907-c1a2-4761-ab76-31fe664742a3\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" Oct 30 22:17:04.218067 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:04.218034 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwk54\" (UniqueName: \"kubernetes.io/projected/b7e83907-c1a2-4761-ab76-31fe664742a3-kube-api-access-nwk54\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff\" (UID: \"b7e83907-c1a2-4761-ab76-31fe664742a3\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" Oct 30 22:17:04.218187 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:04.218157 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7e83907-c1a2-4761-ab76-31fe664742a3-istio-token\") pod \"e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff\" (UID: \"b7e83907-c1a2-4761-ab76-31fe664742a3\") " pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" Oct 30 22:17:04.504711 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:04.504680 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" Oct 30 22:17:04.943048 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:04.943016 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff"] Oct 30 22:17:04.946292 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:17:04.946261 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7e83907_c1a2_4761_ab76_31fe664742a3.slice/crio-792388c3c1ce8c1555d7d844a94e3b29c0c390e9b08ee7c33a0212739e4bf791 WatchSource:0}: Error finding container 792388c3c1ce8c1555d7d844a94e3b29c0c390e9b08ee7c33a0212739e4bf791: Status 404 returned error can't find the container with id 792388c3c1ce8c1555d7d844a94e3b29c0c390e9b08ee7c33a0212739e4bf791 Oct 30 22:17:04.948824 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:04.948790 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:17:04.948901 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:04.948857 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:17:04.948936 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:04.948905 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:17:05.515981 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:05.515952 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh" Oct 30 22:17:05.531188 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:05.531150 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" event={"ID":"b7e83907-c1a2-4761-ab76-31fe664742a3","Type":"ContainerStarted","Data":"19139c8c06fc3b0fd89bfc8af4ffbf9dbcffe7c498b7230c916d9bb182c329a2"} Oct 30 22:17:05.531365 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:05.531229 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" event={"ID":"b7e83907-c1a2-4761-ab76-31fe664742a3","Type":"ContainerStarted","Data":"792388c3c1ce8c1555d7d844a94e3b29c0c390e9b08ee7c33a0212739e4bf791"} Oct 30 22:17:05.821270 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:05.821084 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" podStartSLOduration=2.821064652 podStartE2EDuration="2.821064652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:05.818716313 +0000 UTC m=+1586.089414398" watchObservedRunningTime="2026-10-30 22:17:05.821064652 +0000 UTC m=+1586.091762738" Oct 30 22:17:06.140699 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:06.140610 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4"] Oct 30 22:17:06.141113 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:06.140818 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" podUID="f83c719c-83ad-4b5e-be59-a3242cd7ecaf" containerName="istio-proxy" containerID="cri-o://863bd38fff60a4cf064d4be13aa9928b6109fdf653d3d80ee606e25ec7dc1fda" gracePeriod=30 Oct 30 22:17:06.219098 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:06.219062 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66"] Oct 30 22:17:06.219366 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:06.219340 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" podUID="d68726e2-79a3-4c7d-9c64-c06d73bb3077" containerName="istio-proxy" containerID="cri-o://1e9896693033ee317c063c84f2a59a8f776bd55f8161842fb7df2d92378c707c" gracePeriod=30 Oct 30 22:17:06.225344 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:06.225308 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk"] Oct 30 22:17:06.238454 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:06.238415 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs"] Oct 30 22:17:06.238977 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:06.238904 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" podUID="e2801c3c-999f-41aa-900e-5a8b1c9fd8c0" containerName="istio-proxy" containerID="cri-o://396d223632dcdf588956ff14da792b0bbf4375eb40cec10beaa8b9be86b0dec1" gracePeriod=30 Oct 30 22:17:06.239184 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:06.238980 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" podUID="ba68152b-0571-406a-8355-8fc4d3aae15c" containerName="istio-proxy" containerID="cri-o://bfeec9849da7882ff42c68e26d34aed161cfabafebd6beeafe2ad36ef7f4367c" gracePeriod=30 Oct 30 22:17:06.508246 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:06.507519 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" Oct 30 22:17:06.519365 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:06.519315 5918 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 22:17:06.519518 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:06.519377 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" podUID="b7e83907-c1a2-4761-ab76-31fe664742a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 22:17:06.952241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:06.950271 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx"] Oct 30 22:17:07.512241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:07.511252 5918 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 22:17:07.512241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:07.511319 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" podUID="b7e83907-c1a2-4761-ab76-31fe664742a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 22:17:07.600253 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:07.592596 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" podUID="2a4486c4-7472-4dae-bd5c-2c56cd386fcb" containerName="istio-proxy" containerID="cri-o://3bb6bad07e2be4b71bcd28c20ce968861761a835c469ee4c7b6a2c2fc3eb55ee" gracePeriod=30 Oct 30 22:17:08.329300 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:08.329261 5918 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:17:08.329487 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:08.329335 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" podUID="f83c719c-83ad-4b5e-be59-a3242cd7ecaf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:17:08.511241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:08.505607 5918 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 22:17:08.511241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:08.505666 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" podUID="b7e83907-c1a2-4761-ab76-31fe664742a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 22:17:09.482943 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:09.482907 5918 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:17:09.483345 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:09.482974 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" podUID="ba68152b-0571-406a-8355-8fc4d3aae15c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:17:09.509241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:09.508617 5918 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 22:17:09.509241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:09.508678 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" podUID="b7e83907-c1a2-4761-ab76-31fe664742a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 22:17:10.445168 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:10.445137 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff"] Oct 30 22:17:10.445564 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:10.445539 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" podUID="b7e83907-c1a2-4761-ab76-31fe664742a3" containerName="istio-proxy" containerID="cri-o://19139c8c06fc3b0fd89bfc8af4ffbf9dbcffe7c498b7230c916d9bb182c329a2" gracePeriod=30 Oct 30 22:17:10.466164 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:10.466129 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt"] Oct 30 22:17:10.466598 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:10.466560 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" podUID="44ed0322-9b5b-4751-8f7f-5f0c58493345" containerName="istio-proxy" containerID="cri-o://285b949241428a7d5792f0ad39c044b35b5776a0ef693c8df4c05fe24db47cee" gracePeriod=30 Oct 30 22:17:10.472883 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:10.472842 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l"] Oct 30 22:17:10.473274 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:10.473240 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" podUID="0b4e06dd-69d5-485e-acac-62dddf7f51cb" containerName="istio-proxy" containerID="cri-o://772a26e1786f359420389b4c9ac35a88a19b648c28d75d3c53eb326ac36338f3" gracePeriod=30 Oct 30 22:17:11.572280 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.572208 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" Oct 30 22:17:11.641276 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.636436 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" Oct 30 22:17:11.659375 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.656037 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" Oct 30 22:17:11.683752 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.683716 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-istio-podinfo\") pod \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\" (UID: \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\") " Oct 30 22:17:11.684039 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.684020 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-workload-socket\") pod \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\" (UID: \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\") " Oct 30 22:17:11.684502 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.684458 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d68726e2-79a3-4c7d-9c64-c06d73bb3077-istio-data" pod "d68726e2-79a3-4c7d-9c64-c06d73bb3077" (UID: "d68726e2-79a3-4c7d-9c64-c06d73bb3077"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:11.684697 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.684667 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d68726e2-79a3-4c7d-9c64-c06d73bb3077-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d68726e2-79a3-4c7d-9c64-c06d73bb3077-istio-data\") pod \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\" (UID: \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\") " Oct 30 22:17:11.684829 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.684736 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d68726e2-79a3-4c7d-9c64-c06d73bb3077-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d68726e2-79a3-4c7d-9c64-c06d73bb3077-istio-podinfo\") pod \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\" (UID: \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\") " Oct 30 22:17:11.685241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.684916 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-workload-socket" pod "f83c719c-83ad-4b5e-be59-a3242cd7ecaf" (UID: "f83c719c-83ad-4b5e-be59-a3242cd7ecaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:11.685241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.685140 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d68726e2-79a3-4c7d-9c64-c06d73bb3077-istio-token\" (UniqueName: \"kubernetes.io/projected/d68726e2-79a3-4c7d-9c64-c06d73bb3077-istio-token\") pod \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\" (UID: \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\") " Oct 30 22:17:11.685241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.685187 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d68726e2-79a3-4c7d-9c64-c06d73bb3077-kube-api-access-wlbqn\" (UniqueName: \"kubernetes.io/projected/d68726e2-79a3-4c7d-9c64-c06d73bb3077-kube-api-access-wlbqn\") pod \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\" (UID: \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\") " Oct 30 22:17:11.685485 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.685244 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d68726e2-79a3-4c7d-9c64-c06d73bb3077-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d68726e2-79a3-4c7d-9c64-c06d73bb3077-workload-certs\") pod \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\" (UID: \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\") " Oct 30 22:17:11.685547 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.685503 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d68726e2-79a3-4c7d-9c64-c06d73bb3077-workload-certs" pod "d68726e2-79a3-4c7d-9c64-c06d73bb3077" (UID: "d68726e2-79a3-4c7d-9c64-c06d73bb3077"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:11.685623 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.685609 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-istio-envoy\") pod \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\" (UID: \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\") " Oct 30 22:17:11.685746 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.685644 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d68726e2-79a3-4c7d-9c64-c06d73bb3077-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68726e2-79a3-4c7d-9c64-c06d73bb3077-credential-socket\") pod \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\" (UID: \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\") " Oct 30 22:17:11.685746 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.685714 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-istio-data\") pod \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\" (UID: \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\") " Oct 30 22:17:11.685854 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.685779 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-istio-token\" (UniqueName: \"kubernetes.io/projected/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-istio-token\") pod \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\" (UID: \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\") " Oct 30 22:17:11.685854 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.685818 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d68726e2-79a3-4c7d-9c64-c06d73bb3077-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68726e2-79a3-4c7d-9c64-c06d73bb3077-workload-socket\") pod \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\" (UID: \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\") " Oct 30 22:17:11.685953 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.685898 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-kube-api-access-48snv\" (UniqueName: \"kubernetes.io/projected/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-kube-api-access-48snv\") pod \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\" (UID: \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\") " Oct 30 22:17:11.685953 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.685928 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-workload-certs\") pod \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\" (UID: \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\") " Oct 30 22:17:11.686094 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.686055 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d68726e2-79a3-4c7d-9c64-c06d73bb3077-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d68726e2-79a3-4c7d-9c64-c06d73bb3077-istiod-ca-cert\") pod \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\" (UID: \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\") " Oct 30 22:17:11.686094 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.686057 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-istio-data" pod "f83c719c-83ad-4b5e-be59-a3242cd7ecaf" (UID: "f83c719c-83ad-4b5e-be59-a3242cd7ecaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:11.686236 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.686169 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d68726e2-79a3-4c7d-9c64-c06d73bb3077-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d68726e2-79a3-4c7d-9c64-c06d73bb3077-istio-envoy\") pod \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\" (UID: \"d68726e2-79a3-4c7d-9c64-c06d73bb3077\") " Oct 30 22:17:11.686236 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.686228 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-credential-socket\") pod \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\" (UID: \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\") " Oct 30 22:17:11.686346 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.686248 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d68726e2-79a3-4c7d-9c64-c06d73bb3077-credential-socket" pod "d68726e2-79a3-4c7d-9c64-c06d73bb3077" (UID: "d68726e2-79a3-4c7d-9c64-c06d73bb3077"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:11.686346 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.686273 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-istiod-ca-cert\") pod \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\" (UID: \"f83c719c-83ad-4b5e-be59-a3242cd7ecaf\") " Oct 30 22:17:11.686651 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.686620 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-workload-certs" pod "f83c719c-83ad-4b5e-be59-a3242cd7ecaf" (UID: "f83c719c-83ad-4b5e-be59-a3242cd7ecaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:11.687092 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.687067 5918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-workload-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.687186 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.687098 5918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-workload-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.687186 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.687113 5918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d68726e2-79a3-4c7d-9c64-c06d73bb3077-istio-data\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.687186 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.687127 5918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d68726e2-79a3-4c7d-9c64-c06d73bb3077-workload-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.687186 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.687142 5918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68726e2-79a3-4c7d-9c64-c06d73bb3077-credential-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.687186 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.687155 5918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-istio-data\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.687557 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.687531 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d68726e2-79a3-4c7d-9c64-c06d73bb3077-istiod-ca-cert" pod "d68726e2-79a3-4c7d-9c64-c06d73bb3077" (UID: "d68726e2-79a3-4c7d-9c64-c06d73bb3077"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:11.688138 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.688080 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d68726e2-79a3-4c7d-9c64-c06d73bb3077-kube-api-access-wlbqn" pod "d68726e2-79a3-4c7d-9c64-c06d73bb3077" (UID: "d68726e2-79a3-4c7d-9c64-c06d73bb3077"). InnerVolumeSpecName "kube-api-access-wlbqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:11.688670 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.687974 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d68726e2-79a3-4c7d-9c64-c06d73bb3077-workload-socket" pod "d68726e2-79a3-4c7d-9c64-c06d73bb3077" (UID: "d68726e2-79a3-4c7d-9c64-c06d73bb3077"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:11.689303 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.689044 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-credential-socket" pod "f83c719c-83ad-4b5e-be59-a3242cd7ecaf" (UID: "f83c719c-83ad-4b5e-be59-a3242cd7ecaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:11.689421 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.689334 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-istiod-ca-cert" pod "f83c719c-83ad-4b5e-be59-a3242cd7ecaf" (UID: "f83c719c-83ad-4b5e-be59-a3242cd7ecaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:11.690135 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.690080 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d68726e2-79a3-4c7d-9c64-c06d73bb3077-istio-envoy" pod "d68726e2-79a3-4c7d-9c64-c06d73bb3077" (UID: "d68726e2-79a3-4c7d-9c64-c06d73bb3077"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:11.691056 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.690875 5918 generic.go:356] "Generic (PLEG): container finished" podID="e2801c3c-999f-41aa-900e-5a8b1c9fd8c0" containerID="396d223632dcdf588956ff14da792b0bbf4375eb40cec10beaa8b9be86b0dec1" exitCode=0 Oct 30 22:17:11.691056 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.690968 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" event={"ID":"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0","Type":"ContainerDied","Data":"396d223632dcdf588956ff14da792b0bbf4375eb40cec10beaa8b9be86b0dec1"} Oct 30 22:17:11.692640 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.692468 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-kube-api-access-48snv" pod "f83c719c-83ad-4b5e-be59-a3242cd7ecaf" (UID: "f83c719c-83ad-4b5e-be59-a3242cd7ecaf"). InnerVolumeSpecName "kube-api-access-48snv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:11.694302 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.693946 5918 generic.go:356] "Generic (PLEG): container finished" podID="f83c719c-83ad-4b5e-be59-a3242cd7ecaf" containerID="863bd38fff60a4cf064d4be13aa9928b6109fdf653d3d80ee606e25ec7dc1fda" exitCode=0 Oct 30 22:17:11.694302 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.694039 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" event={"ID":"f83c719c-83ad-4b5e-be59-a3242cd7ecaf","Type":"ContainerDied","Data":"863bd38fff60a4cf064d4be13aa9928b6109fdf653d3d80ee606e25ec7dc1fda"} Oct 30 22:17:11.694302 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.694082 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" event={"ID":"f83c719c-83ad-4b5e-be59-a3242cd7ecaf","Type":"ContainerDied","Data":"41f10dc257690b86511682ccb66c43ce553f0db882b031edd29026e101e5dd8f"} Oct 30 22:17:11.694302 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.694113 5918 scope.go:122] "RemoveContainer" containerID="863bd38fff60a4cf064d4be13aa9928b6109fdf653d3d80ee606e25ec7dc1fda" Oct 30 22:17:11.694302 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.694042 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4" Oct 30 22:17:11.697162 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.697141 5918 generic.go:356] "Generic (PLEG): container finished" podID="ba68152b-0571-406a-8355-8fc4d3aae15c" containerID="bfeec9849da7882ff42c68e26d34aed161cfabafebd6beeafe2ad36ef7f4367c" exitCode=0 Oct 30 22:17:11.697567 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.697413 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" Oct 30 22:17:11.697681 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.697666 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" event={"ID":"ba68152b-0571-406a-8355-8fc4d3aae15c","Type":"ContainerDied","Data":"bfeec9849da7882ff42c68e26d34aed161cfabafebd6beeafe2ad36ef7f4367c"} Oct 30 22:17:11.697745 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.697714 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs" event={"ID":"ba68152b-0571-406a-8355-8fc4d3aae15c","Type":"ContainerDied","Data":"b7bb40545b2f3fff0babd928a4125ebb6757cb725bfa4aa2afc675944f532320"} Oct 30 22:17:11.699521 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.699421 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d68726e2-79a3-4c7d-9c64-c06d73bb3077-istio-token" pod "d68726e2-79a3-4c7d-9c64-c06d73bb3077" (UID: "d68726e2-79a3-4c7d-9c64-c06d73bb3077"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:11.701204 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.699731 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d68726e2-79a3-4c7d-9c64-c06d73bb3077-istio-podinfo" pod "d68726e2-79a3-4c7d-9c64-c06d73bb3077" (UID: "d68726e2-79a3-4c7d-9c64-c06d73bb3077"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:17:11.701862 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.701433 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-istio-podinfo" pod "f83c719c-83ad-4b5e-be59-a3242cd7ecaf" (UID: "f83c719c-83ad-4b5e-be59-a3242cd7ecaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:17:11.701862 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.701467 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-istio-token" pod "f83c719c-83ad-4b5e-be59-a3242cd7ecaf" (UID: "f83c719c-83ad-4b5e-be59-a3242cd7ecaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:11.703295 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.703169 5918 generic.go:356] "Generic (PLEG): container finished" podID="d68726e2-79a3-4c7d-9c64-c06d73bb3077" containerID="1e9896693033ee317c063c84f2a59a8f776bd55f8161842fb7df2d92378c707c" exitCode=0 Oct 30 22:17:11.703911 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.703527 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" event={"ID":"d68726e2-79a3-4c7d-9c64-c06d73bb3077","Type":"ContainerDied","Data":"1e9896693033ee317c063c84f2a59a8f776bd55f8161842fb7df2d92378c707c"} Oct 30 22:17:11.703911 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.703577 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" event={"ID":"d68726e2-79a3-4c7d-9c64-c06d73bb3077","Type":"ContainerDied","Data":"d21273178842a99f9e85808d255b5aaf67abc44d805114006057e5bbca9bbe02"} Oct 30 22:17:11.703911 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.703648 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66" Oct 30 22:17:11.713593 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.713561 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-istio-envoy" pod "f83c719c-83ad-4b5e-be59-a3242cd7ecaf" (UID: "f83c719c-83ad-4b5e-be59-a3242cd7ecaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:11.714462 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.714439 5918 scope.go:122] "RemoveContainer" containerID="863bd38fff60a4cf064d4be13aa9928b6109fdf653d3d80ee606e25ec7dc1fda" Oct 30 22:17:11.716697 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:17:11.716357 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"863bd38fff60a4cf064d4be13aa9928b6109fdf653d3d80ee606e25ec7dc1fda\": container with ID starting with 863bd38fff60a4cf064d4be13aa9928b6109fdf653d3d80ee606e25ec7dc1fda not found: ID does not exist" containerID="863bd38fff60a4cf064d4be13aa9928b6109fdf653d3d80ee606e25ec7dc1fda" Oct 30 22:17:11.716697 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.716394 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"863bd38fff60a4cf064d4be13aa9928b6109fdf653d3d80ee606e25ec7dc1fda"} err="failed to get container status \"863bd38fff60a4cf064d4be13aa9928b6109fdf653d3d80ee606e25ec7dc1fda\": rpc error: code = NotFound desc = could not find container \"863bd38fff60a4cf064d4be13aa9928b6109fdf653d3d80ee606e25ec7dc1fda\": container with ID starting with 863bd38fff60a4cf064d4be13aa9928b6109fdf653d3d80ee606e25ec7dc1fda not found: ID does not exist" Oct 30 22:17:11.716697 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.716417 5918 scope.go:122] "RemoveContainer" containerID="bfeec9849da7882ff42c68e26d34aed161cfabafebd6beeafe2ad36ef7f4367c" Oct 30 22:17:11.730431 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.730334 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" Oct 30 22:17:11.732110 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.732066 5918 scope.go:122] "RemoveContainer" containerID="bfeec9849da7882ff42c68e26d34aed161cfabafebd6beeafe2ad36ef7f4367c" Oct 30 22:17:11.732505 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:17:11.732484 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfeec9849da7882ff42c68e26d34aed161cfabafebd6beeafe2ad36ef7f4367c\": container with ID starting with bfeec9849da7882ff42c68e26d34aed161cfabafebd6beeafe2ad36ef7f4367c not found: ID does not exist" containerID="bfeec9849da7882ff42c68e26d34aed161cfabafebd6beeafe2ad36ef7f4367c" Oct 30 22:17:11.732590 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.732517 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfeec9849da7882ff42c68e26d34aed161cfabafebd6beeafe2ad36ef7f4367c"} err="failed to get container status \"bfeec9849da7882ff42c68e26d34aed161cfabafebd6beeafe2ad36ef7f4367c\": rpc error: code = NotFound desc = could not find container \"bfeec9849da7882ff42c68e26d34aed161cfabafebd6beeafe2ad36ef7f4367c\": container with ID starting with bfeec9849da7882ff42c68e26d34aed161cfabafebd6beeafe2ad36ef7f4367c not found: ID does not exist" Oct 30 22:17:11.732590 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.732540 5918 scope.go:122] "RemoveContainer" containerID="1e9896693033ee317c063c84f2a59a8f776bd55f8161842fb7df2d92378c707c" Oct 30 22:17:11.745613 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.745550 5918 scope.go:122] "RemoveContainer" containerID="1e9896693033ee317c063c84f2a59a8f776bd55f8161842fb7df2d92378c707c" Oct 30 22:17:11.746118 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:17:11.746082 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e9896693033ee317c063c84f2a59a8f776bd55f8161842fb7df2d92378c707c\": container with ID starting with 1e9896693033ee317c063c84f2a59a8f776bd55f8161842fb7df2d92378c707c not found: ID does not exist" containerID="1e9896693033ee317c063c84f2a59a8f776bd55f8161842fb7df2d92378c707c" Oct 30 22:17:11.746309 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.746275 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e9896693033ee317c063c84f2a59a8f776bd55f8161842fb7df2d92378c707c"} err="failed to get container status \"1e9896693033ee317c063c84f2a59a8f776bd55f8161842fb7df2d92378c707c\": rpc error: code = NotFound desc = could not find container \"1e9896693033ee317c063c84f2a59a8f776bd55f8161842fb7df2d92378c707c\": container with ID starting with 1e9896693033ee317c063c84f2a59a8f776bd55f8161842fb7df2d92378c707c not found: ID does not exist" Oct 30 22:17:11.787794 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.787765 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-workload-socket\") pod \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\" (UID: \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\") " Oct 30 22:17:11.787940 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.787830 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-istio-envoy\") pod \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\" (UID: \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\") " Oct 30 22:17:11.787940 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.787875 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba68152b-0571-406a-8355-8fc4d3aae15c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba68152b-0571-406a-8355-8fc4d3aae15c-istiod-ca-cert\") pod \"ba68152b-0571-406a-8355-8fc4d3aae15c\" (UID: \"ba68152b-0571-406a-8355-8fc4d3aae15c\") " Oct 30 22:17:11.787940 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.787932 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-istiod-ca-cert\") pod \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\" (UID: \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\") " Oct 30 22:17:11.788121 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.787971 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-workload-certs\") pod \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\" (UID: \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\") " Oct 30 22:17:11.788121 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.788010 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba68152b-0571-406a-8355-8fc4d3aae15c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba68152b-0571-406a-8355-8fc4d3aae15c-istio-envoy\") pod \"ba68152b-0571-406a-8355-8fc4d3aae15c\" (UID: \"ba68152b-0571-406a-8355-8fc4d3aae15c\") " Oct 30 22:17:11.788121 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.788054 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba68152b-0571-406a-8355-8fc4d3aae15c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba68152b-0571-406a-8355-8fc4d3aae15c-workload-certs\") pod \"ba68152b-0571-406a-8355-8fc4d3aae15c\" (UID: \"ba68152b-0571-406a-8355-8fc4d3aae15c\") " Oct 30 22:17:11.788121 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.788072 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-workload-socket" pod "e2801c3c-999f-41aa-900e-5a8b1c9fd8c0" (UID: "e2801c3c-999f-41aa-900e-5a8b1c9fd8c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:11.788399 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.788133 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba68152b-0571-406a-8355-8fc4d3aae15c-istio-token\" (UniqueName: \"kubernetes.io/projected/ba68152b-0571-406a-8355-8fc4d3aae15c-istio-token\") pod \"ba68152b-0571-406a-8355-8fc4d3aae15c\" (UID: \"ba68152b-0571-406a-8355-8fc4d3aae15c\") " Oct 30 22:17:11.788399 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.788167 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-istio-token\" (UniqueName: \"kubernetes.io/projected/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-istio-token\") pod \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\" (UID: \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\") " Oct 30 22:17:11.788399 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.788204 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba68152b-0571-406a-8355-8fc4d3aae15c-kube-api-access-fcgqj\" (UniqueName: \"kubernetes.io/projected/ba68152b-0571-406a-8355-8fc4d3aae15c-kube-api-access-fcgqj\") pod \"ba68152b-0571-406a-8355-8fc4d3aae15c\" (UID: \"ba68152b-0571-406a-8355-8fc4d3aae15c\") " Oct 30 22:17:11.788399 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.788272 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-istio-podinfo\") pod \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\" (UID: \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\") " Oct 30 22:17:11.788399 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.788324 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-workload-certs" pod "e2801c3c-999f-41aa-900e-5a8b1c9fd8c0" (UID: "e2801c3c-999f-41aa-900e-5a8b1c9fd8c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:11.788399 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.788333 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba68152b-0571-406a-8355-8fc4d3aae15c-istiod-ca-cert" pod "ba68152b-0571-406a-8355-8fc4d3aae15c" (UID: "ba68152b-0571-406a-8355-8fc4d3aae15c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:11.788399 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.788343 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-credential-socket\") pod \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\" (UID: \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\") " Oct 30 22:17:11.788897 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.788406 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-istio-data\") pod \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\" (UID: \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\") " Oct 30 22:17:11.788897 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.788419 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-istiod-ca-cert" pod "e2801c3c-999f-41aa-900e-5a8b1c9fd8c0" (UID: "e2801c3c-999f-41aa-900e-5a8b1c9fd8c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:11.788897 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.788542 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-credential-socket" pod "e2801c3c-999f-41aa-900e-5a8b1c9fd8c0" (UID: "e2801c3c-999f-41aa-900e-5a8b1c9fd8c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:11.788897 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.788583 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-kube-api-access-f5bmk\" (UniqueName: \"kubernetes.io/projected/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-kube-api-access-f5bmk\") pod \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\" (UID: \"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0\") " Oct 30 22:17:11.788897 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.788598 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba68152b-0571-406a-8355-8fc4d3aae15c-workload-certs" pod "ba68152b-0571-406a-8355-8fc4d3aae15c" (UID: "ba68152b-0571-406a-8355-8fc4d3aae15c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:11.788897 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.788642 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba68152b-0571-406a-8355-8fc4d3aae15c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba68152b-0571-406a-8355-8fc4d3aae15c-istio-podinfo\") pod \"ba68152b-0571-406a-8355-8fc4d3aae15c\" (UID: \"ba68152b-0571-406a-8355-8fc4d3aae15c\") " Oct 30 22:17:11.788897 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.788684 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba68152b-0571-406a-8355-8fc4d3aae15c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba68152b-0571-406a-8355-8fc4d3aae15c-credential-socket\") pod \"ba68152b-0571-406a-8355-8fc4d3aae15c\" (UID: \"ba68152b-0571-406a-8355-8fc4d3aae15c\") " Oct 30 22:17:11.788897 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.788721 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba68152b-0571-406a-8355-8fc4d3aae15c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba68152b-0571-406a-8355-8fc4d3aae15c-istio-data\") pod \"ba68152b-0571-406a-8355-8fc4d3aae15c\" (UID: \"ba68152b-0571-406a-8355-8fc4d3aae15c\") " Oct 30 22:17:11.788897 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.788790 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba68152b-0571-406a-8355-8fc4d3aae15c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba68152b-0571-406a-8355-8fc4d3aae15c-workload-socket\") pod \"ba68152b-0571-406a-8355-8fc4d3aae15c\" (UID: \"ba68152b-0571-406a-8355-8fc4d3aae15c\") " Oct 30 22:17:11.789356 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.789043 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-istio-data" pod "e2801c3c-999f-41aa-900e-5a8b1c9fd8c0" (UID: "e2801c3c-999f-41aa-900e-5a8b1c9fd8c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:11.789356 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.789234 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba68152b-0571-406a-8355-8fc4d3aae15c-workload-socket" pod "ba68152b-0571-406a-8355-8fc4d3aae15c" (UID: "ba68152b-0571-406a-8355-8fc4d3aae15c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:11.789904 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.789879 5918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d68726e2-79a3-4c7d-9c64-c06d73bb3077-istio-envoy\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.790005 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.789909 5918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-credential-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.790005 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.789925 5918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-istiod-ca-cert\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.790005 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.789940 5918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-istio-podinfo\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.790005 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.789955 5918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba68152b-0571-406a-8355-8fc4d3aae15c-workload-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.790005 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.789969 5918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-workload-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.790005 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.789984 5918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d68726e2-79a3-4c7d-9c64-c06d73bb3077-istio-podinfo\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.790005 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.789997 5918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d68726e2-79a3-4c7d-9c64-c06d73bb3077-istio-token\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.790397 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.790012 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlbqn\" (UniqueName: \"kubernetes.io/projected/d68726e2-79a3-4c7d-9c64-c06d73bb3077-kube-api-access-wlbqn\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.790397 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.790027 5918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba68152b-0571-406a-8355-8fc4d3aae15c-istiod-ca-cert\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.790397 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.790041 5918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-istio-envoy\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.790397 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.790054 5918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-istiod-ca-cert\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.790397 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.790068 5918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-istio-token\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.790397 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.790085 5918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-workload-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.790397 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.790100 5918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68726e2-79a3-4c7d-9c64-c06d73bb3077-workload-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.790397 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.790113 5918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba68152b-0571-406a-8355-8fc4d3aae15c-workload-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.790397 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.790128 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48snv\" (UniqueName: \"kubernetes.io/projected/f83c719c-83ad-4b5e-be59-a3242cd7ecaf-kube-api-access-48snv\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.790397 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.790144 5918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d68726e2-79a3-4c7d-9c64-c06d73bb3077-istiod-ca-cert\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.790397 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.790159 5918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-credential-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.790397 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.790173 5918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-istio-data\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.790397 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.790345 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba68152b-0571-406a-8355-8fc4d3aae15c-credential-socket" pod "ba68152b-0571-406a-8355-8fc4d3aae15c" (UID: "ba68152b-0571-406a-8355-8fc4d3aae15c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:11.790397 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.790394 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba68152b-0571-406a-8355-8fc4d3aae15c-istio-data" pod "ba68152b-0571-406a-8355-8fc4d3aae15c" (UID: "ba68152b-0571-406a-8355-8fc4d3aae15c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:11.791062 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.790790 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-istio-envoy" pod "e2801c3c-999f-41aa-900e-5a8b1c9fd8c0" (UID: "e2801c3c-999f-41aa-900e-5a8b1c9fd8c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:11.791878 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.791851 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-istio-token" pod "e2801c3c-999f-41aa-900e-5a8b1c9fd8c0" (UID: "e2801c3c-999f-41aa-900e-5a8b1c9fd8c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:11.792019 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.791878 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba68152b-0571-406a-8355-8fc4d3aae15c-istio-token" pod "ba68152b-0571-406a-8355-8fc4d3aae15c" (UID: "ba68152b-0571-406a-8355-8fc4d3aae15c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:11.792019 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.792003 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-kube-api-access-f5bmk" pod "e2801c3c-999f-41aa-900e-5a8b1c9fd8c0" (UID: "e2801c3c-999f-41aa-900e-5a8b1c9fd8c0"). InnerVolumeSpecName "kube-api-access-f5bmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:11.792533 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.792503 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba68152b-0571-406a-8355-8fc4d3aae15c-istio-podinfo" pod "ba68152b-0571-406a-8355-8fc4d3aae15c" (UID: "ba68152b-0571-406a-8355-8fc4d3aae15c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:17:11.792625 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.792539 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-istio-podinfo" pod "e2801c3c-999f-41aa-900e-5a8b1c9fd8c0" (UID: "e2801c3c-999f-41aa-900e-5a8b1c9fd8c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:17:11.792954 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.792937 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba68152b-0571-406a-8355-8fc4d3aae15c-istio-envoy" pod "ba68152b-0571-406a-8355-8fc4d3aae15c" (UID: "ba68152b-0571-406a-8355-8fc4d3aae15c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:11.793722 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.793699 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba68152b-0571-406a-8355-8fc4d3aae15c-kube-api-access-fcgqj" pod "ba68152b-0571-406a-8355-8fc4d3aae15c" (UID: "ba68152b-0571-406a-8355-8fc4d3aae15c"). InnerVolumeSpecName "kube-api-access-fcgqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:11.813812 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.813427 5918 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:17:11.813812 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.813496 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" podUID="44ed0322-9b5b-4751-8f7f-5f0c58493345" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:17:11.892192 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.892158 5918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-istio-envoy\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.892336 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.892194 5918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba68152b-0571-406a-8355-8fc4d3aae15c-istio-envoy\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.892336 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.892229 5918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba68152b-0571-406a-8355-8fc4d3aae15c-istio-token\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.892336 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.892244 5918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-istio-token\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.892336 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.892259 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcgqj\" (UniqueName: \"kubernetes.io/projected/ba68152b-0571-406a-8355-8fc4d3aae15c-kube-api-access-fcgqj\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.892336 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.892273 5918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-istio-podinfo\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.892336 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.892289 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5bmk\" (UniqueName: \"kubernetes.io/projected/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0-kube-api-access-f5bmk\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.892336 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.892301 5918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba68152b-0571-406a-8355-8fc4d3aae15c-istio-podinfo\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.892336 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.892313 5918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba68152b-0571-406a-8355-8fc4d3aae15c-credential-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.892336 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:11.892325 5918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba68152b-0571-406a-8355-8fc4d3aae15c-istio-data\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:12.196981 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:12.182085 5918 patch_prober.go:29] interesting pod/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:17:12.196981 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:12.182161 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" podUID="0b4e06dd-69d5-485e-acac-62dddf7f51cb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:17:12.277763 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:12.277690 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs"] Oct 30 22:17:12.288508 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:12.288471 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-jnjqs"] Oct 30 22:17:12.403240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:12.402784 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4"] Oct 30 22:17:12.408187 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:12.408158 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-5krk4"] Oct 30 22:17:12.519936 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:12.519903 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66"] Oct 30 22:17:12.527858 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:12.527812 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-tcd66"] Oct 30 22:17:12.674792 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:12.673440 5918 patch_prober.go:29] interesting pod/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:17:12.674792 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:12.673520 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" podUID="e2801c3c-999f-41aa-900e-5a8b1c9fd8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:17:12.714644 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:12.714612 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" event={"ID":"e2801c3c-999f-41aa-900e-5a8b1c9fd8c0","Type":"ContainerDied","Data":"f55ceab8e728c734fdbc6f84dc64853ecccf893d2702d2d8b8f1fb2a375da963"} Oct 30 22:17:12.714835 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:12.714686 5918 scope.go:122] "RemoveContainer" containerID="396d223632dcdf588956ff14da792b0bbf4375eb40cec10beaa8b9be86b0dec1" Oct 30 22:17:12.714835 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:12.714620 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk" Oct 30 22:17:12.722465 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:12.722433 5918 generic.go:356] "Generic (PLEG): container finished" podID="2a4486c4-7472-4dae-bd5c-2c56cd386fcb" containerID="3bb6bad07e2be4b71bcd28c20ce968861761a835c469ee4c7b6a2c2fc3eb55ee" exitCode=0 Oct 30 22:17:12.722597 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:12.722517 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" event={"ID":"2a4486c4-7472-4dae-bd5c-2c56cd386fcb","Type":"ContainerDied","Data":"3bb6bad07e2be4b71bcd28c20ce968861761a835c469ee4c7b6a2c2fc3eb55ee"} Oct 30 22:17:12.861988 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:12.861965 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" Oct 30 22:17:12.898191 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:12.898131 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk"] Oct 30 22:17:12.901954 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:12.901929 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3783c2-openshift-default-7c54dd88dc-fhqqk"] Oct 30 22:17:12.929354 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:12.929315 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-istio-podinfo\") pod \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\" (UID: \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\") " Oct 30 22:17:12.929538 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:12.929431 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-istiod-ca-cert\") pod \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\" (UID: \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\") " Oct 30 22:17:12.929641 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:12.929621 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-istio-data\") pod \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\" (UID: \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\") " Oct 30 22:17:12.929713 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:12.929697 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-istio-envoy\") pod \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\" (UID: \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\") " Oct 30 22:17:12.929777 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:12.929756 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-workload-certs\") pod \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\" (UID: \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\") " Oct 30 22:17:12.929839 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:12.929827 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-workload-socket\") pod \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\" (UID: \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\") " Oct 30 22:17:12.929921 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:12.929905 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-credential-socket\") pod \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\" (UID: \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\") " Oct 30 22:17:12.929986 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:12.929949 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-istio-token\" (UniqueName: \"kubernetes.io/projected/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-istio-token\") pod \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\" (UID: \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\") " Oct 30 22:17:12.930041 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:12.929979 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-kube-api-access-ctmmv\" (UniqueName: \"kubernetes.io/projected/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-kube-api-access-ctmmv\") pod \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\" (UID: \"2a4486c4-7472-4dae-bd5c-2c56cd386fcb\") " Oct 30 22:17:12.936373 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:12.936337 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-credential-socket" pod "2a4486c4-7472-4dae-bd5c-2c56cd386fcb" (UID: "2a4486c4-7472-4dae-bd5c-2c56cd386fcb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:12.936606 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:12.936573 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-workload-socket" pod "2a4486c4-7472-4dae-bd5c-2c56cd386fcb" (UID: "2a4486c4-7472-4dae-bd5c-2c56cd386fcb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:12.937073 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:12.937040 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-istio-data" pod "2a4486c4-7472-4dae-bd5c-2c56cd386fcb" (UID: "2a4486c4-7472-4dae-bd5c-2c56cd386fcb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:12.937384 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:12.937355 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-istiod-ca-cert" pod "2a4486c4-7472-4dae-bd5c-2c56cd386fcb" (UID: "2a4486c4-7472-4dae-bd5c-2c56cd386fcb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:12.939260 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:12.937825 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-workload-certs" pod "2a4486c4-7472-4dae-bd5c-2c56cd386fcb" (UID: "2a4486c4-7472-4dae-bd5c-2c56cd386fcb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:12.951537 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:12.951483 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-istio-envoy" pod "2a4486c4-7472-4dae-bd5c-2c56cd386fcb" (UID: "2a4486c4-7472-4dae-bd5c-2c56cd386fcb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:12.951537 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:12.951508 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-istio-podinfo" pod "2a4486c4-7472-4dae-bd5c-2c56cd386fcb" (UID: "2a4486c4-7472-4dae-bd5c-2c56cd386fcb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:17:12.951687 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:12.951645 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-kube-api-access-ctmmv" pod "2a4486c4-7472-4dae-bd5c-2c56cd386fcb" (UID: "2a4486c4-7472-4dae-bd5c-2c56cd386fcb"). InnerVolumeSpecName "kube-api-access-ctmmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:12.952554 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:12.952524 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-istio-token" pod "2a4486c4-7472-4dae-bd5c-2c56cd386fcb" (UID: "2a4486c4-7472-4dae-bd5c-2c56cd386fcb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:13.040793 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:13.040644 5918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-istio-podinfo\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:13.040793 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:13.040677 5918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-istiod-ca-cert\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:13.040793 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:13.040691 5918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-istio-data\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:13.040793 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:13.040704 5918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-istio-envoy\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:13.040793 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:13.040717 5918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-workload-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:13.040793 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:13.040730 5918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-workload-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:13.040793 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:13.040743 5918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-credential-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:13.040793 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:13.040755 5918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-istio-token\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:13.040793 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:13.040767 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctmmv\" (UniqueName: \"kubernetes.io/projected/2a4486c4-7472-4dae-bd5c-2c56cd386fcb-kube-api-access-ctmmv\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:13.735041 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:13.735008 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" Oct 30 22:17:13.735041 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:13.735027 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" event={"ID":"2a4486c4-7472-4dae-bd5c-2c56cd386fcb","Type":"ContainerDied","Data":"8699ff23ff5e33b999dee0171b95546ba20ab189c67f61e7ace4ddd3a6fd5524"} Oct 30 22:17:13.735593 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:13.735133 5918 scope.go:122] "RemoveContainer" containerID="3bb6bad07e2be4b71bcd28c20ce968861761a835c469ee4c7b6a2c2fc3eb55ee" Oct 30 22:17:13.834611 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:13.834561 5918 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:17:13.834796 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:13.834636 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx" podUID="2a4486c4-7472-4dae-bd5c-2c56cd386fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:17:13.861081 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:13.861026 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba68152b-0571-406a-8355-8fc4d3aae15c" path="/var/lib/kubelet/pods/ba68152b-0571-406a-8355-8fc4d3aae15c/volumes" Oct 30 22:17:13.861887 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:13.861864 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d68726e2-79a3-4c7d-9c64-c06d73bb3077" path="/var/lib/kubelet/pods/d68726e2-79a3-4c7d-9c64-c06d73bb3077/volumes" Oct 30 22:17:13.862555 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:13.862530 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2801c3c-999f-41aa-900e-5a8b1c9fd8c0" path="/var/lib/kubelet/pods/e2801c3c-999f-41aa-900e-5a8b1c9fd8c0/volumes" Oct 30 22:17:13.863554 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:13.863533 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f83c719c-83ad-4b5e-be59-a3242cd7ecaf" path="/var/lib/kubelet/pods/f83c719c-83ad-4b5e-be59-a3242cd7ecaf/volumes" Oct 30 22:17:13.907275 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:13.907237 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx"] Oct 30 22:17:13.946099 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:13.946066 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-t8tlx"] Oct 30 22:17:15.762699 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.762662 5918 generic.go:356] "Generic (PLEG): container finished" podID="44ed0322-9b5b-4751-8f7f-5f0c58493345" containerID="285b949241428a7d5792f0ad39c044b35b5776a0ef693c8df4c05fe24db47cee" exitCode=0 Oct 30 22:17:15.763054 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.762735 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" event={"ID":"44ed0322-9b5b-4751-8f7f-5f0c58493345","Type":"ContainerDied","Data":"285b949241428a7d5792f0ad39c044b35b5776a0ef693c8df4c05fe24db47cee"} Oct 30 22:17:15.763054 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.762791 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" event={"ID":"44ed0322-9b5b-4751-8f7f-5f0c58493345","Type":"ContainerDied","Data":"f1a0c0a252e47c4ae252c56b505e9dfad43f659e5235ee323c61d89cfaec8e87"} Oct 30 22:17:15.763054 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.762809 5918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1a0c0a252e47c4ae252c56b505e9dfad43f659e5235ee323c61d89cfaec8e87" Oct 30 22:17:15.764531 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.764511 5918 generic.go:356] "Generic (PLEG): container finished" podID="0b4e06dd-69d5-485e-acac-62dddf7f51cb" containerID="772a26e1786f359420389b4c9ac35a88a19b648c28d75d3c53eb326ac36338f3" exitCode=0 Oct 30 22:17:15.764649 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.764577 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" event={"ID":"0b4e06dd-69d5-485e-acac-62dddf7f51cb","Type":"ContainerDied","Data":"772a26e1786f359420389b4c9ac35a88a19b648c28d75d3c53eb326ac36338f3"} Oct 30 22:17:15.764999 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.764986 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" Oct 30 22:17:15.799753 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.799729 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" Oct 30 22:17:15.872797 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.872766 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a4486c4-7472-4dae-bd5c-2c56cd386fcb" path="/var/lib/kubelet/pods/2a4486c4-7472-4dae-bd5c-2c56cd386fcb/volumes" Oct 30 22:17:15.878496 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.878204 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44ed0322-9b5b-4751-8f7f-5f0c58493345-istio-token\" (UniqueName: \"kubernetes.io/projected/44ed0322-9b5b-4751-8f7f-5f0c58493345-istio-token\") pod \"44ed0322-9b5b-4751-8f7f-5f0c58493345\" (UID: \"44ed0322-9b5b-4751-8f7f-5f0c58493345\") " Oct 30 22:17:15.878496 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.878336 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44ed0322-9b5b-4751-8f7f-5f0c58493345-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44ed0322-9b5b-4751-8f7f-5f0c58493345-istio-podinfo\") pod \"44ed0322-9b5b-4751-8f7f-5f0c58493345\" (UID: \"44ed0322-9b5b-4751-8f7f-5f0c58493345\") " Oct 30 22:17:15.878496 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.878434 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44ed0322-9b5b-4751-8f7f-5f0c58493345-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44ed0322-9b5b-4751-8f7f-5f0c58493345-istiod-ca-cert\") pod \"44ed0322-9b5b-4751-8f7f-5f0c58493345\" (UID: \"44ed0322-9b5b-4751-8f7f-5f0c58493345\") " Oct 30 22:17:15.878496 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.878484 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44ed0322-9b5b-4751-8f7f-5f0c58493345-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ed0322-9b5b-4751-8f7f-5f0c58493345-workload-socket\") pod \"44ed0322-9b5b-4751-8f7f-5f0c58493345\" (UID: \"44ed0322-9b5b-4751-8f7f-5f0c58493345\") " Oct 30 22:17:15.878828 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.878543 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44ed0322-9b5b-4751-8f7f-5f0c58493345-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44ed0322-9b5b-4751-8f7f-5f0c58493345-istio-data\") pod \"44ed0322-9b5b-4751-8f7f-5f0c58493345\" (UID: \"44ed0322-9b5b-4751-8f7f-5f0c58493345\") " Oct 30 22:17:15.878828 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.878572 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44ed0322-9b5b-4751-8f7f-5f0c58493345-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44ed0322-9b5b-4751-8f7f-5f0c58493345-workload-certs\") pod \"44ed0322-9b5b-4751-8f7f-5f0c58493345\" (UID: \"44ed0322-9b5b-4751-8f7f-5f0c58493345\") " Oct 30 22:17:15.878828 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.878613 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44ed0322-9b5b-4751-8f7f-5f0c58493345-kube-api-access-k8d9b\" (UniqueName: \"kubernetes.io/projected/44ed0322-9b5b-4751-8f7f-5f0c58493345-kube-api-access-k8d9b\") pod \"44ed0322-9b5b-4751-8f7f-5f0c58493345\" (UID: \"44ed0322-9b5b-4751-8f7f-5f0c58493345\") " Oct 30 22:17:15.878996 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.878876 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44ed0322-9b5b-4751-8f7f-5f0c58493345-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ed0322-9b5b-4751-8f7f-5f0c58493345-credential-socket\") pod \"44ed0322-9b5b-4751-8f7f-5f0c58493345\" (UID: \"44ed0322-9b5b-4751-8f7f-5f0c58493345\") " Oct 30 22:17:15.878996 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.878924 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44ed0322-9b5b-4751-8f7f-5f0c58493345-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44ed0322-9b5b-4751-8f7f-5f0c58493345-istio-envoy\") pod \"44ed0322-9b5b-4751-8f7f-5f0c58493345\" (UID: \"44ed0322-9b5b-4751-8f7f-5f0c58493345\") " Oct 30 22:17:15.878996 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.878923 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ed0322-9b5b-4751-8f7f-5f0c58493345-istio-data" pod "44ed0322-9b5b-4751-8f7f-5f0c58493345" (UID: "44ed0322-9b5b-4751-8f7f-5f0c58493345"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:15.879266 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.879237 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ed0322-9b5b-4751-8f7f-5f0c58493345-workload-socket" pod "44ed0322-9b5b-4751-8f7f-5f0c58493345" (UID: "44ed0322-9b5b-4751-8f7f-5f0c58493345"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:15.879266 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.879254 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ed0322-9b5b-4751-8f7f-5f0c58493345-credential-socket" pod "44ed0322-9b5b-4751-8f7f-5f0c58493345" (UID: "44ed0322-9b5b-4751-8f7f-5f0c58493345"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:15.879532 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.879510 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ed0322-9b5b-4751-8f7f-5f0c58493345-workload-certs" pod "44ed0322-9b5b-4751-8f7f-5f0c58493345" (UID: "44ed0322-9b5b-4751-8f7f-5f0c58493345"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:15.880080 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.880055 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44ed0322-9b5b-4751-8f7f-5f0c58493345-istiod-ca-cert" pod "44ed0322-9b5b-4751-8f7f-5f0c58493345" (UID: "44ed0322-9b5b-4751-8f7f-5f0c58493345"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:15.880201 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.880116 5918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ed0322-9b5b-4751-8f7f-5f0c58493345-credential-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:15.880201 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.880134 5918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ed0322-9b5b-4751-8f7f-5f0c58493345-workload-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:15.880201 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.880148 5918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44ed0322-9b5b-4751-8f7f-5f0c58493345-istio-data\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:15.880201 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.880156 5918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44ed0322-9b5b-4751-8f7f-5f0c58493345-workload-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:15.882046 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.882020 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44ed0322-9b5b-4751-8f7f-5f0c58493345-istio-podinfo" pod "44ed0322-9b5b-4751-8f7f-5f0c58493345" (UID: "44ed0322-9b5b-4751-8f7f-5f0c58493345"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:17:15.882178 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.882154 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44ed0322-9b5b-4751-8f7f-5f0c58493345-kube-api-access-k8d9b" pod "44ed0322-9b5b-4751-8f7f-5f0c58493345" (UID: "44ed0322-9b5b-4751-8f7f-5f0c58493345"). InnerVolumeSpecName "kube-api-access-k8d9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:15.882258 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.882185 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ed0322-9b5b-4751-8f7f-5f0c58493345-istio-envoy" pod "44ed0322-9b5b-4751-8f7f-5f0c58493345" (UID: "44ed0322-9b5b-4751-8f7f-5f0c58493345"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:15.883561 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.883540 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44ed0322-9b5b-4751-8f7f-5f0c58493345-istio-token" pod "44ed0322-9b5b-4751-8f7f-5f0c58493345" (UID: "44ed0322-9b5b-4751-8f7f-5f0c58493345"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:15.981446 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.981417 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b4e06dd-69d5-485e-acac-62dddf7f51cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4e06dd-69d5-485e-acac-62dddf7f51cb-workload-socket\") pod \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\" (UID: \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\") " Oct 30 22:17:15.981624 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.981476 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b4e06dd-69d5-485e-acac-62dddf7f51cb-kube-api-access-mg4st\" (UniqueName: \"kubernetes.io/projected/0b4e06dd-69d5-485e-acac-62dddf7f51cb-kube-api-access-mg4st\") pod \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\" (UID: \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\") " Oct 30 22:17:15.981624 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.981549 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b4e06dd-69d5-485e-acac-62dddf7f51cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b4e06dd-69d5-485e-acac-62dddf7f51cb-istio-envoy\") pod \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\" (UID: \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\") " Oct 30 22:17:15.981624 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.981602 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b4e06dd-69d5-485e-acac-62dddf7f51cb-istio-token\" (UniqueName: \"kubernetes.io/projected/0b4e06dd-69d5-485e-acac-62dddf7f51cb-istio-token\") pod \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\" (UID: \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\") " Oct 30 22:17:15.981818 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.981640 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b4e06dd-69d5-485e-acac-62dddf7f51cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4e06dd-69d5-485e-acac-62dddf7f51cb-credential-socket\") pod \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\" (UID: \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\") " Oct 30 22:17:15.981818 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.981675 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b4e06dd-69d5-485e-acac-62dddf7f51cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b4e06dd-69d5-485e-acac-62dddf7f51cb-workload-certs\") pod \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\" (UID: \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\") " Oct 30 22:17:15.981818 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.981705 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b4e06dd-69d5-485e-acac-62dddf7f51cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b4e06dd-69d5-485e-acac-62dddf7f51cb-istio-data\") pod \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\" (UID: \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\") " Oct 30 22:17:15.981818 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.981734 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b4e06dd-69d5-485e-acac-62dddf7f51cb-workload-socket" pod "0b4e06dd-69d5-485e-acac-62dddf7f51cb" (UID: "0b4e06dd-69d5-485e-acac-62dddf7f51cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:15.982023 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.981842 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b4e06dd-69d5-485e-acac-62dddf7f51cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b4e06dd-69d5-485e-acac-62dddf7f51cb-istiod-ca-cert\") pod \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\" (UID: \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\") " Oct 30 22:17:15.982023 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.981894 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b4e06dd-69d5-485e-acac-62dddf7f51cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b4e06dd-69d5-485e-acac-62dddf7f51cb-istio-podinfo\") pod \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\" (UID: \"0b4e06dd-69d5-485e-acac-62dddf7f51cb\") " Oct 30 22:17:15.982170 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.982136 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b4e06dd-69d5-485e-acac-62dddf7f51cb-workload-certs" pod "0b4e06dd-69d5-485e-acac-62dddf7f51cb" (UID: "0b4e06dd-69d5-485e-acac-62dddf7f51cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:15.982170 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.982151 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b4e06dd-69d5-485e-acac-62dddf7f51cb-credential-socket" pod "0b4e06dd-69d5-485e-acac-62dddf7f51cb" (UID: "0b4e06dd-69d5-485e-acac-62dddf7f51cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:15.982452 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.982431 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b4e06dd-69d5-485e-acac-62dddf7f51cb-istiod-ca-cert" pod "0b4e06dd-69d5-485e-acac-62dddf7f51cb" (UID: "0b4e06dd-69d5-485e-acac-62dddf7f51cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:15.982642 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.982626 5918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4e06dd-69d5-485e-acac-62dddf7f51cb-credential-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:15.982719 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.982644 5918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b4e06dd-69d5-485e-acac-62dddf7f51cb-workload-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:15.982719 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.982653 5918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44ed0322-9b5b-4751-8f7f-5f0c58493345-istio-envoy\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:15.982719 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.982662 5918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44ed0322-9b5b-4751-8f7f-5f0c58493345-istio-token\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:15.982719 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.982670 5918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44ed0322-9b5b-4751-8f7f-5f0c58493345-istio-podinfo\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:15.982719 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.982680 5918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b4e06dd-69d5-485e-acac-62dddf7f51cb-istiod-ca-cert\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:15.982719 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.982688 5918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44ed0322-9b5b-4751-8f7f-5f0c58493345-istiod-ca-cert\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:15.982719 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.982696 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8d9b\" (UniqueName: \"kubernetes.io/projected/44ed0322-9b5b-4751-8f7f-5f0c58493345-kube-api-access-k8d9b\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:15.982719 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.982705 5918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4e06dd-69d5-485e-acac-62dddf7f51cb-workload-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:15.983115 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.982999 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b4e06dd-69d5-485e-acac-62dddf7f51cb-istio-data" pod "0b4e06dd-69d5-485e-acac-62dddf7f51cb" (UID: "0b4e06dd-69d5-485e-acac-62dddf7f51cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:15.984592 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.984566 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b4e06dd-69d5-485e-acac-62dddf7f51cb-kube-api-access-mg4st" pod "0b4e06dd-69d5-485e-acac-62dddf7f51cb" (UID: "0b4e06dd-69d5-485e-acac-62dddf7f51cb"). InnerVolumeSpecName "kube-api-access-mg4st". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:15.984801 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.984783 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b4e06dd-69d5-485e-acac-62dddf7f51cb-istio-token" pod "0b4e06dd-69d5-485e-acac-62dddf7f51cb" (UID: "0b4e06dd-69d5-485e-acac-62dddf7f51cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:15.985009 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.984988 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b4e06dd-69d5-485e-acac-62dddf7f51cb-istio-envoy" pod "0b4e06dd-69d5-485e-acac-62dddf7f51cb" (UID: "0b4e06dd-69d5-485e-acac-62dddf7f51cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:15.985462 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:15.985441 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b4e06dd-69d5-485e-acac-62dddf7f51cb-istio-podinfo" pod "0b4e06dd-69d5-485e-acac-62dddf7f51cb" (UID: "0b4e06dd-69d5-485e-acac-62dddf7f51cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:17:16.085036 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.084950 5918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b4e06dd-69d5-485e-acac-62dddf7f51cb-istio-data\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:16.085036 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.084987 5918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b4e06dd-69d5-485e-acac-62dddf7f51cb-istio-podinfo\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:16.085036 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.085005 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg4st\" (UniqueName: \"kubernetes.io/projected/0b4e06dd-69d5-485e-acac-62dddf7f51cb-kube-api-access-mg4st\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:16.085036 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.085020 5918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b4e06dd-69d5-485e-acac-62dddf7f51cb-istio-envoy\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:16.085036 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.085033 5918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b4e06dd-69d5-485e-acac-62dddf7f51cb-istio-token\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:16.638944 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.638916 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" Oct 30 22:17:16.773797 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.773765 5918 generic.go:356] "Generic (PLEG): container finished" podID="b7e83907-c1a2-4761-ab76-31fe664742a3" containerID="19139c8c06fc3b0fd89bfc8af4ffbf9dbcffe7c498b7230c916d9bb182c329a2" exitCode=0 Oct 30 22:17:16.774227 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.773820 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" Oct 30 22:17:16.774227 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.773855 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" event={"ID":"b7e83907-c1a2-4761-ab76-31fe664742a3","Type":"ContainerDied","Data":"19139c8c06fc3b0fd89bfc8af4ffbf9dbcffe7c498b7230c916d9bb182c329a2"} Oct 30 22:17:16.774227 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.773941 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff" event={"ID":"b7e83907-c1a2-4761-ab76-31fe664742a3","Type":"ContainerDied","Data":"792388c3c1ce8c1555d7d844a94e3b29c0c390e9b08ee7c33a0212739e4bf791"} Oct 30 22:17:16.774227 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.773965 5918 scope.go:122] "RemoveContainer" containerID="19139c8c06fc3b0fd89bfc8af4ffbf9dbcffe7c498b7230c916d9bb182c329a2" Oct 30 22:17:16.775594 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.775572 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" Oct 30 22:17:16.775709 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.775593 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l" event={"ID":"0b4e06dd-69d5-485e-acac-62dddf7f51cb","Type":"ContainerDied","Data":"7994a68f55645987b5caf249568f8ed676692bdf0884a8e85e975b656b1544bd"} Oct 30 22:17:16.775709 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.775571 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt" Oct 30 22:17:16.786047 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.786027 5918 scope.go:122] "RemoveContainer" containerID="19139c8c06fc3b0fd89bfc8af4ffbf9dbcffe7c498b7230c916d9bb182c329a2" Oct 30 22:17:16.786424 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:17:16.786394 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19139c8c06fc3b0fd89bfc8af4ffbf9dbcffe7c498b7230c916d9bb182c329a2\": container with ID starting with 19139c8c06fc3b0fd89bfc8af4ffbf9dbcffe7c498b7230c916d9bb182c329a2 not found: ID does not exist" containerID="19139c8c06fc3b0fd89bfc8af4ffbf9dbcffe7c498b7230c916d9bb182c329a2" Oct 30 22:17:16.786526 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.786430 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19139c8c06fc3b0fd89bfc8af4ffbf9dbcffe7c498b7230c916d9bb182c329a2"} err="failed to get container status \"19139c8c06fc3b0fd89bfc8af4ffbf9dbcffe7c498b7230c916d9bb182c329a2\": rpc error: code = NotFound desc = could not find container \"19139c8c06fc3b0fd89bfc8af4ffbf9dbcffe7c498b7230c916d9bb182c329a2\": container with ID starting with 19139c8c06fc3b0fd89bfc8af4ffbf9dbcffe7c498b7230c916d9bb182c329a2 not found: ID does not exist" Oct 30 22:17:16.786526 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.786453 5918 scope.go:122] "RemoveContainer" containerID="772a26e1786f359420389b4c9ac35a88a19b648c28d75d3c53eb326ac36338f3" Oct 30 22:17:16.794666 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.794644 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7e83907-c1a2-4761-ab76-31fe664742a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7e83907-c1a2-4761-ab76-31fe664742a3-workload-socket\") pod \"b7e83907-c1a2-4761-ab76-31fe664742a3\" (UID: \"b7e83907-c1a2-4761-ab76-31fe664742a3\") " Oct 30 22:17:16.794749 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.794728 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7e83907-c1a2-4761-ab76-31fe664742a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7e83907-c1a2-4761-ab76-31fe664742a3-istio-data\") pod \"b7e83907-c1a2-4761-ab76-31fe664742a3\" (UID: \"b7e83907-c1a2-4761-ab76-31fe664742a3\") " Oct 30 22:17:16.794808 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.794794 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7e83907-c1a2-4761-ab76-31fe664742a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7e83907-c1a2-4761-ab76-31fe664742a3-credential-socket\") pod \"b7e83907-c1a2-4761-ab76-31fe664742a3\" (UID: \"b7e83907-c1a2-4761-ab76-31fe664742a3\") " Oct 30 22:17:16.794951 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.794934 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7e83907-c1a2-4761-ab76-31fe664742a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7e83907-c1a2-4761-ab76-31fe664742a3-istio-podinfo\") pod \"b7e83907-c1a2-4761-ab76-31fe664742a3\" (UID: \"b7e83907-c1a2-4761-ab76-31fe664742a3\") " Oct 30 22:17:16.795012 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.794983 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7e83907-c1a2-4761-ab76-31fe664742a3-kube-api-access-nwk54\" (UniqueName: \"kubernetes.io/projected/b7e83907-c1a2-4761-ab76-31fe664742a3-kube-api-access-nwk54\") pod \"b7e83907-c1a2-4761-ab76-31fe664742a3\" (UID: \"b7e83907-c1a2-4761-ab76-31fe664742a3\") " Oct 30 22:17:16.795012 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.794934 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7e83907-c1a2-4761-ab76-31fe664742a3-workload-socket" pod "b7e83907-c1a2-4761-ab76-31fe664742a3" (UID: "b7e83907-c1a2-4761-ab76-31fe664742a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:16.795122 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.794984 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7e83907-c1a2-4761-ab76-31fe664742a3-istio-data" pod "b7e83907-c1a2-4761-ab76-31fe664742a3" (UID: "b7e83907-c1a2-4761-ab76-31fe664742a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:16.795122 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.795026 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7e83907-c1a2-4761-ab76-31fe664742a3-istio-token\" (UniqueName: \"kubernetes.io/projected/b7e83907-c1a2-4761-ab76-31fe664742a3-istio-token\") pod \"b7e83907-c1a2-4761-ab76-31fe664742a3\" (UID: \"b7e83907-c1a2-4761-ab76-31fe664742a3\") " Oct 30 22:17:16.795122 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.795080 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7e83907-c1a2-4761-ab76-31fe664742a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7e83907-c1a2-4761-ab76-31fe664742a3-workload-certs\") pod \"b7e83907-c1a2-4761-ab76-31fe664742a3\" (UID: \"b7e83907-c1a2-4761-ab76-31fe664742a3\") " Oct 30 22:17:16.795122 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.795097 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7e83907-c1a2-4761-ab76-31fe664742a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7e83907-c1a2-4761-ab76-31fe664742a3-istiod-ca-cert\") pod \"b7e83907-c1a2-4761-ab76-31fe664742a3\" (UID: \"b7e83907-c1a2-4761-ab76-31fe664742a3\") " Oct 30 22:17:16.795344 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.795117 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7e83907-c1a2-4761-ab76-31fe664742a3-credential-socket" pod "b7e83907-c1a2-4761-ab76-31fe664742a3" (UID: "b7e83907-c1a2-4761-ab76-31fe664742a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:16.795344 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.795156 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7e83907-c1a2-4761-ab76-31fe664742a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7e83907-c1a2-4761-ab76-31fe664742a3-istio-envoy\") pod \"b7e83907-c1a2-4761-ab76-31fe664742a3\" (UID: \"b7e83907-c1a2-4761-ab76-31fe664742a3\") " Oct 30 22:17:16.795445 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.795359 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7e83907-c1a2-4761-ab76-31fe664742a3-workload-certs" pod "b7e83907-c1a2-4761-ab76-31fe664742a3" (UID: "b7e83907-c1a2-4761-ab76-31fe664742a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:16.795618 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.795573 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7e83907-c1a2-4761-ab76-31fe664742a3-istiod-ca-cert" pod "b7e83907-c1a2-4761-ab76-31fe664742a3" (UID: "b7e83907-c1a2-4761-ab76-31fe664742a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:16.795857 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.795837 5918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7e83907-c1a2-4761-ab76-31fe664742a3-workload-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:16.795935 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.795864 5918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7e83907-c1a2-4761-ab76-31fe664742a3-istio-data\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:16.795935 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.795879 5918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7e83907-c1a2-4761-ab76-31fe664742a3-credential-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:16.795935 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.795892 5918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7e83907-c1a2-4761-ab76-31fe664742a3-workload-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:16.795935 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.795905 5918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7e83907-c1a2-4761-ab76-31fe664742a3-istiod-ca-cert\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:16.797291 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.797206 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7e83907-c1a2-4761-ab76-31fe664742a3-istio-podinfo" pod "b7e83907-c1a2-4761-ab76-31fe664742a3" (UID: "b7e83907-c1a2-4761-ab76-31fe664742a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:17:16.797661 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.797639 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7e83907-c1a2-4761-ab76-31fe664742a3-kube-api-access-nwk54" pod "b7e83907-c1a2-4761-ab76-31fe664742a3" (UID: "b7e83907-c1a2-4761-ab76-31fe664742a3"). InnerVolumeSpecName "kube-api-access-nwk54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:16.797858 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.797841 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7e83907-c1a2-4761-ab76-31fe664742a3-istio-envoy" pod "b7e83907-c1a2-4761-ab76-31fe664742a3" (UID: "b7e83907-c1a2-4761-ab76-31fe664742a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:16.797945 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.797859 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7e83907-c1a2-4761-ab76-31fe664742a3-istio-token" pod "b7e83907-c1a2-4761-ab76-31fe664742a3" (UID: "b7e83907-c1a2-4761-ab76-31fe664742a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:16.897396 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.897365 5918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7e83907-c1a2-4761-ab76-31fe664742a3-istio-podinfo\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:16.897396 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.897394 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwk54\" (UniqueName: \"kubernetes.io/projected/b7e83907-c1a2-4761-ab76-31fe664742a3-kube-api-access-nwk54\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:16.897570 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.897409 5918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7e83907-c1a2-4761-ab76-31fe664742a3-istio-token\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:16.897570 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:16.897421 5918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7e83907-c1a2-4761-ab76-31fe664742a3-istio-envoy\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:17.014176 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:17.014139 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt"] Oct 30 22:17:17.024207 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:17.024128 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-5dmtt"] Oct 30 22:17:17.208037 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:17.207999 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l"] Oct 30 22:17:17.221016 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:17.220990 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-tcm8l"] Oct 30 22:17:17.352651 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:17.352555 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff"] Oct 30 22:17:17.356761 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:17.356734 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3ef8140e-openshift-default-6df8574985-8z8ff"] Oct 30 22:17:17.855736 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:17.855702 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b4e06dd-69d5-485e-acac-62dddf7f51cb" path="/var/lib/kubelet/pods/0b4e06dd-69d5-485e-acac-62dddf7f51cb/volumes" Oct 30 22:17:17.856255 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:17.856236 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44ed0322-9b5b-4751-8f7f-5f0c58493345" path="/var/lib/kubelet/pods/44ed0322-9b5b-4751-8f7f-5f0c58493345/volumes" Oct 30 22:17:17.856700 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:17.856685 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7e83907-c1a2-4761-ab76-31fe664742a3" path="/var/lib/kubelet/pods/b7e83907-c1a2-4761-ab76-31fe664742a3/volumes" Oct 30 22:17:21.275621 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:17:21.275584 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:17:21.624239 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.624135 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn"] Oct 30 22:17:21.624819 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.624802 5918 cpu_manager.go:414] "containerMap: removing container" podUID="0b4e06dd-69d5-485e-acac-62dddf7f51cb" containerName="istio-proxy" Oct 30 22:17:21.624819 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.624818 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b4e06dd-69d5-485e-acac-62dddf7f51cb" containerName="istio-proxy" Oct 30 22:17:21.624959 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.624836 5918 cpu_manager.go:414] "containerMap: removing container" podUID="44ed0322-9b5b-4751-8f7f-5f0c58493345" containerName="istio-proxy" Oct 30 22:17:21.624959 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.624841 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44ed0322-9b5b-4751-8f7f-5f0c58493345" containerName="istio-proxy" Oct 30 22:17:21.624959 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.624863 5918 cpu_manager.go:414] "containerMap: removing container" podUID="d68726e2-79a3-4c7d-9c64-c06d73bb3077" containerName="istio-proxy" Oct 30 22:17:21.624959 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.624871 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d68726e2-79a3-4c7d-9c64-c06d73bb3077" containerName="istio-proxy" Oct 30 22:17:21.624959 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.624881 5918 cpu_manager.go:414] "containerMap: removing container" podUID="e2801c3c-999f-41aa-900e-5a8b1c9fd8c0" containerName="istio-proxy" Oct 30 22:17:21.624959 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.624888 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2801c3c-999f-41aa-900e-5a8b1c9fd8c0" containerName="istio-proxy" Oct 30 22:17:21.624959 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.624917 5918 cpu_manager.go:414] "containerMap: removing container" podUID="2a4486c4-7472-4dae-bd5c-2c56cd386fcb" containerName="istio-proxy" Oct 30 22:17:21.624959 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.624925 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a4486c4-7472-4dae-bd5c-2c56cd386fcb" containerName="istio-proxy" Oct 30 22:17:21.624959 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.624939 5918 cpu_manager.go:414] "containerMap: removing container" podUID="f83c719c-83ad-4b5e-be59-a3242cd7ecaf" containerName="istio-proxy" Oct 30 22:17:21.624959 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.624944 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f83c719c-83ad-4b5e-be59-a3242cd7ecaf" containerName="istio-proxy" Oct 30 22:17:21.624959 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.624954 5918 cpu_manager.go:414] "containerMap: removing container" podUID="b7e83907-c1a2-4761-ab76-31fe664742a3" containerName="istio-proxy" Oct 30 22:17:21.624959 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.624958 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7e83907-c1a2-4761-ab76-31fe664742a3" containerName="istio-proxy" Oct 30 22:17:21.625487 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.624968 5918 cpu_manager.go:414] "containerMap: removing container" podUID="ba68152b-0571-406a-8355-8fc4d3aae15c" containerName="istio-proxy" Oct 30 22:17:21.625487 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.624974 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba68152b-0571-406a-8355-8fc4d3aae15c" containerName="istio-proxy" Oct 30 22:17:21.625487 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.625068 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="d68726e2-79a3-4c7d-9c64-c06d73bb3077" containerName="istio-proxy" Oct 30 22:17:21.625487 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.625077 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2801c3c-999f-41aa-900e-5a8b1c9fd8c0" containerName="istio-proxy" Oct 30 22:17:21.625487 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.625093 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a4486c4-7472-4dae-bd5c-2c56cd386fcb" containerName="istio-proxy" Oct 30 22:17:21.625487 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.625104 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f83c719c-83ad-4b5e-be59-a3242cd7ecaf" containerName="istio-proxy" Oct 30 22:17:21.625487 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.625113 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7e83907-c1a2-4761-ab76-31fe664742a3" containerName="istio-proxy" Oct 30 22:17:21.625487 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.625120 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba68152b-0571-406a-8355-8fc4d3aae15c" containerName="istio-proxy" Oct 30 22:17:21.625487 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.625129 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b4e06dd-69d5-485e-acac-62dddf7f51cb" containerName="istio-proxy" Oct 30 22:17:21.625487 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.625136 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="44ed0322-9b5b-4751-8f7f-5f0c58493345" containerName="istio-proxy" Oct 30 22:17:21.630614 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.630595 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn" Oct 30 22:17:21.690033 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.689993 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a491d2d9-f2c7-48aa-9091-7df25f1e0b09-openshift-service-ca\") pod \"maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn\" (UID: \"a491d2d9-f2c7-48aa-9091-7df25f1e0b09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn" Oct 30 22:17:21.690245 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.690042 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5dxw\" (UniqueName: \"kubernetes.io/projected/a491d2d9-f2c7-48aa-9091-7df25f1e0b09-kube-api-access-g5dxw\") pod \"maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn\" (UID: \"a491d2d9-f2c7-48aa-9091-7df25f1e0b09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn" Oct 30 22:17:21.690245 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.690073 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a491d2d9-f2c7-48aa-9091-7df25f1e0b09-metrics-certs\") pod \"maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn\" (UID: \"a491d2d9-f2c7-48aa-9091-7df25f1e0b09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn" Oct 30 22:17:21.690245 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.690230 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a491d2d9-f2c7-48aa-9091-7df25f1e0b09-maas-api-tls\") pod \"maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn\" (UID: \"a491d2d9-f2c7-48aa-9091-7df25f1e0b09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn" Oct 30 22:17:21.739943 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.739899 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-422c1029-serving-cert\"" Oct 30 22:17:21.745341 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.745310 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn"] Oct 30 22:17:21.791149 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.791115 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a491d2d9-f2c7-48aa-9091-7df25f1e0b09-openshift-service-ca\") pod \"maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn\" (UID: \"a491d2d9-f2c7-48aa-9091-7df25f1e0b09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn" Oct 30 22:17:21.791338 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.791153 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5dxw\" (UniqueName: \"kubernetes.io/projected/a491d2d9-f2c7-48aa-9091-7df25f1e0b09-kube-api-access-g5dxw\") pod \"maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn\" (UID: \"a491d2d9-f2c7-48aa-9091-7df25f1e0b09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn" Oct 30 22:17:21.791338 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.791172 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a491d2d9-f2c7-48aa-9091-7df25f1e0b09-metrics-certs\") pod \"maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn\" (UID: \"a491d2d9-f2c7-48aa-9091-7df25f1e0b09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn" Oct 30 22:17:21.791338 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.791322 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a491d2d9-f2c7-48aa-9091-7df25f1e0b09-maas-api-tls\") pod \"maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn\" (UID: \"a491d2d9-f2c7-48aa-9091-7df25f1e0b09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn" Oct 30 22:17:21.791503 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:17:21.791477 5918 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-serving-cert: secret "maas-api-e2e-mt-422c1029-serving-cert" not found Oct 30 22:17:21.791580 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:17:21.791566 5918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a491d2d9-f2c7-48aa-9091-7df25f1e0b09-maas-api-tls podName:a491d2d9-f2c7-48aa-9091-7df25f1e0b09 nodeName:}" failed. No retries permitted until 2026-10-30 22:17:22.291546695 +0000 UTC m=+1602.562244774 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a491d2d9-f2c7-48aa-9091-7df25f1e0b09-maas-api-tls") pod "maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn" (UID: "a491d2d9-f2c7-48aa-9091-7df25f1e0b09") : secret "maas-api-e2e-mt-422c1029-serving-cert" not found Oct 30 22:17:21.791866 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.791847 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a491d2d9-f2c7-48aa-9091-7df25f1e0b09-openshift-service-ca\") pod \"maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn\" (UID: \"a491d2d9-f2c7-48aa-9091-7df25f1e0b09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn" Oct 30 22:17:21.793782 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.793766 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a491d2d9-f2c7-48aa-9091-7df25f1e0b09-metrics-certs\") pod \"maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn\" (UID: \"a491d2d9-f2c7-48aa-9091-7df25f1e0b09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn" Oct 30 22:17:21.863463 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:21.863434 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5dxw\" (UniqueName: \"kubernetes.io/projected/a491d2d9-f2c7-48aa-9091-7df25f1e0b09-kube-api-access-g5dxw\") pod \"maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn\" (UID: \"a491d2d9-f2c7-48aa-9091-7df25f1e0b09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn" Oct 30 22:17:22.297160 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:22.297067 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a491d2d9-f2c7-48aa-9091-7df25f1e0b09-maas-api-tls\") pod \"maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn\" (UID: \"a491d2d9-f2c7-48aa-9091-7df25f1e0b09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn" Oct 30 22:17:22.300181 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:22.300151 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a491d2d9-f2c7-48aa-9091-7df25f1e0b09-maas-api-tls\") pod \"maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn\" (UID: \"a491d2d9-f2c7-48aa-9091-7df25f1e0b09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn" Oct 30 22:17:22.545553 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:22.545516 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn" Oct 30 22:17:22.942886 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:22.942862 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn"] Oct 30 22:17:22.945019 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:17:22.944994 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda491d2d9_f2c7_48aa_9091_7df25f1e0b09.slice/crio-18a75e688e3e8e65b82e9b25a78f3368230f0bd9d11f8a32948babbff943a872 WatchSource:0}: Error finding container 18a75e688e3e8e65b82e9b25a78f3368230f0bd9d11f8a32948babbff943a872: Status 404 returned error can't find the container with id 18a75e688e3e8e65b82e9b25a78f3368230f0bd9d11f8a32948babbff943a872 Oct 30 22:17:23.852381 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:23.852351 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn" event={"ID":"a491d2d9-f2c7-48aa-9091-7df25f1e0b09","Type":"ContainerStarted","Data":"18a75e688e3e8e65b82e9b25a78f3368230f0bd9d11f8a32948babbff943a872"} Oct 30 22:17:24.856032 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:24.855997 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn" event={"ID":"a491d2d9-f2c7-48aa-9091-7df25f1e0b09","Type":"ContainerStarted","Data":"512b299b5cdb1a27f8434db87dc30b0926929a46fd9d353a12cb7437496a1eb4"} Oct 30 22:17:24.856481 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:24.856128 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn" Oct 30 22:17:24.979687 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:24.979635 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn" podStartSLOduration=3.127980157 podStartE2EDuration="3.979619852s" totalImagesPullingTime="851.639695ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:24.978328601 +0000 UTC m=+1605.249026687" watchObservedRunningTime="2026-10-30 22:17:24.979619852 +0000 UTC m=+1605.250317937" Oct 30 22:17:30.882648 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:17:30.882618 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn" Oct 30 22:17:51.274828 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:17:51.274746 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:18:04.820867 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:04.820833 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-64gml"] Oct 30 22:18:04.825115 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:04.825099 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-64gml" Oct 30 22:18:04.876072 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:04.876040 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9b614c-c47575765-qqmbn"] Oct 30 22:18:04.880557 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:04.880532 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9b614c-c47575765-qqmbn" Oct 30 22:18:04.922937 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:04.922906 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-9b614c\"" Oct 30 22:18:04.927988 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:04.927953 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-64gml"] Oct 30 22:18:04.938079 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:04.938032 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-9b614c-dockercfg-cvz9j\"" Oct 30 22:18:04.947439 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:04.947413 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9b614c-c47575765-qqmbn"] Oct 30 22:18:04.948187 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:04.948169 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc8c80b0-9111-4da5-908c-fe7f3f99acb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-64gml\" (UID: \"bc8c80b0-9111-4da5-908c-fe7f3f99acb8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-64gml" Oct 30 22:18:04.948392 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:04.948363 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96lhv\" (UniqueName: \"kubernetes.io/projected/bc8c80b0-9111-4da5-908c-fe7f3f99acb8-kube-api-access-96lhv\") pod \"payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-64gml\" (UID: \"bc8c80b0-9111-4da5-908c-fe7f3f99acb8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-64gml" Oct 30 22:18:04.948524 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:04.948507 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cefcee5b-c92a-4067-8c83-cea422238674-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-9b614c-c47575765-qqmbn\" (UID: \"cefcee5b-c92a-4067-8c83-cea422238674\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9b614c-c47575765-qqmbn" Oct 30 22:18:04.948684 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:04.948664 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swcwn\" (UniqueName: \"kubernetes.io/projected/cefcee5b-c92a-4067-8c83-cea422238674-kube-api-access-swcwn\") pod \"payload-processing-e2e-shared-a-w3-9b614c-c47575765-qqmbn\" (UID: \"cefcee5b-c92a-4067-8c83-cea422238674\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9b614c-c47575765-qqmbn" Oct 30 22:18:05.050206 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:05.050163 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc8c80b0-9111-4da5-908c-fe7f3f99acb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-64gml\" (UID: \"bc8c80b0-9111-4da5-908c-fe7f3f99acb8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-64gml" Oct 30 22:18:05.050412 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:05.050278 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96lhv\" (UniqueName: \"kubernetes.io/projected/bc8c80b0-9111-4da5-908c-fe7f3f99acb8-kube-api-access-96lhv\") pod \"payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-64gml\" (UID: \"bc8c80b0-9111-4da5-908c-fe7f3f99acb8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-64gml" Oct 30 22:18:05.050412 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:05.050334 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cefcee5b-c92a-4067-8c83-cea422238674-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-9b614c-c47575765-qqmbn\" (UID: \"cefcee5b-c92a-4067-8c83-cea422238674\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9b614c-c47575765-qqmbn" Oct 30 22:18:05.050537 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:05.050413 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swcwn\" (UniqueName: \"kubernetes.io/projected/cefcee5b-c92a-4067-8c83-cea422238674-kube-api-access-swcwn\") pod \"payload-processing-e2e-shared-a-w3-9b614c-c47575765-qqmbn\" (UID: \"cefcee5b-c92a-4067-8c83-cea422238674\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9b614c-c47575765-qqmbn" Oct 30 22:18:05.051180 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:05.051147 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc8c80b0-9111-4da5-908c-fe7f3f99acb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-64gml\" (UID: \"bc8c80b0-9111-4da5-908c-fe7f3f99acb8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-64gml" Oct 30 22:18:05.051180 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:05.051181 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cefcee5b-c92a-4067-8c83-cea422238674-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-9b614c-c47575765-qqmbn\" (UID: \"cefcee5b-c92a-4067-8c83-cea422238674\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9b614c-c47575765-qqmbn" Oct 30 22:18:05.127494 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:05.127067 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swcwn\" (UniqueName: \"kubernetes.io/projected/cefcee5b-c92a-4067-8c83-cea422238674-kube-api-access-swcwn\") pod \"payload-processing-e2e-shared-a-w3-9b614c-c47575765-qqmbn\" (UID: \"cefcee5b-c92a-4067-8c83-cea422238674\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9b614c-c47575765-qqmbn" Oct 30 22:18:05.133481 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:05.133446 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96lhv\" (UniqueName: \"kubernetes.io/projected/bc8c80b0-9111-4da5-908c-fe7f3f99acb8-kube-api-access-96lhv\") pod \"payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-64gml\" (UID: \"bc8c80b0-9111-4da5-908c-fe7f3f99acb8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-64gml" Oct 30 22:18:05.142847 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:05.142809 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-64gml" Oct 30 22:18:05.196513 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:05.196478 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9b614c-c47575765-qqmbn" Oct 30 22:18:05.539579 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:05.539549 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-64gml"] Oct 30 22:18:05.540571 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:18:05.540540 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc8c80b0_9111_4da5_908c_fe7f3f99acb8.slice/crio-045d402285259811cb63f0e33ab0132d353ccad73eea05996da9bc248686f285 WatchSource:0}: Error finding container 045d402285259811cb63f0e33ab0132d353ccad73eea05996da9bc248686f285: Status 404 returned error can't find the container with id 045d402285259811cb63f0e33ab0132d353ccad73eea05996da9bc248686f285 Oct 30 22:18:05.658374 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:18:05.655370 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcefcee5b_c92a_4067_8c83_cea422238674.slice/crio-4586260d739d1bee18a097bf51e59c68f1da5707da478eba66926c9518f0398c WatchSource:0}: Error finding container 4586260d739d1bee18a097bf51e59c68f1da5707da478eba66926c9518f0398c: Status 404 returned error can't find the container with id 4586260d739d1bee18a097bf51e59c68f1da5707da478eba66926c9518f0398c Oct 30 22:18:05.660082 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:05.660046 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9b614c-c47575765-qqmbn"] Oct 30 22:18:06.168584 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:06.168545 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9b614c-c47575765-qqmbn" event={"ID":"cefcee5b-c92a-4067-8c83-cea422238674","Type":"ContainerStarted","Data":"d58a92385779f79ed60ddb253dd6e752f4ea2225388bbefda44e648fd240e700"} Oct 30 22:18:06.169074 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:06.168610 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9b614c-c47575765-qqmbn" event={"ID":"cefcee5b-c92a-4067-8c83-cea422238674","Type":"ContainerStarted","Data":"4586260d739d1bee18a097bf51e59c68f1da5707da478eba66926c9518f0398c"} Oct 30 22:18:06.169074 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:06.168746 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9b614c-c47575765-qqmbn" Oct 30 22:18:06.170233 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:06.170191 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-64gml" event={"ID":"bc8c80b0-9111-4da5-908c-fe7f3f99acb8","Type":"ContainerStarted","Data":"b17f2c33bff5377b211e62bc8eb6ddf437b6e411617dc2a02e55b482c5f47a3a"} Oct 30 22:18:06.170329 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:06.170270 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-64gml" event={"ID":"bc8c80b0-9111-4da5-908c-fe7f3f99acb8","Type":"ContainerStarted","Data":"045d402285259811cb63f0e33ab0132d353ccad73eea05996da9bc248686f285"} Oct 30 22:18:06.170329 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:06.170298 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-64gml" Oct 30 22:18:06.288149 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:06.288101 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9b614c-c47575765-qqmbn" podStartSLOduration=2.288087793 podStartE2EDuration="2.288087793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:06.287568942 +0000 UTC m=+1646.558267026" watchObservedRunningTime="2026-10-30 22:18:06.288087793 +0000 UTC m=+1646.558785939" Oct 30 22:18:06.409070 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:06.409012 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-64gml" podStartSLOduration=2.408997452 podStartE2EDuration="2.408997452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:06.406158064 +0000 UTC m=+1646.676856148" watchObservedRunningTime="2026-10-30 22:18:06.408997452 +0000 UTC m=+1646.679695557" Oct 30 22:18:17.180671 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:17.180644 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-64gml" Oct 30 22:18:17.181059 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:17.180706 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9b614c-c47575765-qqmbn" Oct 30 22:18:17.863426 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:17.863398 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z"] Oct 30 22:18:17.876720 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:17.876698 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z" Oct 30 22:18:17.928835 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:17.928802 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z"] Oct 30 22:18:17.951244 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:17.951202 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-41e4de-serving-cert\"" Oct 30 22:18:18.042713 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:18.042682 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d4e2b8-0f58-4b32-8227-6e3418b8521d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z\" (UID: \"d0d4e2b8-0f58-4b32-8227-6e3418b8521d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z" Oct 30 22:18:18.042883 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:18.042738 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0d4e2b8-0f58-4b32-8227-6e3418b8521d-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z\" (UID: \"d0d4e2b8-0f58-4b32-8227-6e3418b8521d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z" Oct 30 22:18:18.042883 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:18.042835 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0d4e2b8-0f58-4b32-8227-6e3418b8521d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z\" (UID: \"d0d4e2b8-0f58-4b32-8227-6e3418b8521d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z" Oct 30 22:18:18.043015 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:18.042992 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wnpm\" (UniqueName: \"kubernetes.io/projected/d0d4e2b8-0f58-4b32-8227-6e3418b8521d-kube-api-access-7wnpm\") pod \"maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z\" (UID: \"d0d4e2b8-0f58-4b32-8227-6e3418b8521d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z" Oct 30 22:18:18.144498 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:18.144406 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0d4e2b8-0f58-4b32-8227-6e3418b8521d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z\" (UID: \"d0d4e2b8-0f58-4b32-8227-6e3418b8521d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z" Oct 30 22:18:18.144661 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:18:18.144556 5918 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-41e4de-serving-cert: secret "maas-api-e2e-shared-a-w6-41e4de-serving-cert" not found Oct 30 22:18:18.144661 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:18.144558 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wnpm\" (UniqueName: \"kubernetes.io/projected/d0d4e2b8-0f58-4b32-8227-6e3418b8521d-kube-api-access-7wnpm\") pod \"maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z\" (UID: \"d0d4e2b8-0f58-4b32-8227-6e3418b8521d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z" Oct 30 22:18:18.144661 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:18:18.144618 5918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0d4e2b8-0f58-4b32-8227-6e3418b8521d-maas-api-tls podName:d0d4e2b8-0f58-4b32-8227-6e3418b8521d nodeName:}" failed. No retries permitted until 2026-10-30 22:18:18.644598781 +0000 UTC m=+1658.915296868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d0d4e2b8-0f58-4b32-8227-6e3418b8521d-maas-api-tls") pod "maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z" (UID: "d0d4e2b8-0f58-4b32-8227-6e3418b8521d") : secret "maas-api-e2e-shared-a-w6-41e4de-serving-cert" not found Oct 30 22:18:18.144799 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:18.144774 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d4e2b8-0f58-4b32-8227-6e3418b8521d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z\" (UID: \"d0d4e2b8-0f58-4b32-8227-6e3418b8521d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z" Oct 30 22:18:18.144849 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:18.144834 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0d4e2b8-0f58-4b32-8227-6e3418b8521d-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z\" (UID: \"d0d4e2b8-0f58-4b32-8227-6e3418b8521d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z" Oct 30 22:18:18.145388 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:18.145368 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d4e2b8-0f58-4b32-8227-6e3418b8521d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z\" (UID: \"d0d4e2b8-0f58-4b32-8227-6e3418b8521d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z" Oct 30 22:18:18.147428 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:18.147408 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0d4e2b8-0f58-4b32-8227-6e3418b8521d-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z\" (UID: \"d0d4e2b8-0f58-4b32-8227-6e3418b8521d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z" Oct 30 22:18:18.214546 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:18.214515 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wnpm\" (UniqueName: \"kubernetes.io/projected/d0d4e2b8-0f58-4b32-8227-6e3418b8521d-kube-api-access-7wnpm\") pod \"maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z\" (UID: \"d0d4e2b8-0f58-4b32-8227-6e3418b8521d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z" Oct 30 22:18:18.651925 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:18.651889 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0d4e2b8-0f58-4b32-8227-6e3418b8521d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z\" (UID: \"d0d4e2b8-0f58-4b32-8227-6e3418b8521d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z" Oct 30 22:18:18.654393 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:18.654361 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0d4e2b8-0f58-4b32-8227-6e3418b8521d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z\" (UID: \"d0d4e2b8-0f58-4b32-8227-6e3418b8521d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z" Oct 30 22:18:18.792958 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:18.792921 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z" Oct 30 22:18:19.178871 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:19.178840 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z"] Oct 30 22:18:19.179583 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:18:19.179557 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d4e2b8_0f58_4b32_8227_6e3418b8521d.slice/crio-7ad995170c16d3d8f7d5210892462a7413a69f5361250a0e09c034dac47786ea WatchSource:0}: Error finding container 7ad995170c16d3d8f7d5210892462a7413a69f5361250a0e09c034dac47786ea: Status 404 returned error can't find the container with id 7ad995170c16d3d8f7d5210892462a7413a69f5361250a0e09c034dac47786ea Oct 30 22:18:19.261357 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:19.261319 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z" event={"ID":"d0d4e2b8-0f58-4b32-8227-6e3418b8521d","Type":"ContainerStarted","Data":"7ad995170c16d3d8f7d5210892462a7413a69f5361250a0e09c034dac47786ea"} Oct 30 22:18:20.269791 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:20.269710 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z" event={"ID":"d0d4e2b8-0f58-4b32-8227-6e3418b8521d","Type":"ContainerStarted","Data":"513e936f8b524119e699a9f0e5022c2943bf08b12a004a983ae4cfe053846eb3"} Oct 30 22:18:20.269791 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:20.269782 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z" Oct 30 22:18:20.396622 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:20.396566 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z" podStartSLOduration=2.581687746 podStartE2EDuration="3.39655126s" totalImagesPullingTime="814.863514ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:20.39378512 +0000 UTC m=+1660.664483205" watchObservedRunningTime="2026-10-30 22:18:20.39655126 +0000 UTC m=+1660.667249410" Oct 30 22:18:21.277428 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:18:21.277394 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:18:26.283999 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:26.283971 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z" Oct 30 22:18:29.374831 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:29.374802 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2"] Oct 30 22:18:29.379016 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:29.378999 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2" Oct 30 22:18:29.433598 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:29.433573 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-b55a43-serving-cert\"" Oct 30 22:18:29.486415 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:29.486387 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2"] Oct 30 22:18:29.504005 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:29.503975 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vqtf\" (UniqueName: \"kubernetes.io/projected/ace7d02d-5b11-4848-844c-e96dfab5e47a-kube-api-access-4vqtf\") pod \"maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2\" (UID: \"ace7d02d-5b11-4848-844c-e96dfab5e47a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2" Oct 30 22:18:29.504258 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:29.504233 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ace7d02d-5b11-4848-844c-e96dfab5e47a-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2\" (UID: \"ace7d02d-5b11-4848-844c-e96dfab5e47a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2" Oct 30 22:18:29.504358 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:29.504332 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ace7d02d-5b11-4848-844c-e96dfab5e47a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2\" (UID: \"ace7d02d-5b11-4848-844c-e96dfab5e47a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2" Oct 30 22:18:29.504485 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:29.504470 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ace7d02d-5b11-4848-844c-e96dfab5e47a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2\" (UID: \"ace7d02d-5b11-4848-844c-e96dfab5e47a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2" Oct 30 22:18:29.606500 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:29.606460 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ace7d02d-5b11-4848-844c-e96dfab5e47a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2\" (UID: \"ace7d02d-5b11-4848-844c-e96dfab5e47a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2" Oct 30 22:18:29.606940 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:18:29.606813 5918 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b55a43-serving-cert: secret "maas-api-e2e-shared-a-w2-b55a43-serving-cert" not found Oct 30 22:18:29.607085 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:29.607054 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vqtf\" (UniqueName: \"kubernetes.io/projected/ace7d02d-5b11-4848-844c-e96dfab5e47a-kube-api-access-4vqtf\") pod \"maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2\" (UID: \"ace7d02d-5b11-4848-844c-e96dfab5e47a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2" Oct 30 22:18:29.607289 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:18:29.607135 5918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ace7d02d-5b11-4848-844c-e96dfab5e47a-maas-api-tls podName:ace7d02d-5b11-4848-844c-e96dfab5e47a nodeName:}" failed. No retries permitted until 2026-10-30 22:18:30.107107015 +0000 UTC m=+1670.377805078 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ace7d02d-5b11-4848-844c-e96dfab5e47a-maas-api-tls") pod "maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2" (UID: "ace7d02d-5b11-4848-844c-e96dfab5e47a") : secret "maas-api-e2e-shared-a-w2-b55a43-serving-cert" not found Oct 30 22:18:29.607371 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:29.607315 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ace7d02d-5b11-4848-844c-e96dfab5e47a-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2\" (UID: \"ace7d02d-5b11-4848-844c-e96dfab5e47a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2" Oct 30 22:18:29.607431 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:29.607371 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ace7d02d-5b11-4848-844c-e96dfab5e47a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2\" (UID: \"ace7d02d-5b11-4848-844c-e96dfab5e47a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2" Oct 30 22:18:29.608279 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:29.608253 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ace7d02d-5b11-4848-844c-e96dfab5e47a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2\" (UID: \"ace7d02d-5b11-4848-844c-e96dfab5e47a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2" Oct 30 22:18:29.610330 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:29.610309 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ace7d02d-5b11-4848-844c-e96dfab5e47a-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2\" (UID: \"ace7d02d-5b11-4848-844c-e96dfab5e47a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2" Oct 30 22:18:29.670626 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:29.670510 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vqtf\" (UniqueName: \"kubernetes.io/projected/ace7d02d-5b11-4848-844c-e96dfab5e47a-kube-api-access-4vqtf\") pod \"maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2\" (UID: \"ace7d02d-5b11-4848-844c-e96dfab5e47a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2" Oct 30 22:18:30.115199 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:30.115156 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ace7d02d-5b11-4848-844c-e96dfab5e47a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2\" (UID: \"ace7d02d-5b11-4848-844c-e96dfab5e47a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2" Oct 30 22:18:30.117786 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:30.117763 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ace7d02d-5b11-4848-844c-e96dfab5e47a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2\" (UID: \"ace7d02d-5b11-4848-844c-e96dfab5e47a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2" Oct 30 22:18:30.292903 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:30.292867 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2" Oct 30 22:18:30.683869 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:30.683845 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2"] Oct 30 22:18:30.685356 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:18:30.685327 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podace7d02d_5b11_4848_844c_e96dfab5e47a.slice/crio-e4ce65d7ae87b151be93828cc406d1e7868427a46a1ef4556e4edb8108c79663 WatchSource:0}: Error finding container e4ce65d7ae87b151be93828cc406d1e7868427a46a1ef4556e4edb8108c79663: Status 404 returned error can't find the container with id e4ce65d7ae87b151be93828cc406d1e7868427a46a1ef4556e4edb8108c79663 Oct 30 22:18:31.344754 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:31.344714 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2" event={"ID":"ace7d02d-5b11-4848-844c-e96dfab5e47a","Type":"ContainerStarted","Data":"e4ce65d7ae87b151be93828cc406d1e7868427a46a1ef4556e4edb8108c79663"} Oct 30 22:18:32.353533 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:32.353501 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2" event={"ID":"ace7d02d-5b11-4848-844c-e96dfab5e47a","Type":"ContainerStarted","Data":"af3b912dd08e0aa55662373e42cecde8ad3f245da2658bd7ac376063ac13a344"} Oct 30 22:18:32.353964 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:32.353701 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2" Oct 30 22:18:32.488022 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:32.487968 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2" podStartSLOduration=2.654840348 podStartE2EDuration="3.487954677s" totalImagesPullingTime="833.114329ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:32.487021559 +0000 UTC m=+1672.757719645" watchObservedRunningTime="2026-10-30 22:18:32.487954677 +0000 UTC m=+1672.758652800" Oct 30 22:18:38.366363 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:18:38.366338 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2" Oct 30 22:18:51.276545 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:18:51.276499 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:19:21.275444 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:19:21.275355 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:19:38.586125 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:38.586087 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm"] Oct 30 22:19:38.586646 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:38.586362 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" podUID="2e8dced5-a68f-436b-ab03-da8d8e879315" containerName="istio-proxy" containerID="cri-o://c30a776aa45e76a131a1ae8b3e8b9727e5edd0d45e6cd298873b1e6be976bc30" gracePeriod=30 Oct 30 22:19:40.107031 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:40.106991 5918 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:19:40.107456 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:40.107045 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" podUID="2e8dced5-a68f-436b-ab03-da8d8e879315" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:19:43.857914 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:43.857850 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" Oct 30 22:19:43.874776 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:43.874749 5918 generic.go:356] "Generic (PLEG): container finished" podID="2e8dced5-a68f-436b-ab03-da8d8e879315" containerID="c30a776aa45e76a131a1ae8b3e8b9727e5edd0d45e6cd298873b1e6be976bc30" exitCode=0 Oct 30 22:19:43.874898 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:43.874792 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" Oct 30 22:19:43.874898 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:43.874835 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" event={"ID":"2e8dced5-a68f-436b-ab03-da8d8e879315","Type":"ContainerDied","Data":"c30a776aa45e76a131a1ae8b3e8b9727e5edd0d45e6cd298873b1e6be976bc30"} Oct 30 22:19:43.874898 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:43.874885 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm" event={"ID":"2e8dced5-a68f-436b-ab03-da8d8e879315","Type":"ContainerDied","Data":"bb4de6e5731e838563b5a663fc334d40c884d02c576ff4b27c5d2d743fd2decb"} Oct 30 22:19:43.874996 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:43.874911 5918 scope.go:122] "RemoveContainer" containerID="c30a776aa45e76a131a1ae8b3e8b9727e5edd0d45e6cd298873b1e6be976bc30" Oct 30 22:19:43.885306 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:43.885289 5918 scope.go:122] "RemoveContainer" containerID="c30a776aa45e76a131a1ae8b3e8b9727e5edd0d45e6cd298873b1e6be976bc30" Oct 30 22:19:43.885597 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:19:43.885580 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c30a776aa45e76a131a1ae8b3e8b9727e5edd0d45e6cd298873b1e6be976bc30\": container with ID starting with c30a776aa45e76a131a1ae8b3e8b9727e5edd0d45e6cd298873b1e6be976bc30 not found: ID does not exist" containerID="c30a776aa45e76a131a1ae8b3e8b9727e5edd0d45e6cd298873b1e6be976bc30" Oct 30 22:19:43.885644 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:43.885607 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c30a776aa45e76a131a1ae8b3e8b9727e5edd0d45e6cd298873b1e6be976bc30"} err="failed to get container status \"c30a776aa45e76a131a1ae8b3e8b9727e5edd0d45e6cd298873b1e6be976bc30\": rpc error: code = NotFound desc = could not find container \"c30a776aa45e76a131a1ae8b3e8b9727e5edd0d45e6cd298873b1e6be976bc30\": container with ID starting with c30a776aa45e76a131a1ae8b3e8b9727e5edd0d45e6cd298873b1e6be976bc30 not found: ID does not exist" Oct 30 22:19:43.981694 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:43.981663 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e8dced5-a68f-436b-ab03-da8d8e879315-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8dced5-a68f-436b-ab03-da8d8e879315-workload-socket\") pod \"2e8dced5-a68f-436b-ab03-da8d8e879315\" (UID: \"2e8dced5-a68f-436b-ab03-da8d8e879315\") " Oct 30 22:19:43.981872 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:43.981727 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e8dced5-a68f-436b-ab03-da8d8e879315-kube-api-access-lwxcl\" (UniqueName: \"kubernetes.io/projected/2e8dced5-a68f-436b-ab03-da8d8e879315-kube-api-access-lwxcl\") pod \"2e8dced5-a68f-436b-ab03-da8d8e879315\" (UID: \"2e8dced5-a68f-436b-ab03-da8d8e879315\") " Oct 30 22:19:43.981872 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:43.981762 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e8dced5-a68f-436b-ab03-da8d8e879315-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e8dced5-a68f-436b-ab03-da8d8e879315-istio-envoy\") pod \"2e8dced5-a68f-436b-ab03-da8d8e879315\" (UID: \"2e8dced5-a68f-436b-ab03-da8d8e879315\") " Oct 30 22:19:43.981872 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:43.981842 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e8dced5-a68f-436b-ab03-da8d8e879315-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e8dced5-a68f-436b-ab03-da8d8e879315-istiod-ca-cert\") pod \"2e8dced5-a68f-436b-ab03-da8d8e879315\" (UID: \"2e8dced5-a68f-436b-ab03-da8d8e879315\") " Oct 30 22:19:43.982015 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:43.981925 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e8dced5-a68f-436b-ab03-da8d8e879315-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e8dced5-a68f-436b-ab03-da8d8e879315-istio-podinfo\") pod \"2e8dced5-a68f-436b-ab03-da8d8e879315\" (UID: \"2e8dced5-a68f-436b-ab03-da8d8e879315\") " Oct 30 22:19:43.982015 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:43.981976 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8dced5-a68f-436b-ab03-da8d8e879315-workload-socket" pod "2e8dced5-a68f-436b-ab03-da8d8e879315" (UID: "2e8dced5-a68f-436b-ab03-da8d8e879315"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:43.982015 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:43.981997 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e8dced5-a68f-436b-ab03-da8d8e879315-istio-token\" (UniqueName: \"kubernetes.io/projected/2e8dced5-a68f-436b-ab03-da8d8e879315-istio-token\") pod \"2e8dced5-a68f-436b-ab03-da8d8e879315\" (UID: \"2e8dced5-a68f-436b-ab03-da8d8e879315\") " Oct 30 22:19:43.982163 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:43.982093 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e8dced5-a68f-436b-ab03-da8d8e879315-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e8dced5-a68f-436b-ab03-da8d8e879315-istio-data\") pod \"2e8dced5-a68f-436b-ab03-da8d8e879315\" (UID: \"2e8dced5-a68f-436b-ab03-da8d8e879315\") " Oct 30 22:19:43.982238 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:43.982171 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e8dced5-a68f-436b-ab03-da8d8e879315-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8dced5-a68f-436b-ab03-da8d8e879315-credential-socket\") pod \"2e8dced5-a68f-436b-ab03-da8d8e879315\" (UID: \"2e8dced5-a68f-436b-ab03-da8d8e879315\") " Oct 30 22:19:43.982313 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:43.982295 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e8dced5-a68f-436b-ab03-da8d8e879315-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e8dced5-a68f-436b-ab03-da8d8e879315-workload-certs\") pod \"2e8dced5-a68f-436b-ab03-da8d8e879315\" (UID: \"2e8dced5-a68f-436b-ab03-da8d8e879315\") " Oct 30 22:19:43.982381 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:43.982316 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e8dced5-a68f-436b-ab03-da8d8e879315-istiod-ca-cert" pod "2e8dced5-a68f-436b-ab03-da8d8e879315" (UID: "2e8dced5-a68f-436b-ab03-da8d8e879315"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:19:43.982586 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:43.982560 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8dced5-a68f-436b-ab03-da8d8e879315-credential-socket" pod "2e8dced5-a68f-436b-ab03-da8d8e879315" (UID: "2e8dced5-a68f-436b-ab03-da8d8e879315"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:43.982696 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:43.982598 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8dced5-a68f-436b-ab03-da8d8e879315-istio-data" pod "2e8dced5-a68f-436b-ab03-da8d8e879315" (UID: "2e8dced5-a68f-436b-ab03-da8d8e879315"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:43.982907 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:43.982861 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8dced5-a68f-436b-ab03-da8d8e879315-workload-certs" pod "2e8dced5-a68f-436b-ab03-da8d8e879315" (UID: "2e8dced5-a68f-436b-ab03-da8d8e879315"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:43.983303 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:43.983277 5918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e8dced5-a68f-436b-ab03-da8d8e879315-workload-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:43.983416 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:43.983304 5918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8dced5-a68f-436b-ab03-da8d8e879315-workload-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:43.983416 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:43.983319 5918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e8dced5-a68f-436b-ab03-da8d8e879315-istiod-ca-cert\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:43.983416 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:43.983333 5918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e8dced5-a68f-436b-ab03-da8d8e879315-istio-data\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:43.983416 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:43.983347 5918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8dced5-a68f-436b-ab03-da8d8e879315-credential-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:43.984448 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:43.984411 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e8dced5-a68f-436b-ab03-da8d8e879315-kube-api-access-lwxcl" pod "2e8dced5-a68f-436b-ab03-da8d8e879315" (UID: "2e8dced5-a68f-436b-ab03-da8d8e879315"). InnerVolumeSpecName "kube-api-access-lwxcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:43.984543 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:43.984504 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8dced5-a68f-436b-ab03-da8d8e879315-istio-envoy" pod "2e8dced5-a68f-436b-ab03-da8d8e879315" (UID: "2e8dced5-a68f-436b-ab03-da8d8e879315"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:43.984601 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:43.984537 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e8dced5-a68f-436b-ab03-da8d8e879315-istio-token" pod "2e8dced5-a68f-436b-ab03-da8d8e879315" (UID: "2e8dced5-a68f-436b-ab03-da8d8e879315"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:43.984705 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:43.984689 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e8dced5-a68f-436b-ab03-da8d8e879315-istio-podinfo" pod "2e8dced5-a68f-436b-ab03-da8d8e879315" (UID: "2e8dced5-a68f-436b-ab03-da8d8e879315"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:19:44.084515 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:44.084482 5918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e8dced5-a68f-436b-ab03-da8d8e879315-istio-podinfo\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:44.084515 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:44.084510 5918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e8dced5-a68f-436b-ab03-da8d8e879315-istio-token\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:44.084515 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:44.084521 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwxcl\" (UniqueName: \"kubernetes.io/projected/2e8dced5-a68f-436b-ab03-da8d8e879315-kube-api-access-lwxcl\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:44.084746 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:44.084532 5918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e8dced5-a68f-436b-ab03-da8d8e879315-istio-envoy\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:44.301180 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:44.301147 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm"] Oct 30 22:19:44.306859 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:44.306832 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-q6fbm"] Oct 30 22:19:45.852667 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:45.852632 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e8dced5-a68f-436b-ab03-da8d8e879315" path="/var/lib/kubelet/pods/2e8dced5-a68f-436b-ab03-da8d8e879315/volumes" Oct 30 22:19:50.487135 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:50.487100 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7"] Oct 30 22:19:50.487659 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:50.487392 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7" podUID="583871c1-8b4a-4145-a350-f3413686ad38" containerName="discovery" containerID="cri-o://84123bd6ca67e1721791fd49d2d218313167be4fc78df6035b024e41de75688d" gracePeriod=30 Oct 30 22:19:50.802503 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:50.802475 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7" Oct 30 22:19:50.910361 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:50.902654 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/583871c1-8b4a-4145-a350-f3413686ad38-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/583871c1-8b4a-4145-a350-f3413686ad38-local-certs\") pod \"583871c1-8b4a-4145-a350-f3413686ad38\" (UID: \"583871c1-8b4a-4145-a350-f3413686ad38\") " Oct 30 22:19:50.910361 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:50.902793 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/583871c1-8b4a-4145-a350-f3413686ad38-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/583871c1-8b4a-4145-a350-f3413686ad38-istio-csr-ca-configmap\") pod \"583871c1-8b4a-4145-a350-f3413686ad38\" (UID: \"583871c1-8b4a-4145-a350-f3413686ad38\") " Oct 30 22:19:50.910361 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:50.902844 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/583871c1-8b4a-4145-a350-f3413686ad38-istio-token\" (UniqueName: \"kubernetes.io/projected/583871c1-8b4a-4145-a350-f3413686ad38-istio-token\") pod \"583871c1-8b4a-4145-a350-f3413686ad38\" (UID: \"583871c1-8b4a-4145-a350-f3413686ad38\") " Oct 30 22:19:50.910361 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:50.902887 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/583871c1-8b4a-4145-a350-f3413686ad38-kube-api-access-kkqcd\" (UniqueName: \"kubernetes.io/projected/583871c1-8b4a-4145-a350-f3413686ad38-kube-api-access-kkqcd\") pod \"583871c1-8b4a-4145-a350-f3413686ad38\" (UID: \"583871c1-8b4a-4145-a350-f3413686ad38\") " Oct 30 22:19:50.910361 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:50.903007 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/583871c1-8b4a-4145-a350-f3413686ad38-cacerts\" (UniqueName: \"kubernetes.io/secret/583871c1-8b4a-4145-a350-f3413686ad38-cacerts\") pod \"583871c1-8b4a-4145-a350-f3413686ad38\" (UID: \"583871c1-8b4a-4145-a350-f3413686ad38\") " Oct 30 22:19:50.910361 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:50.903049 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/583871c1-8b4a-4145-a350-f3413686ad38-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/583871c1-8b4a-4145-a350-f3413686ad38-istio-csr-dns-cert\") pod \"583871c1-8b4a-4145-a350-f3413686ad38\" (UID: \"583871c1-8b4a-4145-a350-f3413686ad38\") " Oct 30 22:19:50.910361 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:50.903091 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/583871c1-8b4a-4145-a350-f3413686ad38-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/583871c1-8b4a-4145-a350-f3413686ad38-istio-kubeconfig\") pod \"583871c1-8b4a-4145-a350-f3413686ad38\" (UID: \"583871c1-8b4a-4145-a350-f3413686ad38\") " Oct 30 22:19:50.910361 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:50.904819 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/583871c1-8b4a-4145-a350-f3413686ad38-istio-csr-ca-configmap" pod "583871c1-8b4a-4145-a350-f3413686ad38" (UID: "583871c1-8b4a-4145-a350-f3413686ad38"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:19:50.957025 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:50.956384 5918 generic.go:356] "Generic (PLEG): container finished" podID="583871c1-8b4a-4145-a350-f3413686ad38" containerID="84123bd6ca67e1721791fd49d2d218313167be4fc78df6035b024e41de75688d" exitCode=0 Oct 30 22:19:50.957025 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:50.956484 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7" event={"ID":"583871c1-8b4a-4145-a350-f3413686ad38","Type":"ContainerDied","Data":"84123bd6ca67e1721791fd49d2d218313167be4fc78df6035b024e41de75688d"} Oct 30 22:19:50.957025 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:50.956530 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7" event={"ID":"583871c1-8b4a-4145-a350-f3413686ad38","Type":"ContainerDied","Data":"0d48b1d8ad28d68457ccbbb4eafe15fceba0d84fa4185aca8bbe7472f0534de7"} Oct 30 22:19:50.957025 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:50.956555 5918 scope.go:122] "RemoveContainer" containerID="84123bd6ca67e1721791fd49d2d218313167be4fc78df6035b024e41de75688d" Oct 30 22:19:50.957025 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:50.956707 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7" Oct 30 22:19:50.960694 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:50.960523 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/583871c1-8b4a-4145-a350-f3413686ad38-istio-kubeconfig" pod "583871c1-8b4a-4145-a350-f3413686ad38" (UID: "583871c1-8b4a-4145-a350-f3413686ad38"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:19:50.961148 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:50.961112 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583871c1-8b4a-4145-a350-f3413686ad38-local-certs" pod "583871c1-8b4a-4145-a350-f3413686ad38" (UID: "583871c1-8b4a-4145-a350-f3413686ad38"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:50.961535 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:50.961501 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/583871c1-8b4a-4145-a350-f3413686ad38-istio-csr-dns-cert" pod "583871c1-8b4a-4145-a350-f3413686ad38" (UID: "583871c1-8b4a-4145-a350-f3413686ad38"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:19:50.964418 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:50.964382 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/583871c1-8b4a-4145-a350-f3413686ad38-istio-token" pod "583871c1-8b4a-4145-a350-f3413686ad38" (UID: "583871c1-8b4a-4145-a350-f3413686ad38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:50.965863 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:50.965820 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/583871c1-8b4a-4145-a350-f3413686ad38-kube-api-access-kkqcd" pod "583871c1-8b4a-4145-a350-f3413686ad38" (UID: "583871c1-8b4a-4145-a350-f3413686ad38"). InnerVolumeSpecName "kube-api-access-kkqcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:50.996409 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:50.995372 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/583871c1-8b4a-4145-a350-f3413686ad38-cacerts" pod "583871c1-8b4a-4145-a350-f3413686ad38" (UID: "583871c1-8b4a-4145-a350-f3413686ad38"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:19:51.007057 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.006508 5918 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/583871c1-8b4a-4145-a350-f3413686ad38-local-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:51.007057 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.006545 5918 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/583871c1-8b4a-4145-a350-f3413686ad38-istio-csr-ca-configmap\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:51.007057 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.006563 5918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/583871c1-8b4a-4145-a350-f3413686ad38-istio-token\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:51.007057 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.006579 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkqcd\" (UniqueName: \"kubernetes.io/projected/583871c1-8b4a-4145-a350-f3413686ad38-kube-api-access-kkqcd\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:51.007057 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.006605 5918 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/583871c1-8b4a-4145-a350-f3413686ad38-cacerts\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:51.007057 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.006621 5918 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/583871c1-8b4a-4145-a350-f3413686ad38-istio-csr-dns-cert\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:51.007057 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.006635 5918 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/583871c1-8b4a-4145-a350-f3413686ad38-istio-kubeconfig\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:51.020149 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.018689 5918 scope.go:122] "RemoveContainer" containerID="84123bd6ca67e1721791fd49d2d218313167be4fc78df6035b024e41de75688d" Oct 30 22:19:51.020533 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:19:51.020415 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84123bd6ca67e1721791fd49d2d218313167be4fc78df6035b024e41de75688d\": container with ID starting with 84123bd6ca67e1721791fd49d2d218313167be4fc78df6035b024e41de75688d not found: ID does not exist" containerID="84123bd6ca67e1721791fd49d2d218313167be4fc78df6035b024e41de75688d" Oct 30 22:19:51.020533 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.020459 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84123bd6ca67e1721791fd49d2d218313167be4fc78df6035b024e41de75688d"} err="failed to get container status \"84123bd6ca67e1721791fd49d2d218313167be4fc78df6035b024e41de75688d\": rpc error: code = NotFound desc = could not find container \"84123bd6ca67e1721791fd49d2d218313167be4fc78df6035b024e41de75688d\": container with ID starting with 84123bd6ca67e1721791fd49d2d218313167be4fc78df6035b024e41de75688d not found: ID does not exist" Oct 30 22:19:51.140392 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.140351 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z"] Oct 30 22:19:51.141270 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.141246 5918 cpu_manager.go:414] "containerMap: removing container" podUID="583871c1-8b4a-4145-a350-f3413686ad38" containerName="discovery" Oct 30 22:19:51.141270 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.141268 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="583871c1-8b4a-4145-a350-f3413686ad38" containerName="discovery" Oct 30 22:19:51.141401 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.141346 5918 cpu_manager.go:414] "containerMap: removing container" podUID="2e8dced5-a68f-436b-ab03-da8d8e879315" containerName="istio-proxy" Oct 30 22:19:51.141401 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.141354 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e8dced5-a68f-436b-ab03-da8d8e879315" containerName="istio-proxy" Oct 30 22:19:51.141536 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.141521 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="583871c1-8b4a-4145-a350-f3413686ad38" containerName="discovery" Oct 30 22:19:51.141589 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.141553 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e8dced5-a68f-436b-ab03-da8d8e879315" containerName="istio-proxy" Oct 30 22:19:51.148021 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.147994 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:19:51.208712 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.208685 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-24aa4c-openshift-default-dockercfg-rgl52\"" Oct 30 22:19:51.209255 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.209230 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9afe2de-950a-495b-b234-4bab76e0dadc-istio-podinfo\") pod \"e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z\" (UID: \"c9afe2de-950a-495b-b234-4bab76e0dadc\") " pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:19:51.209324 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.209291 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9afe2de-950a-495b-b234-4bab76e0dadc-istio-envoy\") pod \"e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z\" (UID: \"c9afe2de-950a-495b-b234-4bab76e0dadc\") " pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:19:51.209440 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.209417 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwdpr\" (UniqueName: \"kubernetes.io/projected/c9afe2de-950a-495b-b234-4bab76e0dadc-kube-api-access-pwdpr\") pod \"e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z\" (UID: \"c9afe2de-950a-495b-b234-4bab76e0dadc\") " pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:19:51.209582 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.209562 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9afe2de-950a-495b-b234-4bab76e0dadc-workload-socket\") pod \"e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z\" (UID: \"c9afe2de-950a-495b-b234-4bab76e0dadc\") " pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:19:51.209631 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.209612 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9afe2de-950a-495b-b234-4bab76e0dadc-istio-data\") pod \"e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z\" (UID: \"c9afe2de-950a-495b-b234-4bab76e0dadc\") " pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:19:51.209677 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.209655 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9afe2de-950a-495b-b234-4bab76e0dadc-credential-socket\") pod \"e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z\" (UID: \"c9afe2de-950a-495b-b234-4bab76e0dadc\") " pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:19:51.209730 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.209717 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9afe2de-950a-495b-b234-4bab76e0dadc-workload-certs\") pod \"e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z\" (UID: \"c9afe2de-950a-495b-b234-4bab76e0dadc\") " pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:19:51.209784 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.209745 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9afe2de-950a-495b-b234-4bab76e0dadc-istiod-ca-cert\") pod \"e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z\" (UID: \"c9afe2de-950a-495b-b234-4bab76e0dadc\") " pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:19:51.209833 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.209819 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9afe2de-950a-495b-b234-4bab76e0dadc-istio-token\") pod \"e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z\" (UID: \"c9afe2de-950a-495b-b234-4bab76e0dadc\") " pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:19:51.217820 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.217791 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z"] Oct 30 22:19:51.293091 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:19:51.289290 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:19:51.311872 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.311833 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9afe2de-950a-495b-b234-4bab76e0dadc-istio-podinfo\") pod \"e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z\" (UID: \"c9afe2de-950a-495b-b234-4bab76e0dadc\") " pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:19:51.312052 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.311920 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9afe2de-950a-495b-b234-4bab76e0dadc-istio-envoy\") pod \"e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z\" (UID: \"c9afe2de-950a-495b-b234-4bab76e0dadc\") " pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:19:51.312052 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.311983 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwdpr\" (UniqueName: \"kubernetes.io/projected/c9afe2de-950a-495b-b234-4bab76e0dadc-kube-api-access-pwdpr\") pod \"e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z\" (UID: \"c9afe2de-950a-495b-b234-4bab76e0dadc\") " pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:19:51.312175 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.312092 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9afe2de-950a-495b-b234-4bab76e0dadc-workload-socket\") pod \"e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z\" (UID: \"c9afe2de-950a-495b-b234-4bab76e0dadc\") " pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:19:51.312175 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.312129 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9afe2de-950a-495b-b234-4bab76e0dadc-istio-data\") pod \"e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z\" (UID: \"c9afe2de-950a-495b-b234-4bab76e0dadc\") " pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:19:51.312326 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.312188 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9afe2de-950a-495b-b234-4bab76e0dadc-credential-socket\") pod \"e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z\" (UID: \"c9afe2de-950a-495b-b234-4bab76e0dadc\") " pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:19:51.312326 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.312250 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9afe2de-950a-495b-b234-4bab76e0dadc-workload-certs\") pod \"e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z\" (UID: \"c9afe2de-950a-495b-b234-4bab76e0dadc\") " pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:19:51.312326 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.312289 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9afe2de-950a-495b-b234-4bab76e0dadc-istiod-ca-cert\") pod \"e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z\" (UID: \"c9afe2de-950a-495b-b234-4bab76e0dadc\") " pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:19:51.312478 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.312345 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9afe2de-950a-495b-b234-4bab76e0dadc-istio-token\") pod \"e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z\" (UID: \"c9afe2de-950a-495b-b234-4bab76e0dadc\") " pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:19:51.313586 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.313562 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9afe2de-950a-495b-b234-4bab76e0dadc-istio-data\") pod \"e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z\" (UID: \"c9afe2de-950a-495b-b234-4bab76e0dadc\") " pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:19:51.313826 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.313807 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9afe2de-950a-495b-b234-4bab76e0dadc-workload-socket\") pod \"e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z\" (UID: \"c9afe2de-950a-495b-b234-4bab76e0dadc\") " pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:19:51.314060 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.314036 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9afe2de-950a-495b-b234-4bab76e0dadc-workload-certs\") pod \"e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z\" (UID: \"c9afe2de-950a-495b-b234-4bab76e0dadc\") " pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:19:51.314263 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.314235 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9afe2de-950a-495b-b234-4bab76e0dadc-credential-socket\") pod \"e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z\" (UID: \"c9afe2de-950a-495b-b234-4bab76e0dadc\") " pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:19:51.314679 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.314655 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9afe2de-950a-495b-b234-4bab76e0dadc-istiod-ca-cert\") pod \"e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z\" (UID: \"c9afe2de-950a-495b-b234-4bab76e0dadc\") " pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:19:51.316086 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.316061 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9afe2de-950a-495b-b234-4bab76e0dadc-istio-envoy\") pod \"e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z\" (UID: \"c9afe2de-950a-495b-b234-4bab76e0dadc\") " pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:19:51.316284 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.316267 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9afe2de-950a-495b-b234-4bab76e0dadc-istio-podinfo\") pod \"e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z\" (UID: \"c9afe2de-950a-495b-b234-4bab76e0dadc\") " pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:19:51.376273 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.376233 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9afe2de-950a-495b-b234-4bab76e0dadc-istio-token\") pod \"e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z\" (UID: \"c9afe2de-950a-495b-b234-4bab76e0dadc\") " pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:19:51.376560 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.376536 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwdpr\" (UniqueName: \"kubernetes.io/projected/c9afe2de-950a-495b-b234-4bab76e0dadc-kube-api-access-pwdpr\") pod \"e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z\" (UID: \"c9afe2de-950a-495b-b234-4bab76e0dadc\") " pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:19:51.399794 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.399764 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7"] Oct 30 22:19:51.409470 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.409443 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-pgvh7"] Oct 30 22:19:51.463409 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.463377 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:19:51.853031 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.852947 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="583871c1-8b4a-4145-a350-f3413686ad38" path="/var/lib/kubelet/pods/583871c1-8b4a-4145-a350-f3413686ad38/volumes" Oct 30 22:19:51.923625 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.923598 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z"] Oct 30 22:19:51.925058 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:19:51.925017 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9afe2de_950a_495b_b234_4bab76e0dadc.slice/crio-8e0dc47261f5fc0d5884d4e085e6ca182b323657070ab586d4d090b33a2763b3 WatchSource:0}: Error finding container 8e0dc47261f5fc0d5884d4e085e6ca182b323657070ab586d4d090b33a2763b3: Status 404 returned error can't find the container with id 8e0dc47261f5fc0d5884d4e085e6ca182b323657070ab586d4d090b33a2763b3 Oct 30 22:19:51.927741 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.927707 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:19:51.927828 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.927776 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:19:51.927828 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.927802 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:19:51.964892 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:51.964863 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" event={"ID":"c9afe2de-950a-495b-b234-4bab76e0dadc","Type":"ContainerStarted","Data":"8e0dc47261f5fc0d5884d4e085e6ca182b323657070ab586d4d090b33a2763b3"} Oct 30 22:19:52.206815 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:52.206777 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7"] Oct 30 22:19:52.207013 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:52.206989 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" podUID="845afe35-58c3-43c9-b276-66247d39c613" containerName="istio-proxy" containerID="cri-o://2d42a7149e6fb2265b23cd05a63f2afbb7efbf0c1b9ecac8680ecc4fa20369f0" gracePeriod=30 Oct 30 22:19:52.982598 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:52.982546 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" event={"ID":"c9afe2de-950a-495b-b234-4bab76e0dadc","Type":"ContainerStarted","Data":"b2dd8535cb8c97525862cf1e145697db1fa661b8f3cc41fb51a2e927ceacc805"} Oct 30 22:19:53.463924 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:53.463889 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:19:53.470929 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:53.470904 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:19:53.600164 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:53.599903 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" podStartSLOduration=2.5998856139999997 podStartE2EDuration="2.599885614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:53.114018177 +0000 UTC m=+1753.384716259" watchObservedRunningTime="2026-10-30 22:19:53.599885614 +0000 UTC m=+1753.870583712" Oct 30 22:19:53.990552 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:53.990524 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:19:53.991569 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:53.991545 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:19:54.120022 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:54.119986 5918 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:19:54.120181 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:54.120055 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" podUID="845afe35-58c3-43c9-b276-66247d39c613" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:19:55.781642 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:55.781591 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-81f37a-9454c4d75-fmvwr"] Oct 30 22:19:55.787177 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:55.787155 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-81f37a-9454c4d75-fmvwr" Oct 30 22:19:55.855987 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:55.855955 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-81f37a-9454c4d75-fmvwr"] Oct 30 22:19:55.902247 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:55.892913 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-81f37a-dockercfg-nkchp\"" Oct 30 22:19:55.902247 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:55.895464 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-81f37a\"" Oct 30 22:19:55.903822 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:55.903790 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f998d536-e995-48fb-9c20-c5cfbabaaa2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-81f37a-9454c4d75-fmvwr\" (UID: \"f998d536-e995-48fb-9c20-c5cfbabaaa2f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-81f37a-9454c4d75-fmvwr" Oct 30 22:19:55.904273 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:55.904253 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r827\" (UniqueName: \"kubernetes.io/projected/f998d536-e995-48fb-9c20-c5cfbabaaa2f-kube-api-access-8r827\") pod \"payload-pre-processing-e2e-api-a-81f37a-9454c4d75-fmvwr\" (UID: \"f998d536-e995-48fb-9c20-c5cfbabaaa2f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-81f37a-9454c4d75-fmvwr" Oct 30 22:19:56.030387 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:56.030342 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r827\" (UniqueName: \"kubernetes.io/projected/f998d536-e995-48fb-9c20-c5cfbabaaa2f-kube-api-access-8r827\") pod \"payload-pre-processing-e2e-api-a-81f37a-9454c4d75-fmvwr\" (UID: \"f998d536-e995-48fb-9c20-c5cfbabaaa2f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-81f37a-9454c4d75-fmvwr" Oct 30 22:19:56.030800 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:56.030776 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f998d536-e995-48fb-9c20-c5cfbabaaa2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-81f37a-9454c4d75-fmvwr\" (UID: \"f998d536-e995-48fb-9c20-c5cfbabaaa2f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-81f37a-9454c4d75-fmvwr" Oct 30 22:19:56.031941 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:56.031885 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f998d536-e995-48fb-9c20-c5cfbabaaa2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-81f37a-9454c4d75-fmvwr\" (UID: \"f998d536-e995-48fb-9c20-c5cfbabaaa2f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-81f37a-9454c4d75-fmvwr" Oct 30 22:19:56.095603 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:56.095576 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r827\" (UniqueName: \"kubernetes.io/projected/f998d536-e995-48fb-9c20-c5cfbabaaa2f-kube-api-access-8r827\") pod \"payload-pre-processing-e2e-api-a-81f37a-9454c4d75-fmvwr\" (UID: \"f998d536-e995-48fb-9c20-c5cfbabaaa2f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-81f37a-9454c4d75-fmvwr" Oct 30 22:19:56.106534 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:56.106504 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-81f37a-9454c4d75-fmvwr" Oct 30 22:19:56.500841 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:56.500813 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-81f37a-9454c4d75-fmvwr"] Oct 30 22:19:57.060017 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:57.059976 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-81f37a-9454c4d75-fmvwr" event={"ID":"f998d536-e995-48fb-9c20-c5cfbabaaa2f","Type":"ContainerStarted","Data":"8d143da20bcc86da56a8b143b96dd06e0d35fb07dcff4ecbda89e9dffff4771b"} Oct 30 22:19:57.060437 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:57.060071 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-81f37a-9454c4d75-fmvwr" event={"ID":"f998d536-e995-48fb-9c20-c5cfbabaaa2f","Type":"ContainerStarted","Data":"9da80486782d9eed4e0e734526192bd035fc18f2e8e23c3fd5def8d00ed34a23"} Oct 30 22:19:57.060437 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:57.060274 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-81f37a-9454c4d75-fmvwr" Oct 30 22:19:57.181787 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:57.181741 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-81f37a-9454c4d75-fmvwr" podStartSLOduration=2.181727545 podStartE2EDuration="2.181727545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:57.178971214 +0000 UTC m=+1757.449669314" watchObservedRunningTime="2026-10-30 22:19:57.181727545 +0000 UTC m=+1757.452425628" Oct 30 22:19:57.508398 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:57.508363 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" Oct 30 22:19:57.576663 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:57.576581 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/845afe35-58c3-43c9-b276-66247d39c613-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/845afe35-58c3-43c9-b276-66247d39c613-istio-envoy\") pod \"845afe35-58c3-43c9-b276-66247d39c613\" (UID: \"845afe35-58c3-43c9-b276-66247d39c613\") " Oct 30 22:19:57.576663 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:57.576619 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/845afe35-58c3-43c9-b276-66247d39c613-istio-token\" (UniqueName: \"kubernetes.io/projected/845afe35-58c3-43c9-b276-66247d39c613-istio-token\") pod \"845afe35-58c3-43c9-b276-66247d39c613\" (UID: \"845afe35-58c3-43c9-b276-66247d39c613\") " Oct 30 22:19:57.576890 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:57.576674 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/845afe35-58c3-43c9-b276-66247d39c613-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/845afe35-58c3-43c9-b276-66247d39c613-istio-data\") pod \"845afe35-58c3-43c9-b276-66247d39c613\" (UID: \"845afe35-58c3-43c9-b276-66247d39c613\") " Oct 30 22:19:57.576890 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:57.576713 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/845afe35-58c3-43c9-b276-66247d39c613-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/845afe35-58c3-43c9-b276-66247d39c613-istiod-ca-cert\") pod \"845afe35-58c3-43c9-b276-66247d39c613\" (UID: \"845afe35-58c3-43c9-b276-66247d39c613\") " Oct 30 22:19:57.576890 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:57.576773 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/845afe35-58c3-43c9-b276-66247d39c613-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/845afe35-58c3-43c9-b276-66247d39c613-credential-socket\") pod \"845afe35-58c3-43c9-b276-66247d39c613\" (UID: \"845afe35-58c3-43c9-b276-66247d39c613\") " Oct 30 22:19:57.577052 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:57.576928 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/845afe35-58c3-43c9-b276-66247d39c613-kube-api-access-h5mqm\" (UniqueName: \"kubernetes.io/projected/845afe35-58c3-43c9-b276-66247d39c613-kube-api-access-h5mqm\") pod \"845afe35-58c3-43c9-b276-66247d39c613\" (UID: \"845afe35-58c3-43c9-b276-66247d39c613\") " Oct 30 22:19:57.577052 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:57.576948 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/845afe35-58c3-43c9-b276-66247d39c613-istio-data" pod "845afe35-58c3-43c9-b276-66247d39c613" (UID: "845afe35-58c3-43c9-b276-66247d39c613"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:57.577052 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:57.576997 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/845afe35-58c3-43c9-b276-66247d39c613-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/845afe35-58c3-43c9-b276-66247d39c613-workload-socket\") pod \"845afe35-58c3-43c9-b276-66247d39c613\" (UID: \"845afe35-58c3-43c9-b276-66247d39c613\") " Oct 30 22:19:57.577052 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:57.577031 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/845afe35-58c3-43c9-b276-66247d39c613-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/845afe35-58c3-43c9-b276-66247d39c613-workload-certs\") pod \"845afe35-58c3-43c9-b276-66247d39c613\" (UID: \"845afe35-58c3-43c9-b276-66247d39c613\") " Oct 30 22:19:57.577314 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:57.577070 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/845afe35-58c3-43c9-b276-66247d39c613-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/845afe35-58c3-43c9-b276-66247d39c613-istio-podinfo\") pod \"845afe35-58c3-43c9-b276-66247d39c613\" (UID: \"845afe35-58c3-43c9-b276-66247d39c613\") " Oct 30 22:19:57.577434 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:57.577070 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/845afe35-58c3-43c9-b276-66247d39c613-credential-socket" pod "845afe35-58c3-43c9-b276-66247d39c613" (UID: "845afe35-58c3-43c9-b276-66247d39c613"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:57.577516 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:57.577171 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/845afe35-58c3-43c9-b276-66247d39c613-istiod-ca-cert" pod "845afe35-58c3-43c9-b276-66247d39c613" (UID: "845afe35-58c3-43c9-b276-66247d39c613"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:19:57.577516 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:57.577312 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/845afe35-58c3-43c9-b276-66247d39c613-workload-socket" pod "845afe35-58c3-43c9-b276-66247d39c613" (UID: "845afe35-58c3-43c9-b276-66247d39c613"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:57.577516 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:57.577335 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/845afe35-58c3-43c9-b276-66247d39c613-workload-certs" pod "845afe35-58c3-43c9-b276-66247d39c613" (UID: "845afe35-58c3-43c9-b276-66247d39c613"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:57.577762 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:57.577744 5918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/845afe35-58c3-43c9-b276-66247d39c613-istio-data\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:57.577999 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:57.577765 5918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/845afe35-58c3-43c9-b276-66247d39c613-istiod-ca-cert\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:57.577999 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:57.577776 5918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/845afe35-58c3-43c9-b276-66247d39c613-credential-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:57.577999 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:57.577788 5918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/845afe35-58c3-43c9-b276-66247d39c613-workload-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:57.577999 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:57.577802 5918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/845afe35-58c3-43c9-b276-66247d39c613-workload-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:57.579085 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:57.579060 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/845afe35-58c3-43c9-b276-66247d39c613-istio-token" pod "845afe35-58c3-43c9-b276-66247d39c613" (UID: "845afe35-58c3-43c9-b276-66247d39c613"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:57.579495 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:57.579475 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/845afe35-58c3-43c9-b276-66247d39c613-istio-podinfo" pod "845afe35-58c3-43c9-b276-66247d39c613" (UID: "845afe35-58c3-43c9-b276-66247d39c613"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:19:57.579495 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:57.579483 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/845afe35-58c3-43c9-b276-66247d39c613-istio-envoy" pod "845afe35-58c3-43c9-b276-66247d39c613" (UID: "845afe35-58c3-43c9-b276-66247d39c613"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:57.579649 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:57.579632 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/845afe35-58c3-43c9-b276-66247d39c613-kube-api-access-h5mqm" pod "845afe35-58c3-43c9-b276-66247d39c613" (UID: "845afe35-58c3-43c9-b276-66247d39c613"). InnerVolumeSpecName "kube-api-access-h5mqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:57.679008 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:57.678975 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5mqm\" (UniqueName: \"kubernetes.io/projected/845afe35-58c3-43c9-b276-66247d39c613-kube-api-access-h5mqm\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:57.679008 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:57.679002 5918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/845afe35-58c3-43c9-b276-66247d39c613-istio-podinfo\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:57.679008 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:57.679012 5918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/845afe35-58c3-43c9-b276-66247d39c613-istio-envoy\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:57.679265 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:57.679022 5918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/845afe35-58c3-43c9-b276-66247d39c613-istio-token\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:58.068922 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:58.068882 5918 generic.go:356] "Generic (PLEG): container finished" podID="845afe35-58c3-43c9-b276-66247d39c613" containerID="2d42a7149e6fb2265b23cd05a63f2afbb7efbf0c1b9ecac8680ecc4fa20369f0" exitCode=0 Oct 30 22:19:58.070070 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:58.070047 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" Oct 30 22:19:58.070719 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:58.070699 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" event={"ID":"845afe35-58c3-43c9-b276-66247d39c613","Type":"ContainerDied","Data":"2d42a7149e6fb2265b23cd05a63f2afbb7efbf0c1b9ecac8680ecc4fa20369f0"} Oct 30 22:19:58.070778 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:58.070751 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7" event={"ID":"845afe35-58c3-43c9-b276-66247d39c613","Type":"ContainerDied","Data":"022e83e2c0b3895bc2dc12755e4af19a43f259233af25085fc338c6ff5516310"} Oct 30 22:19:58.070820 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:58.070782 5918 scope.go:122] "RemoveContainer" containerID="2d42a7149e6fb2265b23cd05a63f2afbb7efbf0c1b9ecac8680ecc4fa20369f0" Oct 30 22:19:58.085943 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:58.085921 5918 scope.go:122] "RemoveContainer" containerID="2d42a7149e6fb2265b23cd05a63f2afbb7efbf0c1b9ecac8680ecc4fa20369f0" Oct 30 22:19:58.086532 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:19:58.086495 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d42a7149e6fb2265b23cd05a63f2afbb7efbf0c1b9ecac8680ecc4fa20369f0\": container with ID starting with 2d42a7149e6fb2265b23cd05a63f2afbb7efbf0c1b9ecac8680ecc4fa20369f0 not found: ID does not exist" containerID="2d42a7149e6fb2265b23cd05a63f2afbb7efbf0c1b9ecac8680ecc4fa20369f0" Oct 30 22:19:58.086640 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:58.086544 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d42a7149e6fb2265b23cd05a63f2afbb7efbf0c1b9ecac8680ecc4fa20369f0"} err="failed to get container status \"2d42a7149e6fb2265b23cd05a63f2afbb7efbf0c1b9ecac8680ecc4fa20369f0\": rpc error: code = NotFound desc = could not find container \"2d42a7149e6fb2265b23cd05a63f2afbb7efbf0c1b9ecac8680ecc4fa20369f0\": container with ID starting with 2d42a7149e6fb2265b23cd05a63f2afbb7efbf0c1b9ecac8680ecc4fa20369f0 not found: ID does not exist" Oct 30 22:19:58.244019 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:58.243980 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7"] Oct 30 22:19:58.250561 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:58.250534 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-q54d7"] Oct 30 22:19:59.853005 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:19:59.852975 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="845afe35-58c3-43c9-b276-66247d39c613" path="/var/lib/kubelet/pods/845afe35-58c3-43c9-b276-66247d39c613/volumes" Oct 30 22:20:01.739973 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:01.739938 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-j52jw"] Oct 30 22:20:01.740820 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:01.740799 5918 cpu_manager.go:414] "containerMap: removing container" podUID="845afe35-58c3-43c9-b276-66247d39c613" containerName="istio-proxy" Oct 30 22:20:01.740933 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:01.740823 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="845afe35-58c3-43c9-b276-66247d39c613" containerName="istio-proxy" Oct 30 22:20:01.741034 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:01.741020 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="845afe35-58c3-43c9-b276-66247d39c613" containerName="istio-proxy" Oct 30 22:20:01.744543 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:01.744520 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-j52jw" Oct 30 22:20:01.806577 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:01.806539 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-j52jw"] Oct 30 22:20:01.842263 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:01.842202 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a821fb11-c9cd-4e19-8710-3d01c7c42955-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-j52jw\" (UID: \"a821fb11-c9cd-4e19-8710-3d01c7c42955\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-j52jw" Oct 30 22:20:01.842445 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:01.842340 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdl5x\" (UniqueName: \"kubernetes.io/projected/a821fb11-c9cd-4e19-8710-3d01c7c42955-kube-api-access-fdl5x\") pod \"payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-j52jw\" (UID: \"a821fb11-c9cd-4e19-8710-3d01c7c42955\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-j52jw" Oct 30 22:20:01.875243 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:01.864496 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-24aa4c\"" Oct 30 22:20:01.875243 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:01.864650 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-24aa4c-dockercfg-2l8fr\"" Oct 30 22:20:01.947236 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:01.943869 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdl5x\" (UniqueName: \"kubernetes.io/projected/a821fb11-c9cd-4e19-8710-3d01c7c42955-kube-api-access-fdl5x\") pod \"payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-j52jw\" (UID: \"a821fb11-c9cd-4e19-8710-3d01c7c42955\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-j52jw" Oct 30 22:20:01.947236 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:01.944208 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a821fb11-c9cd-4e19-8710-3d01c7c42955-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-j52jw\" (UID: \"a821fb11-c9cd-4e19-8710-3d01c7c42955\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-j52jw" Oct 30 22:20:01.947236 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:01.945363 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a821fb11-c9cd-4e19-8710-3d01c7c42955-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-j52jw\" (UID: \"a821fb11-c9cd-4e19-8710-3d01c7c42955\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-j52jw" Oct 30 22:20:01.968244 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:01.964197 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-8p6dn"] Oct 30 22:20:01.991528 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:01.990624 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-8p6dn" Oct 30 22:20:02.010970 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.010939 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdl5x\" (UniqueName: \"kubernetes.io/projected/a821fb11-c9cd-4e19-8710-3d01c7c42955-kube-api-access-fdl5x\") pod \"payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-j52jw\" (UID: \"a821fb11-c9cd-4e19-8710-3d01c7c42955\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-j52jw" Oct 30 22:20:02.039338 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.039301 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-8p6dn"] Oct 30 22:20:02.046094 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.046062 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44gm5\" (UniqueName: \"kubernetes.io/projected/035926cf-f020-4c5b-ad47-7b69e12cf03f-kube-api-access-44gm5\") pod \"payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-8p6dn\" (UID: \"035926cf-f020-4c5b-ad47-7b69e12cf03f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-8p6dn" Oct 30 22:20:02.046300 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.046127 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/035926cf-f020-4c5b-ad47-7b69e12cf03f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-8p6dn\" (UID: \"035926cf-f020-4c5b-ad47-7b69e12cf03f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-8p6dn" Oct 30 22:20:02.059000 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.058950 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-j52jw" Oct 30 22:20:02.148235 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.147346 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44gm5\" (UniqueName: \"kubernetes.io/projected/035926cf-f020-4c5b-ad47-7b69e12cf03f-kube-api-access-44gm5\") pod \"payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-8p6dn\" (UID: \"035926cf-f020-4c5b-ad47-7b69e12cf03f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-8p6dn" Oct 30 22:20:02.148235 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.147435 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/035926cf-f020-4c5b-ad47-7b69e12cf03f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-8p6dn\" (UID: \"035926cf-f020-4c5b-ad47-7b69e12cf03f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-8p6dn" Oct 30 22:20:02.163932 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.160088 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/035926cf-f020-4c5b-ad47-7b69e12cf03f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-8p6dn\" (UID: \"035926cf-f020-4c5b-ad47-7b69e12cf03f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-8p6dn" Oct 30 22:20:02.229559 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.229523 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44gm5\" (UniqueName: \"kubernetes.io/projected/035926cf-f020-4c5b-ad47-7b69e12cf03f-kube-api-access-44gm5\") pod \"payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-8p6dn\" (UID: \"035926cf-f020-4c5b-ad47-7b69e12cf03f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-8p6dn" Oct 30 22:20:02.295114 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.286160 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr"] Oct 30 22:20:02.295666 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.295634 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:20:02.311538 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.311334 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-8p6dn" Oct 30 22:20:02.332905 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.332792 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr"] Oct 30 22:20:02.351305 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.351257 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-4511b0-openshift-default-dockercfg-c2qbt\"" Oct 30 22:20:02.353735 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.352976 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0143cf33-1443-4232-8b8e-2f7e448b78c8-workload-socket\") pod \"e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr\" (UID: \"0143cf33-1443-4232-8b8e-2f7e448b78c8\") " pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:20:02.353735 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.353072 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0143cf33-1443-4232-8b8e-2f7e448b78c8-credential-socket\") pod \"e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr\" (UID: \"0143cf33-1443-4232-8b8e-2f7e448b78c8\") " pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:20:02.353735 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.353157 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0143cf33-1443-4232-8b8e-2f7e448b78c8-istio-podinfo\") pod \"e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr\" (UID: \"0143cf33-1443-4232-8b8e-2f7e448b78c8\") " pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:20:02.353735 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.353198 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0143cf33-1443-4232-8b8e-2f7e448b78c8-istio-data\") pod \"e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr\" (UID: \"0143cf33-1443-4232-8b8e-2f7e448b78c8\") " pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:20:02.353735 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.353241 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0143cf33-1443-4232-8b8e-2f7e448b78c8-istiod-ca-cert\") pod \"e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr\" (UID: \"0143cf33-1443-4232-8b8e-2f7e448b78c8\") " pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:20:02.353735 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.353377 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpl8h\" (UniqueName: \"kubernetes.io/projected/0143cf33-1443-4232-8b8e-2f7e448b78c8-kube-api-access-lpl8h\") pod \"e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr\" (UID: \"0143cf33-1443-4232-8b8e-2f7e448b78c8\") " pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:20:02.353735 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.353407 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0143cf33-1443-4232-8b8e-2f7e448b78c8-workload-certs\") pod \"e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr\" (UID: \"0143cf33-1443-4232-8b8e-2f7e448b78c8\") " pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:20:02.353735 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.353426 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0143cf33-1443-4232-8b8e-2f7e448b78c8-istio-token\") pod \"e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr\" (UID: \"0143cf33-1443-4232-8b8e-2f7e448b78c8\") " pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:20:02.353735 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.353519 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0143cf33-1443-4232-8b8e-2f7e448b78c8-istio-envoy\") pod \"e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr\" (UID: \"0143cf33-1443-4232-8b8e-2f7e448b78c8\") " pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:20:02.463107 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.463065 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0143cf33-1443-4232-8b8e-2f7e448b78c8-credential-socket\") pod \"e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr\" (UID: \"0143cf33-1443-4232-8b8e-2f7e448b78c8\") " pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:20:02.463356 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.463179 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0143cf33-1443-4232-8b8e-2f7e448b78c8-istio-podinfo\") pod \"e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr\" (UID: \"0143cf33-1443-4232-8b8e-2f7e448b78c8\") " pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:20:02.463356 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.463245 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0143cf33-1443-4232-8b8e-2f7e448b78c8-istio-data\") pod \"e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr\" (UID: \"0143cf33-1443-4232-8b8e-2f7e448b78c8\") " pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:20:02.463356 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.463274 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0143cf33-1443-4232-8b8e-2f7e448b78c8-istiod-ca-cert\") pod \"e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr\" (UID: \"0143cf33-1443-4232-8b8e-2f7e448b78c8\") " pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:20:02.463356 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.463340 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpl8h\" (UniqueName: \"kubernetes.io/projected/0143cf33-1443-4232-8b8e-2f7e448b78c8-kube-api-access-lpl8h\") pod \"e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr\" (UID: \"0143cf33-1443-4232-8b8e-2f7e448b78c8\") " pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:20:02.463594 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.463388 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0143cf33-1443-4232-8b8e-2f7e448b78c8-workload-certs\") pod \"e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr\" (UID: \"0143cf33-1443-4232-8b8e-2f7e448b78c8\") " pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:20:02.463594 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.463540 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0143cf33-1443-4232-8b8e-2f7e448b78c8-istio-token\") pod \"e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr\" (UID: \"0143cf33-1443-4232-8b8e-2f7e448b78c8\") " pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:20:02.463741 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.463723 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0143cf33-1443-4232-8b8e-2f7e448b78c8-istio-envoy\") pod \"e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr\" (UID: \"0143cf33-1443-4232-8b8e-2f7e448b78c8\") " pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:20:02.463814 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.463799 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0143cf33-1443-4232-8b8e-2f7e448b78c8-workload-socket\") pod \"e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr\" (UID: \"0143cf33-1443-4232-8b8e-2f7e448b78c8\") " pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:20:02.474252 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.464363 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0143cf33-1443-4232-8b8e-2f7e448b78c8-workload-socket\") pod \"e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr\" (UID: \"0143cf33-1443-4232-8b8e-2f7e448b78c8\") " pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:20:02.474252 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.464622 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0143cf33-1443-4232-8b8e-2f7e448b78c8-credential-socket\") pod \"e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr\" (UID: \"0143cf33-1443-4232-8b8e-2f7e448b78c8\") " pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:20:02.475973 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.475938 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0143cf33-1443-4232-8b8e-2f7e448b78c8-istio-data\") pod \"e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr\" (UID: \"0143cf33-1443-4232-8b8e-2f7e448b78c8\") " pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:20:02.477624 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.476732 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0143cf33-1443-4232-8b8e-2f7e448b78c8-istiod-ca-cert\") pod \"e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr\" (UID: \"0143cf33-1443-4232-8b8e-2f7e448b78c8\") " pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:20:02.477624 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.477287 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0143cf33-1443-4232-8b8e-2f7e448b78c8-workload-certs\") pod \"e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr\" (UID: \"0143cf33-1443-4232-8b8e-2f7e448b78c8\") " pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:20:02.478719 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.478692 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0143cf33-1443-4232-8b8e-2f7e448b78c8-istio-podinfo\") pod \"e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr\" (UID: \"0143cf33-1443-4232-8b8e-2f7e448b78c8\") " pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:20:02.481570 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.481521 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0143cf33-1443-4232-8b8e-2f7e448b78c8-istio-envoy\") pod \"e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr\" (UID: \"0143cf33-1443-4232-8b8e-2f7e448b78c8\") " pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:20:02.540146 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.540118 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpl8h\" (UniqueName: \"kubernetes.io/projected/0143cf33-1443-4232-8b8e-2f7e448b78c8-kube-api-access-lpl8h\") pod \"e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr\" (UID: \"0143cf33-1443-4232-8b8e-2f7e448b78c8\") " pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:20:02.541548 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.541527 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0143cf33-1443-4232-8b8e-2f7e448b78c8-istio-token\") pod \"e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr\" (UID: \"0143cf33-1443-4232-8b8e-2f7e448b78c8\") " pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:20:02.547918 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.547884 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-j52jw"] Oct 30 22:20:02.549037 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:20:02.549010 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda821fb11_c9cd_4e19_8710_3d01c7c42955.slice/crio-e1f928cf4724ea1eaa25ce1232a602c4001c96f479630266d1557a789a65dddd WatchSource:0}: Error finding container e1f928cf4724ea1eaa25ce1232a602c4001c96f479630266d1557a789a65dddd: Status 404 returned error can't find the container with id e1f928cf4724ea1eaa25ce1232a602c4001c96f479630266d1557a789a65dddd Oct 30 22:20:02.617075 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.617048 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:20:02.723646 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:20:02.723605 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod035926cf_f020_4c5b_ad47_7b69e12cf03f.slice/crio-956353c5f047a3bed5851e386efbfc29f75a864068adfc7a82d728f20f7aff21 WatchSource:0}: Error finding container 956353c5f047a3bed5851e386efbfc29f75a864068adfc7a82d728f20f7aff21: Status 404 returned error can't find the container with id 956353c5f047a3bed5851e386efbfc29f75a864068adfc7a82d728f20f7aff21 Oct 30 22:20:02.724029 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:02.724003 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-8p6dn"] Oct 30 22:20:03.027792 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:03.027753 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr"] Oct 30 22:20:03.054655 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:03.054607 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:20:03.054820 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:03.054706 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:20:03.054820 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:03.054747 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:20:03.133500 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:03.133473 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-8p6dn" event={"ID":"035926cf-f020-4c5b-ad47-7b69e12cf03f","Type":"ContainerStarted","Data":"485b973e528b4d6c5ce7684f83e9be5b7dac16ac34fffcc2a07bbbf7d49b3f08"} Oct 30 22:20:03.133630 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:03.133521 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-8p6dn" event={"ID":"035926cf-f020-4c5b-ad47-7b69e12cf03f","Type":"ContainerStarted","Data":"956353c5f047a3bed5851e386efbfc29f75a864068adfc7a82d728f20f7aff21"} Oct 30 22:20:03.133704 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:03.133671 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-8p6dn" Oct 30 22:20:03.135071 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:03.135034 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" event={"ID":"0143cf33-1443-4232-8b8e-2f7e448b78c8","Type":"ContainerStarted","Data":"0c410ad0f1c205974423da976be1fc12ff3da30181f758179c779e689fce3128"} Oct 30 22:20:03.137708 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:03.137675 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-j52jw" event={"ID":"a821fb11-c9cd-4e19-8710-3d01c7c42955","Type":"ContainerStarted","Data":"b756979dfc3580baac429da6c0dad365f31ed7d8b98499a037589208125e6f79"} Oct 30 22:20:03.137833 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:03.137726 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-j52jw" event={"ID":"a821fb11-c9cd-4e19-8710-3d01c7c42955","Type":"ContainerStarted","Data":"e1f928cf4724ea1eaa25ce1232a602c4001c96f479630266d1557a789a65dddd"} Oct 30 22:20:03.137833 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:03.137755 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-j52jw" Oct 30 22:20:03.253952 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:03.253901 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-8p6dn" podStartSLOduration=2.253887916 podStartE2EDuration="2.253887916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:03.252983408 +0000 UTC m=+1763.523681495" watchObservedRunningTime="2026-10-30 22:20:03.253887916 +0000 UTC m=+1763.524586000" Oct 30 22:20:03.383847 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:03.376920 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-j52jw" podStartSLOduration=2.376900658 podStartE2EDuration="2.376900658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:03.37305376 +0000 UTC m=+1763.643751845" watchObservedRunningTime="2026-10-30 22:20:03.376900658 +0000 UTC m=+1763.647598744" Oct 30 22:20:04.147856 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:04.147816 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" event={"ID":"0143cf33-1443-4232-8b8e-2f7e448b78c8","Type":"ContainerStarted","Data":"ce65856e6076097a65eae2b4ba21cbd54253e07cd10d5936e91128498208645f"} Oct 30 22:20:04.323477 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:04.323431 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" podStartSLOduration=2.323418948 podStartE2EDuration="2.323418948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:04.322445959 +0000 UTC m=+1764.593144042" watchObservedRunningTime="2026-10-30 22:20:04.323418948 +0000 UTC m=+1764.594117031" Oct 30 22:20:04.617428 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:04.617393 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:20:04.625012 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:04.624982 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:20:05.155725 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:05.155691 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:20:05.156624 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:05.156600 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:20:08.073801 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:08.073775 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-81f37a-9454c4d75-fmvwr" Oct 30 22:20:14.151697 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:14.151668 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-j52jw" Oct 30 22:20:14.152118 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:14.151733 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-8p6dn" Oct 30 22:20:21.275807 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:20:21.275771 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:20:31.427085 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:31.419906 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ff338b-6799d5d75c-bhrjf"] Oct 30 22:20:31.450571 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:31.449446 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ff338b-6799d5d75c-bhrjf" Oct 30 22:20:31.470731 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:31.470662 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ff338b-6799d5d75c-bhrjf"] Oct 30 22:20:31.564661 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:31.564631 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23bb4082-3d3d-415c-afb6-9f0d547a064f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ff338b-6799d5d75c-bhrjf\" (UID: \"23bb4082-3d3d-415c-afb6-9f0d547a064f\") " pod="openshift-ingress/payload-processing-e2e-api-b-ff338b-6799d5d75c-bhrjf" Oct 30 22:20:31.564840 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:31.564825 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqtcw\" (UniqueName: \"kubernetes.io/projected/23bb4082-3d3d-415c-afb6-9f0d547a064f-kube-api-access-kqtcw\") pod \"payload-processing-e2e-api-b-ff338b-6799d5d75c-bhrjf\" (UID: \"23bb4082-3d3d-415c-afb6-9f0d547a064f\") " pod="openshift-ingress/payload-processing-e2e-api-b-ff338b-6799d5d75c-bhrjf" Oct 30 22:20:31.567529 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:31.567512 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-ff338b\"" Oct 30 22:20:31.567529 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:31.567524 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-ff338b-dockercfg-rxdvr\"" Oct 30 22:20:31.665668 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:31.665631 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqtcw\" (UniqueName: \"kubernetes.io/projected/23bb4082-3d3d-415c-afb6-9f0d547a064f-kube-api-access-kqtcw\") pod \"payload-processing-e2e-api-b-ff338b-6799d5d75c-bhrjf\" (UID: \"23bb4082-3d3d-415c-afb6-9f0d547a064f\") " pod="openshift-ingress/payload-processing-e2e-api-b-ff338b-6799d5d75c-bhrjf" Oct 30 22:20:31.665840 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:31.665820 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23bb4082-3d3d-415c-afb6-9f0d547a064f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ff338b-6799d5d75c-bhrjf\" (UID: \"23bb4082-3d3d-415c-afb6-9f0d547a064f\") " pod="openshift-ingress/payload-processing-e2e-api-b-ff338b-6799d5d75c-bhrjf" Oct 30 22:20:31.666592 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:31.666572 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23bb4082-3d3d-415c-afb6-9f0d547a064f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ff338b-6799d5d75c-bhrjf\" (UID: \"23bb4082-3d3d-415c-afb6-9f0d547a064f\") " pod="openshift-ingress/payload-processing-e2e-api-b-ff338b-6799d5d75c-bhrjf" Oct 30 22:20:31.729068 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:31.728990 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqtcw\" (UniqueName: \"kubernetes.io/projected/23bb4082-3d3d-415c-afb6-9f0d547a064f-kube-api-access-kqtcw\") pod \"payload-processing-e2e-api-b-ff338b-6799d5d75c-bhrjf\" (UID: \"23bb4082-3d3d-415c-afb6-9f0d547a064f\") " pod="openshift-ingress/payload-processing-e2e-api-b-ff338b-6799d5d75c-bhrjf" Oct 30 22:20:31.811988 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:31.811955 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ff338b-6799d5d75c-bhrjf" Oct 30 22:20:32.195923 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:32.195888 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ff338b-6799d5d75c-bhrjf"] Oct 30 22:20:32.482431 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:32.482332 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ff338b-6799d5d75c-bhrjf" event={"ID":"23bb4082-3d3d-415c-afb6-9f0d547a064f","Type":"ContainerStarted","Data":"07c684721254c83f16bdec17426357d9c96cc233ddd9cf180af1558492f1524f"} Oct 30 22:20:32.482431 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:32.482395 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ff338b-6799d5d75c-bhrjf" event={"ID":"23bb4082-3d3d-415c-afb6-9f0d547a064f","Type":"ContainerStarted","Data":"5e353b71a3e407684a91ba8524a1d5f8bd51757fa91dc601bce66303a7aa58ec"} Oct 30 22:20:32.482431 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:32.482424 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-ff338b-6799d5d75c-bhrjf" Oct 30 22:20:32.627117 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:32.627063 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-ff338b-6799d5d75c-bhrjf" podStartSLOduration=1.6270474350000002 podStartE2EDuration="1.627047435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:32.623863319 +0000 UTC m=+1792.894561400" watchObservedRunningTime="2026-10-30 22:20:32.627047435 +0000 UTC m=+1792.897745519" Oct 30 22:20:43.491695 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:20:43.491668 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-ff338b-6799d5d75c-bhrjf" Oct 30 22:20:51.278699 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:20:51.278612 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:21:01.053090 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:01.053059 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-f2qhs"] Oct 30 22:21:01.058547 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:01.058527 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-f2qhs" Oct 30 22:21:01.116348 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:01.116309 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-f2qhs"] Oct 30 22:21:01.133930 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:01.133897 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b159c891-8623-463e-b65f-fc1f9d2dd2cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-f2qhs\" (UID: \"b159c891-8623-463e-b65f-fc1f9d2dd2cb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-f2qhs" Oct 30 22:21:01.134266 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:01.134199 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwkz5\" (UniqueName: \"kubernetes.io/projected/b159c891-8623-463e-b65f-fc1f9d2dd2cb-kube-api-access-dwkz5\") pod \"payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-f2qhs\" (UID: \"b159c891-8623-463e-b65f-fc1f9d2dd2cb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-f2qhs" Oct 30 22:21:01.162956 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:01.162930 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-7cfd50\"" Oct 30 22:21:01.162956 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:01.162931 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-7cfd50-dockercfg-pcllc\"" Oct 30 22:21:01.240287 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:01.240241 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwkz5\" (UniqueName: \"kubernetes.io/projected/b159c891-8623-463e-b65f-fc1f9d2dd2cb-kube-api-access-dwkz5\") pod \"payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-f2qhs\" (UID: \"b159c891-8623-463e-b65f-fc1f9d2dd2cb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-f2qhs" Oct 30 22:21:01.242832 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:01.240727 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b159c891-8623-463e-b65f-fc1f9d2dd2cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-f2qhs\" (UID: \"b159c891-8623-463e-b65f-fc1f9d2dd2cb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-f2qhs" Oct 30 22:21:01.244121 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:01.244094 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b159c891-8623-463e-b65f-fc1f9d2dd2cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-f2qhs\" (UID: \"b159c891-8623-463e-b65f-fc1f9d2dd2cb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-f2qhs" Oct 30 22:21:01.314294 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:01.311292 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwkz5\" (UniqueName: \"kubernetes.io/projected/b159c891-8623-463e-b65f-fc1f9d2dd2cb-kube-api-access-dwkz5\") pod \"payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-f2qhs\" (UID: \"b159c891-8623-463e-b65f-fc1f9d2dd2cb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-f2qhs" Oct 30 22:21:01.372568 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:01.372532 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-f2qhs" Oct 30 22:21:01.754854 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:01.754827 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-f2qhs"] Oct 30 22:21:01.756293 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:21:01.756263 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb159c891_8623_463e_b65f_fc1f9d2dd2cb.slice/crio-4f4ea24d68f14a73aaea75e7572cffbd76bdae4e5f4cb46649fa93bd38f37b9b WatchSource:0}: Error finding container 4f4ea24d68f14a73aaea75e7572cffbd76bdae4e5f4cb46649fa93bd38f37b9b: Status 404 returned error can't find the container with id 4f4ea24d68f14a73aaea75e7572cffbd76bdae4e5f4cb46649fa93bd38f37b9b Oct 30 22:21:02.700847 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:02.700808 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-f2qhs" event={"ID":"b159c891-8623-463e-b65f-fc1f9d2dd2cb","Type":"ContainerStarted","Data":"1972fa778b46df012159e1d23a7e1568db444bab9bea0fed1af2c9bbea2dee62"} Oct 30 22:21:02.701326 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:02.700871 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-f2qhs" event={"ID":"b159c891-8623-463e-b65f-fc1f9d2dd2cb","Type":"ContainerStarted","Data":"4f4ea24d68f14a73aaea75e7572cffbd76bdae4e5f4cb46649fa93bd38f37b9b"} Oct 30 22:21:02.701326 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:02.701030 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-f2qhs" Oct 30 22:21:02.820789 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:02.820735 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-f2qhs" podStartSLOduration=1.82071961 podStartE2EDuration="1.82071961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:02.820341854 +0000 UTC m=+1823.091039940" watchObservedRunningTime="2026-10-30 22:21:02.82071961 +0000 UTC m=+1823.091417693" Oct 30 22:21:12.961309 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:12.961271 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2"] Oct 30 22:21:12.967099 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:12.967075 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:21:13.036285 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:13.025198 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-18edc4-openshift-default-dockercfg-srxg4\"" Oct 30 22:21:13.133539 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:13.133448 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/147ff712-6d65-4184-9125-9ae9391707dd-workload-certs\") pod \"e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2\" (UID: \"147ff712-6d65-4184-9125-9ae9391707dd\") " pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:21:13.133754 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:13.133565 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/147ff712-6d65-4184-9125-9ae9391707dd-istio-envoy\") pod \"e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2\" (UID: \"147ff712-6d65-4184-9125-9ae9391707dd\") " pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:21:13.133754 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:13.133646 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/147ff712-6d65-4184-9125-9ae9391707dd-workload-socket\") pod \"e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2\" (UID: \"147ff712-6d65-4184-9125-9ae9391707dd\") " pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:21:13.133754 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:13.133682 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/147ff712-6d65-4184-9125-9ae9391707dd-credential-socket\") pod \"e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2\" (UID: \"147ff712-6d65-4184-9125-9ae9391707dd\") " pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:21:13.133754 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:13.133713 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/147ff712-6d65-4184-9125-9ae9391707dd-istio-podinfo\") pod \"e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2\" (UID: \"147ff712-6d65-4184-9125-9ae9391707dd\") " pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:21:13.133974 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:13.133789 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/147ff712-6d65-4184-9125-9ae9391707dd-istio-token\") pod \"e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2\" (UID: \"147ff712-6d65-4184-9125-9ae9391707dd\") " pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:21:13.133974 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:13.133875 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/147ff712-6d65-4184-9125-9ae9391707dd-istio-data\") pod \"e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2\" (UID: \"147ff712-6d65-4184-9125-9ae9391707dd\") " pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:21:13.133974 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:13.133937 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/147ff712-6d65-4184-9125-9ae9391707dd-istiod-ca-cert\") pod \"e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2\" (UID: \"147ff712-6d65-4184-9125-9ae9391707dd\") " pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:21:13.133974 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:13.133972 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w7mw\" (UniqueName: \"kubernetes.io/projected/147ff712-6d65-4184-9125-9ae9391707dd-kube-api-access-9w7mw\") pod \"e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2\" (UID: \"147ff712-6d65-4184-9125-9ae9391707dd\") " pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:21:13.169282 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:13.169242 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2"] Oct 30 22:21:13.234939 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:13.234850 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/147ff712-6d65-4184-9125-9ae9391707dd-istio-envoy\") pod \"e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2\" (UID: \"147ff712-6d65-4184-9125-9ae9391707dd\") " pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:21:13.234939 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:13.234924 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/147ff712-6d65-4184-9125-9ae9391707dd-workload-socket\") pod \"e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2\" (UID: \"147ff712-6d65-4184-9125-9ae9391707dd\") " pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:21:13.235163 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:13.234956 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/147ff712-6d65-4184-9125-9ae9391707dd-credential-socket\") pod \"e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2\" (UID: \"147ff712-6d65-4184-9125-9ae9391707dd\") " pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:21:13.235163 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:13.234984 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/147ff712-6d65-4184-9125-9ae9391707dd-istio-podinfo\") pod \"e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2\" (UID: \"147ff712-6d65-4184-9125-9ae9391707dd\") " pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:21:13.235163 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:13.235052 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/147ff712-6d65-4184-9125-9ae9391707dd-istio-token\") pod \"e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2\" (UID: \"147ff712-6d65-4184-9125-9ae9391707dd\") " pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:21:13.235163 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:13.235123 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/147ff712-6d65-4184-9125-9ae9391707dd-istio-data\") pod \"e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2\" (UID: \"147ff712-6d65-4184-9125-9ae9391707dd\") " pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:21:13.235440 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:13.235183 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/147ff712-6d65-4184-9125-9ae9391707dd-istiod-ca-cert\") pod \"e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2\" (UID: \"147ff712-6d65-4184-9125-9ae9391707dd\") " pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:21:13.235440 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:13.235238 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w7mw\" (UniqueName: \"kubernetes.io/projected/147ff712-6d65-4184-9125-9ae9391707dd-kube-api-access-9w7mw\") pod \"e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2\" (UID: \"147ff712-6d65-4184-9125-9ae9391707dd\") " pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:21:13.235440 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:13.235398 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/147ff712-6d65-4184-9125-9ae9391707dd-workload-certs\") pod \"e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2\" (UID: \"147ff712-6d65-4184-9125-9ae9391707dd\") " pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:21:13.235440 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:13.235412 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/147ff712-6d65-4184-9125-9ae9391707dd-workload-socket\") pod \"e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2\" (UID: \"147ff712-6d65-4184-9125-9ae9391707dd\") " pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:21:13.236149 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:13.236125 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/147ff712-6d65-4184-9125-9ae9391707dd-credential-socket\") pod \"e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2\" (UID: \"147ff712-6d65-4184-9125-9ae9391707dd\") " pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:21:13.236359 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:13.236158 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/147ff712-6d65-4184-9125-9ae9391707dd-istio-data\") pod \"e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2\" (UID: \"147ff712-6d65-4184-9125-9ae9391707dd\") " pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:21:13.236359 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:13.236160 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/147ff712-6d65-4184-9125-9ae9391707dd-workload-certs\") pod \"e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2\" (UID: \"147ff712-6d65-4184-9125-9ae9391707dd\") " pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:21:13.236562 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:13.236392 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/147ff712-6d65-4184-9125-9ae9391707dd-istiod-ca-cert\") pod \"e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2\" (UID: \"147ff712-6d65-4184-9125-9ae9391707dd\") " pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:21:13.237537 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:13.237514 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/147ff712-6d65-4184-9125-9ae9391707dd-istio-envoy\") pod \"e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2\" (UID: \"147ff712-6d65-4184-9125-9ae9391707dd\") " pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:21:13.237900 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:13.237881 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/147ff712-6d65-4184-9125-9ae9391707dd-istio-podinfo\") pod \"e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2\" (UID: \"147ff712-6d65-4184-9125-9ae9391707dd\") " pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:21:13.311505 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:13.299347 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w7mw\" (UniqueName: \"kubernetes.io/projected/147ff712-6d65-4184-9125-9ae9391707dd-kube-api-access-9w7mw\") pod \"e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2\" (UID: \"147ff712-6d65-4184-9125-9ae9391707dd\") " pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:21:13.319185 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:13.319131 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/147ff712-6d65-4184-9125-9ae9391707dd-istio-token\") pod \"e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2\" (UID: \"147ff712-6d65-4184-9125-9ae9391707dd\") " pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:21:13.584491 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:13.584402 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:21:13.710262 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:13.710228 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-f2qhs" Oct 30 22:21:14.031197 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:21:14.030356 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod147ff712_6d65_4184_9125_9ae9391707dd.slice/crio-520dcb6ccbcc682e6f2e8fb1d0dc7abdab470317354eccc2a707394ecd6a8693 WatchSource:0}: Error finding container 520dcb6ccbcc682e6f2e8fb1d0dc7abdab470317354eccc2a707394ecd6a8693: Status 404 returned error can't find the container with id 520dcb6ccbcc682e6f2e8fb1d0dc7abdab470317354eccc2a707394ecd6a8693 Oct 30 22:21:14.031197 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:14.030705 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2"] Oct 30 22:21:14.034092 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:14.034052 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:21:14.034230 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:14.034141 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:21:14.034230 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:14.034189 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:21:14.791956 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:14.791919 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" event={"ID":"147ff712-6d65-4184-9125-9ae9391707dd","Type":"ContainerStarted","Data":"6004971db453dac3f21f59c48449d733113448c0d32f6acb612ea3fe9c5e71df"} Oct 30 22:21:14.792144 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:14.791980 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" event={"ID":"147ff712-6d65-4184-9125-9ae9391707dd","Type":"ContainerStarted","Data":"520dcb6ccbcc682e6f2e8fb1d0dc7abdab470317354eccc2a707394ecd6a8693"} Oct 30 22:21:15.019538 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:15.019474 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" podStartSLOduration=3.019455799 podStartE2EDuration="3.019455799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:14.966536113 +0000 UTC m=+1835.237234200" watchObservedRunningTime="2026-10-30 22:21:15.019455799 +0000 UTC m=+1835.290153886" Oct 30 22:21:15.510621 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:15.510583 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-81f37a-895cb88f-wwbk6"] Oct 30 22:21:15.515126 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:15.515103 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81f37a-895cb88f-wwbk6" Oct 30 22:21:15.571588 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:15.571546 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-81f37a-serving-cert\"" Oct 30 22:21:15.575889 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:15.575860 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-81f37a-895cb88f-wwbk6"] Oct 30 22:21:15.584883 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:15.584853 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:21:15.592290 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:15.592261 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:21:15.678079 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:15.678048 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36861cad-68a4-4d18-b2b6-de583e5dab2b-maas-api-tls\") pod \"maas-api-e2e-api-a-81f37a-895cb88f-wwbk6\" (UID: \"36861cad-68a4-4d18-b2b6-de583e5dab2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81f37a-895cb88f-wwbk6" Oct 30 22:21:15.678079 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:15.678083 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36861cad-68a4-4d18-b2b6-de583e5dab2b-openshift-service-ca\") pod \"maas-api-e2e-api-a-81f37a-895cb88f-wwbk6\" (UID: \"36861cad-68a4-4d18-b2b6-de583e5dab2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81f37a-895cb88f-wwbk6" Oct 30 22:21:15.678305 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:15.678210 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7nts\" (UniqueName: \"kubernetes.io/projected/36861cad-68a4-4d18-b2b6-de583e5dab2b-kube-api-access-f7nts\") pod \"maas-api-e2e-api-a-81f37a-895cb88f-wwbk6\" (UID: \"36861cad-68a4-4d18-b2b6-de583e5dab2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81f37a-895cb88f-wwbk6" Oct 30 22:21:15.678358 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:15.678338 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36861cad-68a4-4d18-b2b6-de583e5dab2b-metrics-certs\") pod \"maas-api-e2e-api-a-81f37a-895cb88f-wwbk6\" (UID: \"36861cad-68a4-4d18-b2b6-de583e5dab2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81f37a-895cb88f-wwbk6" Oct 30 22:21:15.780139 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:15.780041 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36861cad-68a4-4d18-b2b6-de583e5dab2b-maas-api-tls\") pod \"maas-api-e2e-api-a-81f37a-895cb88f-wwbk6\" (UID: \"36861cad-68a4-4d18-b2b6-de583e5dab2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81f37a-895cb88f-wwbk6" Oct 30 22:21:15.780139 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:15.780080 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36861cad-68a4-4d18-b2b6-de583e5dab2b-openshift-service-ca\") pod \"maas-api-e2e-api-a-81f37a-895cb88f-wwbk6\" (UID: \"36861cad-68a4-4d18-b2b6-de583e5dab2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81f37a-895cb88f-wwbk6" Oct 30 22:21:15.780139 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:15.780142 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7nts\" (UniqueName: \"kubernetes.io/projected/36861cad-68a4-4d18-b2b6-de583e5dab2b-kube-api-access-f7nts\") pod \"maas-api-e2e-api-a-81f37a-895cb88f-wwbk6\" (UID: \"36861cad-68a4-4d18-b2b6-de583e5dab2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81f37a-895cb88f-wwbk6" Oct 30 22:21:15.780444 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:21:15.780208 5918 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-81f37a-serving-cert: secret "maas-api-e2e-api-a-81f37a-serving-cert" not found Oct 30 22:21:15.780444 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:15.780245 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36861cad-68a4-4d18-b2b6-de583e5dab2b-metrics-certs\") pod \"maas-api-e2e-api-a-81f37a-895cb88f-wwbk6\" (UID: \"36861cad-68a4-4d18-b2b6-de583e5dab2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81f37a-895cb88f-wwbk6" Oct 30 22:21:15.780444 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:21:15.780309 5918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36861cad-68a4-4d18-b2b6-de583e5dab2b-maas-api-tls podName:36861cad-68a4-4d18-b2b6-de583e5dab2b nodeName:}" failed. No retries permitted until 2026-10-30 22:21:16.280288521 +0000 UTC m=+1836.550986608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/36861cad-68a4-4d18-b2b6-de583e5dab2b-maas-api-tls") pod "maas-api-e2e-api-a-81f37a-895cb88f-wwbk6" (UID: "36861cad-68a4-4d18-b2b6-de583e5dab2b") : secret "maas-api-e2e-api-a-81f37a-serving-cert" not found Oct 30 22:21:15.780894 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:15.780868 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36861cad-68a4-4d18-b2b6-de583e5dab2b-openshift-service-ca\") pod \"maas-api-e2e-api-a-81f37a-895cb88f-wwbk6\" (UID: \"36861cad-68a4-4d18-b2b6-de583e5dab2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81f37a-895cb88f-wwbk6" Oct 30 22:21:15.786258 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:15.783798 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36861cad-68a4-4d18-b2b6-de583e5dab2b-metrics-certs\") pod \"maas-api-e2e-api-a-81f37a-895cb88f-wwbk6\" (UID: \"36861cad-68a4-4d18-b2b6-de583e5dab2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81f37a-895cb88f-wwbk6" Oct 30 22:21:15.811154 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:15.806603 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:21:15.811154 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:15.808526 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:21:15.843949 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:15.843914 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7nts\" (UniqueName: \"kubernetes.io/projected/36861cad-68a4-4d18-b2b6-de583e5dab2b-kube-api-access-f7nts\") pod \"maas-api-e2e-api-a-81f37a-895cb88f-wwbk6\" (UID: \"36861cad-68a4-4d18-b2b6-de583e5dab2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81f37a-895cb88f-wwbk6" Oct 30 22:21:16.289081 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:16.289044 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36861cad-68a4-4d18-b2b6-de583e5dab2b-maas-api-tls\") pod \"maas-api-e2e-api-a-81f37a-895cb88f-wwbk6\" (UID: \"36861cad-68a4-4d18-b2b6-de583e5dab2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81f37a-895cb88f-wwbk6" Oct 30 22:21:16.292186 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:16.292158 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36861cad-68a4-4d18-b2b6-de583e5dab2b-maas-api-tls\") pod \"maas-api-e2e-api-a-81f37a-895cb88f-wwbk6\" (UID: \"36861cad-68a4-4d18-b2b6-de583e5dab2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81f37a-895cb88f-wwbk6" Oct 30 22:21:16.430711 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:16.430675 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81f37a-895cb88f-wwbk6" Oct 30 22:21:16.822984 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:16.822937 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-81f37a-895cb88f-wwbk6"] Oct 30 22:21:16.829538 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:21:16.829506 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36861cad_68a4_4d18_b2b6_de583e5dab2b.slice/crio-d6952ca848ba1016231f57bbfdcd03acc028b041672f969a1c31c88e01732537 WatchSource:0}: Error finding container d6952ca848ba1016231f57bbfdcd03acc028b041672f969a1c31c88e01732537: Status 404 returned error can't find the container with id d6952ca848ba1016231f57bbfdcd03acc028b041672f969a1c31c88e01732537 Oct 30 22:21:17.821533 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:17.821492 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81f37a-895cb88f-wwbk6" event={"ID":"36861cad-68a4-4d18-b2b6-de583e5dab2b","Type":"ContainerStarted","Data":"78abbc7625dd5d27f50f93634900bc2afe45a6a8b26f97b619bf8670312c6ed5"} Oct 30 22:21:17.821713 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:17.821556 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81f37a-895cb88f-wwbk6" event={"ID":"36861cad-68a4-4d18-b2b6-de583e5dab2b","Type":"ContainerStarted","Data":"d6952ca848ba1016231f57bbfdcd03acc028b041672f969a1c31c88e01732537"} Oct 30 22:21:17.821767 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:17.821738 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81f37a-895cb88f-wwbk6" Oct 30 22:21:17.946651 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:17.946576 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81f37a-895cb88f-wwbk6" podStartSLOduration=2.148037705 podStartE2EDuration="2.946556635s" totalImagesPullingTime="798.51893ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:17.943540888 +0000 UTC m=+1838.214238972" watchObservedRunningTime="2026-10-30 22:21:17.946556635 +0000 UTC m=+1838.217254720" Oct 30 22:21:21.342189 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:21:21.342149 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:21:23.835374 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:23.835344 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81f37a-895cb88f-wwbk6" Oct 30 22:21:34.727196 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:34.727159 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6920-86dc93"] Oct 30 22:21:34.731506 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:34.731489 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6920-86dc93" Oct 30 22:21:34.785935 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:34.785906 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6920-86dc93"] Oct 30 22:21:34.832567 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:34.832533 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-9tw55\"" Oct 30 22:21:34.905038 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:34.905001 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58jtr\" (UniqueName: \"kubernetes.io/projected/75d39258-6c64-426c-bbb4-446629ec21a1-kube-api-access-58jtr\") pod \"test-curl-6920-86dc93\" (UID: \"75d39258-6c64-426c-bbb4-446629ec21a1\") " pod="opendatahub/test-curl-6920-86dc93" Oct 30 22:21:35.006456 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:35.006365 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58jtr\" (UniqueName: \"kubernetes.io/projected/75d39258-6c64-426c-bbb4-446629ec21a1-kube-api-access-58jtr\") pod \"test-curl-6920-86dc93\" (UID: \"75d39258-6c64-426c-bbb4-446629ec21a1\") " pod="opendatahub/test-curl-6920-86dc93" Oct 30 22:21:35.070837 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:35.070801 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58jtr\" (UniqueName: \"kubernetes.io/projected/75d39258-6c64-426c-bbb4-446629ec21a1-kube-api-access-58jtr\") pod \"test-curl-6920-86dc93\" (UID: \"75d39258-6c64-426c-bbb4-446629ec21a1\") " pod="opendatahub/test-curl-6920-86dc93" Oct 30 22:21:35.344116 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:35.344031 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6920-86dc93" Oct 30 22:21:35.732817 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:35.732786 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6920-86dc93"] Oct 30 22:21:35.733815 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:21:35.733780 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75d39258_6c64_426c_bbb4_446629ec21a1.slice/crio-925269685c63c66a3676008e8a3d8221731953daee2f2284014196e6770aab72 WatchSource:0}: Error finding container 925269685c63c66a3676008e8a3d8221731953daee2f2284014196e6770aab72: Status 404 returned error can't find the container with id 925269685c63c66a3676008e8a3d8221731953daee2f2284014196e6770aab72 Oct 30 22:21:35.952297 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:35.952261 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6920-86dc93" event={"ID":"75d39258-6c64-426c-bbb4-446629ec21a1","Type":"ContainerStarted","Data":"925269685c63c66a3676008e8a3d8221731953daee2f2284014196e6770aab72"} Oct 30 22:21:38.984315 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:38.984279 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6920-86dc93" event={"ID":"75d39258-6c64-426c-bbb4-446629ec21a1","Type":"ContainerStarted","Data":"93ddc1445fa7c9061b91e326641f2850db594dca0ad27a51b239e1170bb1f8f2"} Oct 30 22:21:39.147320 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:39.147186 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6920-86dc93" podStartSLOduration=2.088048895 podStartE2EDuration="5.147173693s" totalImagesPullingTime="3.059124798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:39.101635974 +0000 UTC m=+1859.372334039" watchObservedRunningTime="2026-10-30 22:21:39.147173693 +0000 UTC m=+1859.417871777" Oct 30 22:21:40.604306 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:40.604273 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6920-86dc93"] Oct 30 22:21:40.610186 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:40.610155 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6920-86dc93"] Oct 30 22:21:40.998357 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:40.998320 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6920-86dc93" podUID="75d39258-6c64-426c-bbb4-446629ec21a1" containerName="test-curl-6920-86dc93" containerID="cri-o://93ddc1445fa7c9061b91e326641f2850db594dca0ad27a51b239e1170bb1f8f2" gracePeriod=2 Oct 30 22:21:41.061518 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:21:41.061484 5918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6920-86dc93\" is forbidden: User \"system:node:ip-10-0-137-81.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-81.us-west-1.compute.internal' and this object" podUID="75d39258-6c64-426c-bbb4-446629ec21a1" pod="opendatahub/test-curl-6920-86dc93" Oct 30 22:21:41.519689 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:41.519659 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw"] Oct 30 22:21:41.520435 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:41.520414 5918 cpu_manager.go:414] "containerMap: removing container" podUID="75d39258-6c64-426c-bbb4-446629ec21a1" containerName="test-curl-6920-86dc93" Oct 30 22:21:41.520555 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:41.520435 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75d39258-6c64-426c-bbb4-446629ec21a1" containerName="test-curl-6920-86dc93" Oct 30 22:21:41.520656 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:41.520642 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="75d39258-6c64-426c-bbb4-446629ec21a1" containerName="test-curl-6920-86dc93" Oct 30 22:21:41.524505 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:41.524484 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw" Oct 30 22:21:41.576284 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:21:41.576248 5918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6920-86dc93\" is forbidden: User \"system:node:ip-10-0-137-81.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-81.us-west-1.compute.internal' and this object" podUID="75d39258-6c64-426c-bbb4-446629ec21a1" pod="opendatahub/test-curl-6920-86dc93" Oct 30 22:21:41.581127 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:41.581086 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-4511b0-serving-cert\"" Oct 30 22:21:41.584599 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:41.584574 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw"] Oct 30 22:21:41.603425 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:41.603395 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae6513bd-92b2-4a98-85ee-a02ab40fe705-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw\" (UID: \"ae6513bd-92b2-4a98-85ee-a02ab40fe705\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw" Oct 30 22:21:41.603620 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:41.603599 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae6513bd-92b2-4a98-85ee-a02ab40fe705-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw\" (UID: \"ae6513bd-92b2-4a98-85ee-a02ab40fe705\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw" Oct 30 22:21:41.603772 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:41.603756 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae6513bd-92b2-4a98-85ee-a02ab40fe705-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw\" (UID: \"ae6513bd-92b2-4a98-85ee-a02ab40fe705\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw" Oct 30 22:21:41.603920 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:41.603900 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5mbn\" (UniqueName: \"kubernetes.io/projected/ae6513bd-92b2-4a98-85ee-a02ab40fe705-kube-api-access-g5mbn\") pod \"maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw\" (UID: \"ae6513bd-92b2-4a98-85ee-a02ab40fe705\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw" Oct 30 22:21:41.705274 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:41.705237 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae6513bd-92b2-4a98-85ee-a02ab40fe705-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw\" (UID: \"ae6513bd-92b2-4a98-85ee-a02ab40fe705\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw" Oct 30 22:21:41.705763 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:41.705388 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae6513bd-92b2-4a98-85ee-a02ab40fe705-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw\" (UID: \"ae6513bd-92b2-4a98-85ee-a02ab40fe705\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw" Oct 30 22:21:41.705763 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:21:41.705504 5918 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4511b0-serving-cert: secret "maas-api-e2e-shared-b-w2-4511b0-serving-cert" not found Oct 30 22:21:41.705763 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:41.705552 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae6513bd-92b2-4a98-85ee-a02ab40fe705-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw\" (UID: \"ae6513bd-92b2-4a98-85ee-a02ab40fe705\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw" Oct 30 22:21:41.705763 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:21:41.705578 5918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae6513bd-92b2-4a98-85ee-a02ab40fe705-maas-api-tls podName:ae6513bd-92b2-4a98-85ee-a02ab40fe705 nodeName:}" failed. No retries permitted until 2026-10-30 22:21:42.205554024 +0000 UTC m=+1862.476252102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ae6513bd-92b2-4a98-85ee-a02ab40fe705-maas-api-tls") pod "maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw" (UID: "ae6513bd-92b2-4a98-85ee-a02ab40fe705") : secret "maas-api-e2e-shared-b-w2-4511b0-serving-cert" not found Oct 30 22:21:41.705763 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:41.705730 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5mbn\" (UniqueName: \"kubernetes.io/projected/ae6513bd-92b2-4a98-85ee-a02ab40fe705-kube-api-access-g5mbn\") pod \"maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw\" (UID: \"ae6513bd-92b2-4a98-85ee-a02ab40fe705\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw" Oct 30 22:21:41.706329 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:41.706305 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae6513bd-92b2-4a98-85ee-a02ab40fe705-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw\" (UID: \"ae6513bd-92b2-4a98-85ee-a02ab40fe705\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw" Oct 30 22:21:41.708336 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:41.708309 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae6513bd-92b2-4a98-85ee-a02ab40fe705-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw\" (UID: \"ae6513bd-92b2-4a98-85ee-a02ab40fe705\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw" Oct 30 22:21:41.770754 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:41.770677 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5mbn\" (UniqueName: \"kubernetes.io/projected/ae6513bd-92b2-4a98-85ee-a02ab40fe705-kube-api-access-g5mbn\") pod \"maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw\" (UID: \"ae6513bd-92b2-4a98-85ee-a02ab40fe705\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw" Oct 30 22:21:42.211928 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:42.211898 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae6513bd-92b2-4a98-85ee-a02ab40fe705-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw\" (UID: \"ae6513bd-92b2-4a98-85ee-a02ab40fe705\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw" Oct 30 22:21:42.214481 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:42.214456 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae6513bd-92b2-4a98-85ee-a02ab40fe705-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw\" (UID: \"ae6513bd-92b2-4a98-85ee-a02ab40fe705\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw" Oct 30 22:21:42.438994 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:42.438960 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw" Oct 30 22:21:42.822646 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:42.822616 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw"] Oct 30 22:21:42.823496 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:21:42.823469 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae6513bd_92b2_4a98_85ee_a02ab40fe705.slice/crio-e26c58431e904b07cf41bfe14df1a1698ef802200ca7941580f431afa32bbcb5 WatchSource:0}: Error finding container e26c58431e904b07cf41bfe14df1a1698ef802200ca7941580f431afa32bbcb5: Status 404 returned error can't find the container with id e26c58431e904b07cf41bfe14df1a1698ef802200ca7941580f431afa32bbcb5 Oct 30 22:21:43.020998 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:43.020969 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw" event={"ID":"ae6513bd-92b2-4a98-85ee-a02ab40fe705","Type":"ContainerStarted","Data":"e26c58431e904b07cf41bfe14df1a1698ef802200ca7941580f431afa32bbcb5"} Oct 30 22:21:43.267005 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:43.266982 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6920-86dc93" Oct 30 22:21:43.325960 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:43.325935 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75d39258-6c64-426c-bbb4-446629ec21a1-kube-api-access-58jtr\" (UniqueName: \"kubernetes.io/projected/75d39258-6c64-426c-bbb4-446629ec21a1-kube-api-access-58jtr\") pod \"75d39258-6c64-426c-bbb4-446629ec21a1\" (UID: \"75d39258-6c64-426c-bbb4-446629ec21a1\") " Oct 30 22:21:43.327986 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:43.327964 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75d39258-6c64-426c-bbb4-446629ec21a1-kube-api-access-58jtr" pod "75d39258-6c64-426c-bbb4-446629ec21a1" (UID: "75d39258-6c64-426c-bbb4-446629ec21a1"). InnerVolumeSpecName "kube-api-access-58jtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:43.428081 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:43.428050 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58jtr\" (UniqueName: \"kubernetes.io/projected/75d39258-6c64-426c-bbb4-446629ec21a1-kube-api-access-58jtr\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:43.854257 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:43.854168 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75d39258-6c64-426c-bbb4-446629ec21a1" path="/var/lib/kubelet/pods/75d39258-6c64-426c-bbb4-446629ec21a1/volumes" Oct 30 22:21:44.031652 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:44.031608 5918 generic.go:356] "Generic (PLEG): container finished" podID="75d39258-6c64-426c-bbb4-446629ec21a1" containerID="93ddc1445fa7c9061b91e326641f2850db594dca0ad27a51b239e1170bb1f8f2" exitCode=137 Oct 30 22:21:44.031652 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:44.031655 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6920-86dc93" Oct 30 22:21:44.031906 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:44.031708 5918 scope.go:122] "RemoveContainer" containerID="93ddc1445fa7c9061b91e326641f2850db594dca0ad27a51b239e1170bb1f8f2" Oct 30 22:21:44.033619 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:44.033478 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw" event={"ID":"ae6513bd-92b2-4a98-85ee-a02ab40fe705","Type":"ContainerStarted","Data":"9c5baf4c8a21f142df081ceb272f4cdbe19333e85855f5896df0c24ff20f8468"} Oct 30 22:21:44.033703 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:44.033618 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw" Oct 30 22:21:44.042706 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:44.042689 5918 scope.go:122] "RemoveContainer" containerID="93ddc1445fa7c9061b91e326641f2850db594dca0ad27a51b239e1170bb1f8f2" Oct 30 22:21:44.043003 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:21:44.042984 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93ddc1445fa7c9061b91e326641f2850db594dca0ad27a51b239e1170bb1f8f2\": container with ID starting with 93ddc1445fa7c9061b91e326641f2850db594dca0ad27a51b239e1170bb1f8f2 not found: ID does not exist" containerID="93ddc1445fa7c9061b91e326641f2850db594dca0ad27a51b239e1170bb1f8f2" Oct 30 22:21:44.043070 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:44.043011 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93ddc1445fa7c9061b91e326641f2850db594dca0ad27a51b239e1170bb1f8f2"} err="failed to get container status \"93ddc1445fa7c9061b91e326641f2850db594dca0ad27a51b239e1170bb1f8f2\": rpc error: code = NotFound desc = could not find container \"93ddc1445fa7c9061b91e326641f2850db594dca0ad27a51b239e1170bb1f8f2\": container with ID starting with 93ddc1445fa7c9061b91e326641f2850db594dca0ad27a51b239e1170bb1f8f2 not found: ID does not exist" Oct 30 22:21:44.157977 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:44.157861 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw" podStartSLOduration=2.369554942 podStartE2EDuration="3.157842208s" totalImagesPullingTime="788.287266ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:44.154704461 +0000 UTC m=+1864.425402554" watchObservedRunningTime="2026-10-30 22:21:44.157842208 +0000 UTC m=+1864.428540292" Oct 30 22:21:50.047908 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:50.047875 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw" Oct 30 22:21:51.275881 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:21:51.275843 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:21:53.282054 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:53.282020 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv"] Oct 30 22:21:54.830241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:54.829492 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc"] Oct 30 22:21:54.830241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:54.829713 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" podUID="7df7a416-e18e-4713-89b1-5b3f21e9251d" containerName="istio-proxy" containerID="cri-o://d351f8766665b4e44b2ef7bb2ea09fa5e2d582a074df19124121eef746a39bda" gracePeriod=30 Oct 30 22:21:57.700241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:57.696359 5918 patch_prober.go:29] interesting pod/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:21:57.700241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:57.696437 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" podUID="7df7a416-e18e-4713-89b1-5b3f21e9251d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:21:58.283416 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:21:58.283354 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv" podUID="471d388d-0ab5-4bec-ac57-93d6f4df1bd6" containerName="payload-pre-processing" containerID="cri-o://32b232dd22f404562c2aa606f0fb4967162196dccbaa8dfcce5d039ca6ad2dd1" gracePeriod=25 Oct 30 22:22:00.089825 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:00.089790 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" Oct 30 22:22:00.171283 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:00.171150 5918 generic.go:356] "Generic (PLEG): container finished" podID="7df7a416-e18e-4713-89b1-5b3f21e9251d" containerID="d351f8766665b4e44b2ef7bb2ea09fa5e2d582a074df19124121eef746a39bda" exitCode=0 Oct 30 22:22:00.171283 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:00.171203 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" Oct 30 22:22:00.171283 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:00.171264 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" event={"ID":"7df7a416-e18e-4713-89b1-5b3f21e9251d","Type":"ContainerDied","Data":"d351f8766665b4e44b2ef7bb2ea09fa5e2d582a074df19124121eef746a39bda"} Oct 30 22:22:00.171556 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:00.171333 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc" event={"ID":"7df7a416-e18e-4713-89b1-5b3f21e9251d","Type":"ContainerDied","Data":"bb25eb334ee24e8d05371dda12ed51b11748bb5965f4b63b26bd4fec794af7e1"} Oct 30 22:22:00.171556 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:00.171367 5918 scope.go:122] "RemoveContainer" containerID="d351f8766665b4e44b2ef7bb2ea09fa5e2d582a074df19124121eef746a39bda" Oct 30 22:22:00.181702 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:00.181686 5918 scope.go:122] "RemoveContainer" containerID="d351f8766665b4e44b2ef7bb2ea09fa5e2d582a074df19124121eef746a39bda" Oct 30 22:22:00.182006 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:22:00.181983 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d351f8766665b4e44b2ef7bb2ea09fa5e2d582a074df19124121eef746a39bda\": container with ID starting with d351f8766665b4e44b2ef7bb2ea09fa5e2d582a074df19124121eef746a39bda not found: ID does not exist" containerID="d351f8766665b4e44b2ef7bb2ea09fa5e2d582a074df19124121eef746a39bda" Oct 30 22:22:00.182076 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:00.182022 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d351f8766665b4e44b2ef7bb2ea09fa5e2d582a074df19124121eef746a39bda"} err="failed to get container status \"d351f8766665b4e44b2ef7bb2ea09fa5e2d582a074df19124121eef746a39bda\": rpc error: code = NotFound desc = could not find container \"d351f8766665b4e44b2ef7bb2ea09fa5e2d582a074df19124121eef746a39bda\": container with ID starting with d351f8766665b4e44b2ef7bb2ea09fa5e2d582a074df19124121eef746a39bda not found: ID does not exist" Oct 30 22:22:00.227468 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:00.227441 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7df7a416-e18e-4713-89b1-5b3f21e9251d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7df7a416-e18e-4713-89b1-5b3f21e9251d-istio-podinfo\") pod \"7df7a416-e18e-4713-89b1-5b3f21e9251d\" (UID: \"7df7a416-e18e-4713-89b1-5b3f21e9251d\") " Oct 30 22:22:00.227631 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:00.227502 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7df7a416-e18e-4713-89b1-5b3f21e9251d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7df7a416-e18e-4713-89b1-5b3f21e9251d-workload-certs\") pod \"7df7a416-e18e-4713-89b1-5b3f21e9251d\" (UID: \"7df7a416-e18e-4713-89b1-5b3f21e9251d\") " Oct 30 22:22:00.227631 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:00.227575 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7df7a416-e18e-4713-89b1-5b3f21e9251d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df7a416-e18e-4713-89b1-5b3f21e9251d-credential-socket\") pod \"7df7a416-e18e-4713-89b1-5b3f21e9251d\" (UID: \"7df7a416-e18e-4713-89b1-5b3f21e9251d\") " Oct 30 22:22:00.227631 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:00.227630 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7df7a416-e18e-4713-89b1-5b3f21e9251d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df7a416-e18e-4713-89b1-5b3f21e9251d-workload-socket\") pod \"7df7a416-e18e-4713-89b1-5b3f21e9251d\" (UID: \"7df7a416-e18e-4713-89b1-5b3f21e9251d\") " Oct 30 22:22:00.227804 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:00.227742 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7df7a416-e18e-4713-89b1-5b3f21e9251d-kube-api-access-vbm5p\" (UniqueName: \"kubernetes.io/projected/7df7a416-e18e-4713-89b1-5b3f21e9251d-kube-api-access-vbm5p\") pod \"7df7a416-e18e-4713-89b1-5b3f21e9251d\" (UID: \"7df7a416-e18e-4713-89b1-5b3f21e9251d\") " Oct 30 22:22:00.227867 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:00.227830 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7df7a416-e18e-4713-89b1-5b3f21e9251d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7df7a416-e18e-4713-89b1-5b3f21e9251d-istiod-ca-cert\") pod \"7df7a416-e18e-4713-89b1-5b3f21e9251d\" (UID: \"7df7a416-e18e-4713-89b1-5b3f21e9251d\") " Oct 30 22:22:00.227867 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:00.227858 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7df7a416-e18e-4713-89b1-5b3f21e9251d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7df7a416-e18e-4713-89b1-5b3f21e9251d-istio-envoy\") pod \"7df7a416-e18e-4713-89b1-5b3f21e9251d\" (UID: \"7df7a416-e18e-4713-89b1-5b3f21e9251d\") " Oct 30 22:22:00.227972 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:00.227886 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7df7a416-e18e-4713-89b1-5b3f21e9251d-istio-token\" (UniqueName: \"kubernetes.io/projected/7df7a416-e18e-4713-89b1-5b3f21e9251d-istio-token\") pod \"7df7a416-e18e-4713-89b1-5b3f21e9251d\" (UID: \"7df7a416-e18e-4713-89b1-5b3f21e9251d\") " Oct 30 22:22:00.227972 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:00.227886 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7df7a416-e18e-4713-89b1-5b3f21e9251d-workload-certs" pod "7df7a416-e18e-4713-89b1-5b3f21e9251d" (UID: "7df7a416-e18e-4713-89b1-5b3f21e9251d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:00.227972 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:00.227923 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7df7a416-e18e-4713-89b1-5b3f21e9251d-credential-socket" pod "7df7a416-e18e-4713-89b1-5b3f21e9251d" (UID: "7df7a416-e18e-4713-89b1-5b3f21e9251d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:00.227972 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:00.227966 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7df7a416-e18e-4713-89b1-5b3f21e9251d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7df7a416-e18e-4713-89b1-5b3f21e9251d-istio-data\") pod \"7df7a416-e18e-4713-89b1-5b3f21e9251d\" (UID: \"7df7a416-e18e-4713-89b1-5b3f21e9251d\") " Oct 30 22:22:00.228185 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:00.227968 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7df7a416-e18e-4713-89b1-5b3f21e9251d-workload-socket" pod "7df7a416-e18e-4713-89b1-5b3f21e9251d" (UID: "7df7a416-e18e-4713-89b1-5b3f21e9251d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:00.228306 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:00.228269 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7df7a416-e18e-4713-89b1-5b3f21e9251d-istiod-ca-cert" pod "7df7a416-e18e-4713-89b1-5b3f21e9251d" (UID: "7df7a416-e18e-4713-89b1-5b3f21e9251d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:22:00.229072 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:00.228475 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7df7a416-e18e-4713-89b1-5b3f21e9251d-istio-data" pod "7df7a416-e18e-4713-89b1-5b3f21e9251d" (UID: "7df7a416-e18e-4713-89b1-5b3f21e9251d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:00.229072 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:00.228982 5918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7df7a416-e18e-4713-89b1-5b3f21e9251d-istiod-ca-cert\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:00.229072 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:00.229004 5918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7df7a416-e18e-4713-89b1-5b3f21e9251d-istio-data\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:00.229072 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:00.229019 5918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7df7a416-e18e-4713-89b1-5b3f21e9251d-workload-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:00.229072 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:00.229033 5918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df7a416-e18e-4713-89b1-5b3f21e9251d-credential-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:00.229072 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:00.229047 5918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df7a416-e18e-4713-89b1-5b3f21e9251d-workload-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:00.230092 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:00.230063 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7df7a416-e18e-4713-89b1-5b3f21e9251d-istio-envoy" pod "7df7a416-e18e-4713-89b1-5b3f21e9251d" (UID: "7df7a416-e18e-4713-89b1-5b3f21e9251d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:00.230301 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:00.230280 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7df7a416-e18e-4713-89b1-5b3f21e9251d-istio-token" pod "7df7a416-e18e-4713-89b1-5b3f21e9251d" (UID: "7df7a416-e18e-4713-89b1-5b3f21e9251d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:22:00.230397 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:00.230316 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7df7a416-e18e-4713-89b1-5b3f21e9251d-istio-podinfo" pod "7df7a416-e18e-4713-89b1-5b3f21e9251d" (UID: "7df7a416-e18e-4713-89b1-5b3f21e9251d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:22:00.230441 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:00.230430 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7df7a416-e18e-4713-89b1-5b3f21e9251d-kube-api-access-vbm5p" pod "7df7a416-e18e-4713-89b1-5b3f21e9251d" (UID: "7df7a416-e18e-4713-89b1-5b3f21e9251d"). InnerVolumeSpecName "kube-api-access-vbm5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:22:00.330281 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:00.330245 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbm5p\" (UniqueName: \"kubernetes.io/projected/7df7a416-e18e-4713-89b1-5b3f21e9251d-kube-api-access-vbm5p\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:00.330281 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:00.330274 5918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7df7a416-e18e-4713-89b1-5b3f21e9251d-istio-envoy\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:00.330281 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:00.330283 5918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7df7a416-e18e-4713-89b1-5b3f21e9251d-istio-token\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:00.330281 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:00.330292 5918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7df7a416-e18e-4713-89b1-5b3f21e9251d-istio-podinfo\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:00.610788 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:00.610755 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc"] Oct 30 22:22:00.615620 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:00.615591 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-7srmc"] Oct 30 22:22:01.853912 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:01.853873 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7df7a416-e18e-4713-89b1-5b3f21e9251d" path="/var/lib/kubelet/pods/7df7a416-e18e-4713-89b1-5b3f21e9251d/volumes" Oct 30 22:22:02.267889 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:02.267852 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv" podUID="471d388d-0ab5-4bec-ac57-93d6f4df1bd6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:22:04.205754 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:04.205722 5918 generic.go:356] "Generic (PLEG): container finished" podID="c9afe2de-950a-495b-b234-4bab76e0dadc" containerID="b2dd8535cb8c97525862cf1e145697db1fa661b8f3cc41fb51a2e927ceacc805" exitCode=0 Oct 30 22:22:04.206196 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:04.205797 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" event={"ID":"c9afe2de-950a-495b-b234-4bab76e0dadc","Type":"ContainerDied","Data":"b2dd8535cb8c97525862cf1e145697db1fa661b8f3cc41fb51a2e927ceacc805"} Oct 30 22:22:04.206263 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:04.206202 5918 scope.go:122] "RemoveContainer" containerID="b2dd8535cb8c97525862cf1e145697db1fa661b8f3cc41fb51a2e927ceacc805" Oct 30 22:22:04.207956 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:04.207927 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:22:04.208023 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:04.207997 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:22:04.208059 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:04.208028 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:22:05.225776 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:05.225737 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" event={"ID":"c9afe2de-950a-495b-b234-4bab76e0dadc","Type":"ContainerStarted","Data":"d7d6602f2e7119147d31da9ee8a6157bc52439fa2e568c387071659a14ae00d1"} Oct 30 22:22:05.463551 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:05.463520 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:22:05.464490 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:05.464464 5918 patch_prober.go:29] interesting pod/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 22:22:05.464606 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:05.464512 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" podUID="c9afe2de-950a-495b-b234-4bab76e0dadc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 22:22:06.463980 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:06.463943 5918 patch_prober.go:29] interesting pod/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 22:22:06.464431 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:06.463998 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" podUID="c9afe2de-950a-495b-b234-4bab76e0dadc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 22:22:07.463869 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:07.463830 5918 patch_prober.go:29] interesting pod/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 22:22:07.464049 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:07.463886 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" podUID="c9afe2de-950a-495b-b234-4bab76e0dadc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 22:22:08.463710 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:08.463666 5918 patch_prober.go:29] interesting pod/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 22:22:08.463930 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:08.463729 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" podUID="c9afe2de-950a-495b-b234-4bab76e0dadc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 22:22:09.006285 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:09.005305 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:22:09.464491 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:09.464456 5918 patch_prober.go:29] interesting pod/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 22:22:09.464679 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:09.464510 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" podUID="c9afe2de-950a-495b-b234-4bab76e0dadc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 22:22:10.464358 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:10.464321 5918 patch_prober.go:29] interesting pod/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 22:22:10.464784 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:10.464373 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" podUID="c9afe2de-950a-495b-b234-4bab76e0dadc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 22:22:11.469354 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:11.469324 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:22:11.470480 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:11.470458 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:22:12.268230 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:12.268189 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv" podUID="471d388d-0ab5-4bec-ac57-93d6f4df1bd6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:22:18.325526 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:18.325490 5918 generic.go:356] "Generic (PLEG): container finished" podID="471d388d-0ab5-4bec-ac57-93d6f4df1bd6" containerID="32b232dd22f404562c2aa606f0fb4967162196dccbaa8dfcce5d039ca6ad2dd1" exitCode=0 Oct 30 22:22:18.325911 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:18.325559 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv" event={"ID":"471d388d-0ab5-4bec-ac57-93d6f4df1bd6","Type":"ContainerDied","Data":"32b232dd22f404562c2aa606f0fb4967162196dccbaa8dfcce5d039ca6ad2dd1"} Oct 30 22:22:18.492077 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:18.492054 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv" Oct 30 22:22:18.647665 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:18.647583 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/471d388d-0ab5-4bec-ac57-93d6f4df1bd6-kube-api-access-xrf97\" (UniqueName: \"kubernetes.io/projected/471d388d-0ab5-4bec-ac57-93d6f4df1bd6-kube-api-access-xrf97\") pod \"471d388d-0ab5-4bec-ac57-93d6f4df1bd6\" (UID: \"471d388d-0ab5-4bec-ac57-93d6f4df1bd6\") " Oct 30 22:22:18.647813 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:18.647718 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/471d388d-0ab5-4bec-ac57-93d6f4df1bd6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/471d388d-0ab5-4bec-ac57-93d6f4df1bd6-plugins-config-volume\") pod \"471d388d-0ab5-4bec-ac57-93d6f4df1bd6\" (UID: \"471d388d-0ab5-4bec-ac57-93d6f4df1bd6\") " Oct 30 22:22:18.648192 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:18.648162 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/471d388d-0ab5-4bec-ac57-93d6f4df1bd6-plugins-config-volume" pod "471d388d-0ab5-4bec-ac57-93d6f4df1bd6" (UID: "471d388d-0ab5-4bec-ac57-93d6f4df1bd6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:22:18.648658 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:18.648633 5918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/471d388d-0ab5-4bec-ac57-93d6f4df1bd6-plugins-config-volume\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:18.650353 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:18.650321 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/471d388d-0ab5-4bec-ac57-93d6f4df1bd6-kube-api-access-xrf97" pod "471d388d-0ab5-4bec-ac57-93d6f4df1bd6" (UID: "471d388d-0ab5-4bec-ac57-93d6f4df1bd6"). InnerVolumeSpecName "kube-api-access-xrf97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:22:18.749921 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:18.749889 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrf97\" (UniqueName: \"kubernetes.io/projected/471d388d-0ab5-4bec-ac57-93d6f4df1bd6-kube-api-access-xrf97\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:19.333302 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:19.333256 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv" Oct 30 22:22:19.333302 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:19.333286 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv" event={"ID":"471d388d-0ab5-4bec-ac57-93d6f4df1bd6","Type":"ContainerDied","Data":"53b9f2ee05a265d28238c0f605b6fbf39257df46e1543e93ceb74d142a54f220"} Oct 30 22:22:19.333860 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:19.333361 5918 scope.go:122] "RemoveContainer" containerID="32b232dd22f404562c2aa606f0fb4967162196dccbaa8dfcce5d039ca6ad2dd1" Oct 30 22:22:19.509851 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:19.509817 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv"] Oct 30 22:22:19.559476 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:19.559400 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-h8cdv"] Oct 30 22:22:19.853812 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:19.853776 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="471d388d-0ab5-4bec-ac57-93d6f4df1bd6" path="/var/lib/kubelet/pods/471d388d-0ab5-4bec-ac57-93d6f4df1bd6/volumes" Oct 30 22:22:21.276574 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:22:21.276540 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:22:39.650821 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:39.650784 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65"] Oct 30 22:22:39.651320 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:39.650997 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" podUID="34335b46-c602-47cb-92f4-62df851dd01b" containerName="istio-proxy" containerID="cri-o://ae38f8be1ccacf3468e262e8f536b94d15fa44931a083dc3b81a38dd79e58406" gracePeriod=30 Oct 30 22:22:41.641971 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:41.641932 5918 scope.go:122] "RemoveContainer" containerID="285b949241428a7d5792f0ad39c044b35b5776a0ef693c8df4c05fe24db47cee" Oct 30 22:22:41.748713 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:41.748667 5918 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:22:41.748881 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:41.748735 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" podUID="34335b46-c602-47cb-92f4-62df851dd01b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:22:42.741921 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:42.741890 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk"] Oct 30 22:22:42.742363 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:42.742093 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" podUID="44a3edcb-2c75-49eb-a134-a5d98f105534" containerName="istio-proxy" containerID="cri-o://5637de566e41b2ffe03f1e87e514f794fb8873b51e321d47f56d1de6efe58635" gracePeriod=30 Oct 30 22:22:44.952904 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:44.952878 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" Oct 30 22:22:45.057827 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:45.057793 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34335b46-c602-47cb-92f4-62df851dd01b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34335b46-c602-47cb-92f4-62df851dd01b-istiod-ca-cert\") pod \"34335b46-c602-47cb-92f4-62df851dd01b\" (UID: \"34335b46-c602-47cb-92f4-62df851dd01b\") " Oct 30 22:22:45.058029 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:45.057912 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34335b46-c602-47cb-92f4-62df851dd01b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34335b46-c602-47cb-92f4-62df851dd01b-istio-data\") pod \"34335b46-c602-47cb-92f4-62df851dd01b\" (UID: \"34335b46-c602-47cb-92f4-62df851dd01b\") " Oct 30 22:22:45.058029 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:45.057952 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34335b46-c602-47cb-92f4-62df851dd01b-kube-api-access-6n62t\" (UniqueName: \"kubernetes.io/projected/34335b46-c602-47cb-92f4-62df851dd01b-kube-api-access-6n62t\") pod \"34335b46-c602-47cb-92f4-62df851dd01b\" (UID: \"34335b46-c602-47cb-92f4-62df851dd01b\") " Oct 30 22:22:45.058029 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:45.057970 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34335b46-c602-47cb-92f4-62df851dd01b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34335b46-c602-47cb-92f4-62df851dd01b-workload-certs\") pod \"34335b46-c602-47cb-92f4-62df851dd01b\" (UID: \"34335b46-c602-47cb-92f4-62df851dd01b\") " Oct 30 22:22:45.058202 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:45.058059 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34335b46-c602-47cb-92f4-62df851dd01b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34335b46-c602-47cb-92f4-62df851dd01b-istio-podinfo\") pod \"34335b46-c602-47cb-92f4-62df851dd01b\" (UID: \"34335b46-c602-47cb-92f4-62df851dd01b\") " Oct 30 22:22:45.058202 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:45.058109 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34335b46-c602-47cb-92f4-62df851dd01b-istio-token\" (UniqueName: \"kubernetes.io/projected/34335b46-c602-47cb-92f4-62df851dd01b-istio-token\") pod \"34335b46-c602-47cb-92f4-62df851dd01b\" (UID: \"34335b46-c602-47cb-92f4-62df851dd01b\") " Oct 30 22:22:45.058202 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:45.058172 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34335b46-c602-47cb-92f4-62df851dd01b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34335b46-c602-47cb-92f4-62df851dd01b-istio-envoy\") pod \"34335b46-c602-47cb-92f4-62df851dd01b\" (UID: \"34335b46-c602-47cb-92f4-62df851dd01b\") " Oct 30 22:22:45.058634 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:45.058590 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34335b46-c602-47cb-92f4-62df851dd01b-istio-data" pod "34335b46-c602-47cb-92f4-62df851dd01b" (UID: "34335b46-c602-47cb-92f4-62df851dd01b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:45.058750 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:45.058731 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34335b46-c602-47cb-92f4-62df851dd01b-istiod-ca-cert" pod "34335b46-c602-47cb-92f4-62df851dd01b" (UID: "34335b46-c602-47cb-92f4-62df851dd01b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:22:45.059191 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:45.058826 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34335b46-c602-47cb-92f4-62df851dd01b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34335b46-c602-47cb-92f4-62df851dd01b-credential-socket\") pod \"34335b46-c602-47cb-92f4-62df851dd01b\" (UID: \"34335b46-c602-47cb-92f4-62df851dd01b\") " Oct 30 22:22:45.059191 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:45.059119 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34335b46-c602-47cb-92f4-62df851dd01b-credential-socket" pod "34335b46-c602-47cb-92f4-62df851dd01b" (UID: "34335b46-c602-47cb-92f4-62df851dd01b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:45.061266 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:45.059404 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34335b46-c602-47cb-92f4-62df851dd01b-workload-certs" pod "34335b46-c602-47cb-92f4-62df851dd01b" (UID: "34335b46-c602-47cb-92f4-62df851dd01b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:45.061266 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:45.059247 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34335b46-c602-47cb-92f4-62df851dd01b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34335b46-c602-47cb-92f4-62df851dd01b-workload-socket\") pod \"34335b46-c602-47cb-92f4-62df851dd01b\" (UID: \"34335b46-c602-47cb-92f4-62df851dd01b\") " Oct 30 22:22:45.061266 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:45.059712 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34335b46-c602-47cb-92f4-62df851dd01b-workload-socket" pod "34335b46-c602-47cb-92f4-62df851dd01b" (UID: "34335b46-c602-47cb-92f4-62df851dd01b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:45.064806 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:45.064777 5918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34335b46-c602-47cb-92f4-62df851dd01b-credential-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:45.064911 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:45.064821 5918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34335b46-c602-47cb-92f4-62df851dd01b-workload-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:45.064911 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:45.064835 5918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34335b46-c602-47cb-92f4-62df851dd01b-istiod-ca-cert\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:45.064911 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:45.064849 5918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34335b46-c602-47cb-92f4-62df851dd01b-istio-data\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:45.064911 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:45.064862 5918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34335b46-c602-47cb-92f4-62df851dd01b-workload-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:45.072934 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:45.072903 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34335b46-c602-47cb-92f4-62df851dd01b-istio-podinfo" pod "34335b46-c602-47cb-92f4-62df851dd01b" (UID: "34335b46-c602-47cb-92f4-62df851dd01b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:22:45.074336 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:45.074300 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34335b46-c602-47cb-92f4-62df851dd01b-istio-envoy" pod "34335b46-c602-47cb-92f4-62df851dd01b" (UID: "34335b46-c602-47cb-92f4-62df851dd01b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:45.074631 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:45.074603 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34335b46-c602-47cb-92f4-62df851dd01b-istio-token" pod "34335b46-c602-47cb-92f4-62df851dd01b" (UID: "34335b46-c602-47cb-92f4-62df851dd01b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:22:45.075114 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:45.075092 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34335b46-c602-47cb-92f4-62df851dd01b-kube-api-access-6n62t" pod "34335b46-c602-47cb-92f4-62df851dd01b" (UID: "34335b46-c602-47cb-92f4-62df851dd01b"). InnerVolumeSpecName "kube-api-access-6n62t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:22:45.166238 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:45.166128 5918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34335b46-c602-47cb-92f4-62df851dd01b-istio-envoy\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:45.166238 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:45.166168 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n62t\" (UniqueName: \"kubernetes.io/projected/34335b46-c602-47cb-92f4-62df851dd01b-kube-api-access-6n62t\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:45.166238 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:45.166181 5918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34335b46-c602-47cb-92f4-62df851dd01b-istio-podinfo\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:45.166238 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:45.166189 5918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34335b46-c602-47cb-92f4-62df851dd01b-istio-token\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:45.538777 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:45.538734 5918 generic.go:356] "Generic (PLEG): container finished" podID="34335b46-c602-47cb-92f4-62df851dd01b" containerID="ae38f8be1ccacf3468e262e8f536b94d15fa44931a083dc3b81a38dd79e58406" exitCode=0 Oct 30 22:22:45.538981 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:45.538799 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" Oct 30 22:22:45.538981 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:45.538818 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" event={"ID":"34335b46-c602-47cb-92f4-62df851dd01b","Type":"ContainerDied","Data":"ae38f8be1ccacf3468e262e8f536b94d15fa44931a083dc3b81a38dd79e58406"} Oct 30 22:22:45.538981 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:45.538911 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65" event={"ID":"34335b46-c602-47cb-92f4-62df851dd01b","Type":"ContainerDied","Data":"b80df22b7936f81d5f6e9064e9c51c004511054c66c7ed0e2c1bae0ccd717df9"} Oct 30 22:22:45.538981 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:45.538942 5918 scope.go:122] "RemoveContainer" containerID="ae38f8be1ccacf3468e262e8f536b94d15fa44931a083dc3b81a38dd79e58406" Oct 30 22:22:45.550027 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:45.550008 5918 scope.go:122] "RemoveContainer" containerID="ae38f8be1ccacf3468e262e8f536b94d15fa44931a083dc3b81a38dd79e58406" Oct 30 22:22:45.550391 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:22:45.550369 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae38f8be1ccacf3468e262e8f536b94d15fa44931a083dc3b81a38dd79e58406\": container with ID starting with ae38f8be1ccacf3468e262e8f536b94d15fa44931a083dc3b81a38dd79e58406 not found: ID does not exist" containerID="ae38f8be1ccacf3468e262e8f536b94d15fa44931a083dc3b81a38dd79e58406" Oct 30 22:22:45.550475 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:45.550401 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae38f8be1ccacf3468e262e8f536b94d15fa44931a083dc3b81a38dd79e58406"} err="failed to get container status \"ae38f8be1ccacf3468e262e8f536b94d15fa44931a083dc3b81a38dd79e58406\": rpc error: code = NotFound desc = could not find container \"ae38f8be1ccacf3468e262e8f536b94d15fa44931a083dc3b81a38dd79e58406\": container with ID starting with ae38f8be1ccacf3468e262e8f536b94d15fa44931a083dc3b81a38dd79e58406 not found: ID does not exist" Oct 30 22:22:45.712862 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:45.712826 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65"] Oct 30 22:22:45.763798 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:45.763765 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-ssb65"] Oct 30 22:22:45.852604 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:45.852529 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34335b46-c602-47cb-92f4-62df851dd01b" path="/var/lib/kubelet/pods/34335b46-c602-47cb-92f4-62df851dd01b/volumes" Oct 30 22:22:47.555455 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:47.555368 5918 generic.go:356] "Generic (PLEG): container finished" podID="0143cf33-1443-4232-8b8e-2f7e448b78c8" containerID="ce65856e6076097a65eae2b4ba21cbd54253e07cd10d5936e91128498208645f" exitCode=0 Oct 30 22:22:47.555455 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:47.555443 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" event={"ID":"0143cf33-1443-4232-8b8e-2f7e448b78c8","Type":"ContainerDied","Data":"ce65856e6076097a65eae2b4ba21cbd54253e07cd10d5936e91128498208645f"} Oct 30 22:22:47.555890 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:47.555864 5918 scope.go:122] "RemoveContainer" containerID="ce65856e6076097a65eae2b4ba21cbd54253e07cd10d5936e91128498208645f" Oct 30 22:22:47.557694 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:47.557662 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:22:47.557856 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:47.557732 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:22:47.557856 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:47.557767 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:22:48.061703 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.061678 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" Oct 30 22:22:48.213575 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.207785 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a3edcb-2c75-49eb-a134-a5d98f105534-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a3edcb-2c75-49eb-a134-a5d98f105534-istio-data\") pod \"44a3edcb-2c75-49eb-a134-a5d98f105534\" (UID: \"44a3edcb-2c75-49eb-a134-a5d98f105534\") " Oct 30 22:22:48.213575 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.207856 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44a3edcb-2c75-49eb-a134-a5d98f105534-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a3edcb-2c75-49eb-a134-a5d98f105534-istiod-ca-cert\") pod \"44a3edcb-2c75-49eb-a134-a5d98f105534\" (UID: \"44a3edcb-2c75-49eb-a134-a5d98f105534\") " Oct 30 22:22:48.213575 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.207910 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a3edcb-2c75-49eb-a134-a5d98f105534-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a3edcb-2c75-49eb-a134-a5d98f105534-workload-certs\") pod \"44a3edcb-2c75-49eb-a134-a5d98f105534\" (UID: \"44a3edcb-2c75-49eb-a134-a5d98f105534\") " Oct 30 22:22:48.213575 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.207968 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a3edcb-2c75-49eb-a134-a5d98f105534-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a3edcb-2c75-49eb-a134-a5d98f105534-workload-socket\") pod \"44a3edcb-2c75-49eb-a134-a5d98f105534\" (UID: \"44a3edcb-2c75-49eb-a134-a5d98f105534\") " Oct 30 22:22:48.213575 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.208045 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44a3edcb-2c75-49eb-a134-a5d98f105534-kube-api-access-h2chv\" (UniqueName: \"kubernetes.io/projected/44a3edcb-2c75-49eb-a134-a5d98f105534-kube-api-access-h2chv\") pod \"44a3edcb-2c75-49eb-a134-a5d98f105534\" (UID: \"44a3edcb-2c75-49eb-a134-a5d98f105534\") " Oct 30 22:22:48.213575 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.208097 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44a3edcb-2c75-49eb-a134-a5d98f105534-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a3edcb-2c75-49eb-a134-a5d98f105534-istio-podinfo\") pod \"44a3edcb-2c75-49eb-a134-a5d98f105534\" (UID: \"44a3edcb-2c75-49eb-a134-a5d98f105534\") " Oct 30 22:22:48.213575 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.208120 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44a3edcb-2c75-49eb-a134-a5d98f105534-istio-token\" (UniqueName: \"kubernetes.io/projected/44a3edcb-2c75-49eb-a134-a5d98f105534-istio-token\") pod \"44a3edcb-2c75-49eb-a134-a5d98f105534\" (UID: \"44a3edcb-2c75-49eb-a134-a5d98f105534\") " Oct 30 22:22:48.213575 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.208197 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a3edcb-2c75-49eb-a134-a5d98f105534-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a3edcb-2c75-49eb-a134-a5d98f105534-credential-socket\") pod \"44a3edcb-2c75-49eb-a134-a5d98f105534\" (UID: \"44a3edcb-2c75-49eb-a134-a5d98f105534\") " Oct 30 22:22:48.213575 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.208271 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a3edcb-2c75-49eb-a134-a5d98f105534-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a3edcb-2c75-49eb-a134-a5d98f105534-istio-envoy\") pod \"44a3edcb-2c75-49eb-a134-a5d98f105534\" (UID: \"44a3edcb-2c75-49eb-a134-a5d98f105534\") " Oct 30 22:22:48.213575 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.208781 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a3edcb-2c75-49eb-a134-a5d98f105534-workload-socket" pod "44a3edcb-2c75-49eb-a134-a5d98f105534" (UID: "44a3edcb-2c75-49eb-a134-a5d98f105534"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:48.213575 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.208981 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a3edcb-2c75-49eb-a134-a5d98f105534-istio-data" pod "44a3edcb-2c75-49eb-a134-a5d98f105534" (UID: "44a3edcb-2c75-49eb-a134-a5d98f105534"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:48.213575 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.209024 5918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a3edcb-2c75-49eb-a134-a5d98f105534-workload-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:48.213575 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.209333 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44a3edcb-2c75-49eb-a134-a5d98f105534-istiod-ca-cert" pod "44a3edcb-2c75-49eb-a134-a5d98f105534" (UID: "44a3edcb-2c75-49eb-a134-a5d98f105534"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:22:48.213575 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.209591 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a3edcb-2c75-49eb-a134-a5d98f105534-workload-certs" pod "44a3edcb-2c75-49eb-a134-a5d98f105534" (UID: "44a3edcb-2c75-49eb-a134-a5d98f105534"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:48.213575 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.209766 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a3edcb-2c75-49eb-a134-a5d98f105534-credential-socket" pod "44a3edcb-2c75-49eb-a134-a5d98f105534" (UID: "44a3edcb-2c75-49eb-a134-a5d98f105534"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:48.220001 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.219947 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a3edcb-2c75-49eb-a134-a5d98f105534-istio-envoy" pod "44a3edcb-2c75-49eb-a134-a5d98f105534" (UID: "44a3edcb-2c75-49eb-a134-a5d98f105534"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:48.220551 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.220519 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a3edcb-2c75-49eb-a134-a5d98f105534-istio-token" pod "44a3edcb-2c75-49eb-a134-a5d98f105534" (UID: "44a3edcb-2c75-49eb-a134-a5d98f105534"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:22:48.220777 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.220752 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44a3edcb-2c75-49eb-a134-a5d98f105534-istio-podinfo" pod "44a3edcb-2c75-49eb-a134-a5d98f105534" (UID: "44a3edcb-2c75-49eb-a134-a5d98f105534"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:22:48.222234 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.222188 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a3edcb-2c75-49eb-a134-a5d98f105534-kube-api-access-h2chv" pod "44a3edcb-2c75-49eb-a134-a5d98f105534" (UID: "44a3edcb-2c75-49eb-a134-a5d98f105534"). InnerVolumeSpecName "kube-api-access-h2chv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:22:48.312242 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.311446 5918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a3edcb-2c75-49eb-a134-a5d98f105534-istio-data\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:48.312867 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.312836 5918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a3edcb-2c75-49eb-a134-a5d98f105534-istiod-ca-cert\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:48.313097 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.313079 5918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a3edcb-2c75-49eb-a134-a5d98f105534-workload-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:48.313207 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.313192 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2chv\" (UniqueName: \"kubernetes.io/projected/44a3edcb-2c75-49eb-a134-a5d98f105534-kube-api-access-h2chv\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:48.313348 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.313319 5918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a3edcb-2c75-49eb-a134-a5d98f105534-istio-podinfo\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:48.313348 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.313342 5918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a3edcb-2c75-49eb-a134-a5d98f105534-istio-token\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:48.313533 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.313359 5918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a3edcb-2c75-49eb-a134-a5d98f105534-credential-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:48.313533 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.313374 5918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a3edcb-2c75-49eb-a134-a5d98f105534-istio-envoy\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:48.563314 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.563193 5918 generic.go:356] "Generic (PLEG): container finished" podID="44a3edcb-2c75-49eb-a134-a5d98f105534" containerID="5637de566e41b2ffe03f1e87e514f794fb8873b51e321d47f56d1de6efe58635" exitCode=0 Oct 30 22:22:48.563314 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.563260 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" Oct 30 22:22:48.563314 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.563269 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" event={"ID":"44a3edcb-2c75-49eb-a134-a5d98f105534","Type":"ContainerDied","Data":"5637de566e41b2ffe03f1e87e514f794fb8873b51e321d47f56d1de6efe58635"} Oct 30 22:22:48.563837 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.563326 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk" event={"ID":"44a3edcb-2c75-49eb-a134-a5d98f105534","Type":"ContainerDied","Data":"d04c5d827bf00a93ca76027b303efdd8729ddc581fa91669342e286980c9ce1e"} Oct 30 22:22:48.563837 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.563349 5918 scope.go:122] "RemoveContainer" containerID="5637de566e41b2ffe03f1e87e514f794fb8873b51e321d47f56d1de6efe58635" Oct 30 22:22:48.565756 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.565734 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" event={"ID":"0143cf33-1443-4232-8b8e-2f7e448b78c8","Type":"ContainerStarted","Data":"298f3a6f3559b9399a139a58466a15e2b0f9295da73accc8d9e12e5f55a9b2a3"} Oct 30 22:22:48.574428 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.574412 5918 scope.go:122] "RemoveContainer" containerID="5637de566e41b2ffe03f1e87e514f794fb8873b51e321d47f56d1de6efe58635" Oct 30 22:22:48.574695 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:22:48.574677 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5637de566e41b2ffe03f1e87e514f794fb8873b51e321d47f56d1de6efe58635\": container with ID starting with 5637de566e41b2ffe03f1e87e514f794fb8873b51e321d47f56d1de6efe58635 not found: ID does not exist" containerID="5637de566e41b2ffe03f1e87e514f794fb8873b51e321d47f56d1de6efe58635" Oct 30 22:22:48.574756 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.574702 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5637de566e41b2ffe03f1e87e514f794fb8873b51e321d47f56d1de6efe58635"} err="failed to get container status \"5637de566e41b2ffe03f1e87e514f794fb8873b51e321d47f56d1de6efe58635\": rpc error: code = NotFound desc = could not find container \"5637de566e41b2ffe03f1e87e514f794fb8873b51e321d47f56d1de6efe58635\": container with ID starting with 5637de566e41b2ffe03f1e87e514f794fb8873b51e321d47f56d1de6efe58635 not found: ID does not exist" Oct 30 22:22:48.617292 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.617256 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:22:48.618607 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.618574 5918 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 22:22:48.618752 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.618638 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" podUID="0143cf33-1443-4232-8b8e-2f7e448b78c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 22:22:48.867241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.867137 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk"] Oct 30 22:22:48.877488 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:48.877456 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-sflfk"] Oct 30 22:22:49.617681 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:49.617639 5918 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 22:22:49.618177 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:49.617686 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" podUID="0143cf33-1443-4232-8b8e-2f7e448b78c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 22:22:49.854738 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:49.854705 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44a3edcb-2c75-49eb-a134-a5d98f105534" path="/var/lib/kubelet/pods/44a3edcb-2c75-49eb-a134-a5d98f105534/volumes" Oct 30 22:22:50.155623 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:50.155590 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:22:50.617768 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:50.617727 5918 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 22:22:50.618231 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:50.617800 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" podUID="0143cf33-1443-4232-8b8e-2f7e448b78c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 22:22:51.275709 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:22:51.275667 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:22:51.618293 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:51.617468 5918 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 22:22:51.618293 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:51.617529 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" podUID="0143cf33-1443-4232-8b8e-2f7e448b78c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 22:22:52.617519 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:52.617481 5918 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 22:22:52.617704 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:52.617551 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" podUID="0143cf33-1443-4232-8b8e-2f7e448b78c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 22:22:53.436907 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:53.436868 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8"] Oct 30 22:22:53.437328 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:53.437080 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" podUID="d5d61cd5-fed8-44a1-9b3a-2e7b942f0947" containerName="istio-proxy" containerID="cri-o://887d38aef35e0bd61710f72347e4b052ad5345ea4414894c7a7ded41e1dae6ea" gracePeriod=30 Oct 30 22:22:53.617653 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:53.617609 5918 patch_prober.go:29] interesting pod/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 22:22:53.617868 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:53.617669 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" podUID="0143cf33-1443-4232-8b8e-2f7e448b78c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 22:22:54.623528 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:54.623496 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:22:54.624712 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:54.624688 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:22:54.931430 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:54.931398 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76"] Oct 30 22:22:54.931684 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:54.931659 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" podUID="ed6f9c77-47bb-4e52-9e6e-1493201547dd" containerName="istio-proxy" containerID="cri-o://f7cb38a1c3461af578cf7006f1647421d30cfd936731adcea5240184d9b98871" gracePeriod=30 Oct 30 22:22:55.076038 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:55.076003 5918 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:22:55.076383 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:55.076352 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" podUID="ed6f9c77-47bb-4e52-9e6e-1493201547dd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:22:55.077173 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:55.077142 5918 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:22:55.077344 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:55.077198 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" podUID="d5d61cd5-fed8-44a1-9b3a-2e7b942f0947" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:22:58.675021 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:58.674984 5918 generic.go:356] "Generic (PLEG): container finished" podID="d5d61cd5-fed8-44a1-9b3a-2e7b942f0947" containerID="887d38aef35e0bd61710f72347e4b052ad5345ea4414894c7a7ded41e1dae6ea" exitCode=0 Oct 30 22:22:58.675528 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:58.675046 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" event={"ID":"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947","Type":"ContainerDied","Data":"887d38aef35e0bd61710f72347e4b052ad5345ea4414894c7a7ded41e1dae6ea"} Oct 30 22:22:58.718693 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:58.718638 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" Oct 30 22:22:58.861780 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:58.861746 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-credential-socket\") pod \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\" (UID: \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\") " Oct 30 22:22:58.861943 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:58.861796 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-kube-api-access-l2jtc\" (UniqueName: \"kubernetes.io/projected/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-kube-api-access-l2jtc\") pod \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\" (UID: \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\") " Oct 30 22:22:58.861943 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:58.861848 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-istio-token\" (UniqueName: \"kubernetes.io/projected/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-istio-token\") pod \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\" (UID: \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\") " Oct 30 22:22:58.861943 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:58.861906 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-istio-podinfo\") pod \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\" (UID: \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\") " Oct 30 22:22:58.862068 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:58.861956 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-istiod-ca-cert\") pod \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\" (UID: \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\") " Oct 30 22:22:58.862068 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:58.862008 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-istio-envoy\") pod \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\" (UID: \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\") " Oct 30 22:22:58.862068 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:58.862031 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-credential-socket" pod "d5d61cd5-fed8-44a1-9b3a-2e7b942f0947" (UID: "d5d61cd5-fed8-44a1-9b3a-2e7b942f0947"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:58.862068 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:58.862063 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-workload-certs\") pod \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\" (UID: \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\") " Oct 30 22:22:58.862295 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:58.862189 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-workload-socket\") pod \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\" (UID: \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\") " Oct 30 22:22:58.862387 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:58.862368 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-istio-data\") pod \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\" (UID: \"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947\") " Oct 30 22:22:58.862449 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:58.862394 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-workload-certs" pod "d5d61cd5-fed8-44a1-9b3a-2e7b942f0947" (UID: "d5d61cd5-fed8-44a1-9b3a-2e7b942f0947"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:58.862501 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:58.862480 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-istiod-ca-cert" pod "d5d61cd5-fed8-44a1-9b3a-2e7b942f0947" (UID: "d5d61cd5-fed8-44a1-9b3a-2e7b942f0947"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:22:58.862768 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:58.862745 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-istio-data" pod "d5d61cd5-fed8-44a1-9b3a-2e7b942f0947" (UID: "d5d61cd5-fed8-44a1-9b3a-2e7b942f0947"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:58.862863 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:58.862797 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-workload-socket" pod "d5d61cd5-fed8-44a1-9b3a-2e7b942f0947" (UID: "d5d61cd5-fed8-44a1-9b3a-2e7b942f0947"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:58.863240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:58.863196 5918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-workload-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:58.863279 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:58.863241 5918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-istio-data\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:58.863279 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:58.863257 5918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-credential-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:58.863279 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:58.863271 5918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-istiod-ca-cert\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:58.863374 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:58.863284 5918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-workload-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:58.864322 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:58.864296 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-istio-token" pod "d5d61cd5-fed8-44a1-9b3a-2e7b942f0947" (UID: "d5d61cd5-fed8-44a1-9b3a-2e7b942f0947"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:22:58.864482 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:58.864462 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-kube-api-access-l2jtc" pod "d5d61cd5-fed8-44a1-9b3a-2e7b942f0947" (UID: "d5d61cd5-fed8-44a1-9b3a-2e7b942f0947"). InnerVolumeSpecName "kube-api-access-l2jtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:22:58.864747 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:58.864729 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-istio-envoy" pod "d5d61cd5-fed8-44a1-9b3a-2e7b942f0947" (UID: "d5d61cd5-fed8-44a1-9b3a-2e7b942f0947"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:58.864812 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:58.864762 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-istio-podinfo" pod "d5d61cd5-fed8-44a1-9b3a-2e7b942f0947" (UID: "d5d61cd5-fed8-44a1-9b3a-2e7b942f0947"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:22:58.965037 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:58.965002 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2jtc\" (UniqueName: \"kubernetes.io/projected/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-kube-api-access-l2jtc\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:58.965037 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:58.965033 5918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-istio-token\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:58.965258 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:58.965048 5918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-istio-podinfo\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:58.965258 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:58.965061 5918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947-istio-envoy\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:59.683697 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:59.683667 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" Oct 30 22:22:59.683697 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:59.683678 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8" event={"ID":"d5d61cd5-fed8-44a1-9b3a-2e7b942f0947","Type":"ContainerDied","Data":"ee409938e9596464abc253ff337aad034db402cd366ffce835fbef0c93d41bed"} Oct 30 22:22:59.684378 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:59.683740 5918 scope.go:122] "RemoveContainer" containerID="887d38aef35e0bd61710f72347e4b052ad5345ea4414894c7a7ded41e1dae6ea" Oct 30 22:22:59.865352 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:59.865317 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8"] Oct 30 22:22:59.905565 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:22:59.905528 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-g2kv8"] Oct 30 22:23:00.615528 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:00.615501 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" Oct 30 22:23:00.686304 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:00.686274 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed6f9c77-47bb-4e52-9e6e-1493201547dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed6f9c77-47bb-4e52-9e6e-1493201547dd-istio-envoy\") pod \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\" (UID: \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\") " Oct 30 22:23:00.686760 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:00.686360 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed6f9c77-47bb-4e52-9e6e-1493201547dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed6f9c77-47bb-4e52-9e6e-1493201547dd-workload-socket\") pod \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\" (UID: \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\") " Oct 30 22:23:00.686760 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:00.686416 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed6f9c77-47bb-4e52-9e6e-1493201547dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed6f9c77-47bb-4e52-9e6e-1493201547dd-credential-socket\") pod \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\" (UID: \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\") " Oct 30 22:23:00.686760 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:00.686475 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed6f9c77-47bb-4e52-9e6e-1493201547dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed6f9c77-47bb-4e52-9e6e-1493201547dd-workload-certs\") pod \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\" (UID: \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\") " Oct 30 22:23:00.686760 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:00.686505 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed6f9c77-47bb-4e52-9e6e-1493201547dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed6f9c77-47bb-4e52-9e6e-1493201547dd-istio-podinfo\") pod \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\" (UID: \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\") " Oct 30 22:23:00.686760 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:00.686622 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed6f9c77-47bb-4e52-9e6e-1493201547dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed6f9c77-47bb-4e52-9e6e-1493201547dd-istio-data\") pod \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\" (UID: \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\") " Oct 30 22:23:00.686760 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:00.686662 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed6f9c77-47bb-4e52-9e6e-1493201547dd-istio-token\" (UniqueName: \"kubernetes.io/projected/ed6f9c77-47bb-4e52-9e6e-1493201547dd-istio-token\") pod \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\" (UID: \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\") " Oct 30 22:23:00.686760 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:00.686672 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed6f9c77-47bb-4e52-9e6e-1493201547dd-workload-socket" pod "ed6f9c77-47bb-4e52-9e6e-1493201547dd" (UID: "ed6f9c77-47bb-4e52-9e6e-1493201547dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:00.686760 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:00.686727 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed6f9c77-47bb-4e52-9e6e-1493201547dd-kube-api-access-2wd98\" (UniqueName: \"kubernetes.io/projected/ed6f9c77-47bb-4e52-9e6e-1493201547dd-kube-api-access-2wd98\") pod \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\" (UID: \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\") " Oct 30 22:23:00.687148 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:00.686788 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed6f9c77-47bb-4e52-9e6e-1493201547dd-credential-socket" pod "ed6f9c77-47bb-4e52-9e6e-1493201547dd" (UID: "ed6f9c77-47bb-4e52-9e6e-1493201547dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:00.687148 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:00.686803 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed6f9c77-47bb-4e52-9e6e-1493201547dd-workload-certs" pod "ed6f9c77-47bb-4e52-9e6e-1493201547dd" (UID: "ed6f9c77-47bb-4e52-9e6e-1493201547dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:00.687148 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:00.686906 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed6f9c77-47bb-4e52-9e6e-1493201547dd-istio-data" pod "ed6f9c77-47bb-4e52-9e6e-1493201547dd" (UID: "ed6f9c77-47bb-4e52-9e6e-1493201547dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:00.687148 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:00.686911 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed6f9c77-47bb-4e52-9e6e-1493201547dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed6f9c77-47bb-4e52-9e6e-1493201547dd-istiod-ca-cert\") pod \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\" (UID: \"ed6f9c77-47bb-4e52-9e6e-1493201547dd\") " Oct 30 22:23:00.687371 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:00.687304 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed6f9c77-47bb-4e52-9e6e-1493201547dd-istiod-ca-cert" pod "ed6f9c77-47bb-4e52-9e6e-1493201547dd" (UID: "ed6f9c77-47bb-4e52-9e6e-1493201547dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:00.687573 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:00.687550 5918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed6f9c77-47bb-4e52-9e6e-1493201547dd-istiod-ca-cert\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:00.687649 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:00.687577 5918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed6f9c77-47bb-4e52-9e6e-1493201547dd-workload-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:00.687649 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:00.687593 5918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed6f9c77-47bb-4e52-9e6e-1493201547dd-credential-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:00.687649 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:00.687606 5918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed6f9c77-47bb-4e52-9e6e-1493201547dd-workload-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:00.687649 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:00.687619 5918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed6f9c77-47bb-4e52-9e6e-1493201547dd-istio-data\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:00.689277 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:00.689250 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed6f9c77-47bb-4e52-9e6e-1493201547dd-istio-envoy" pod "ed6f9c77-47bb-4e52-9e6e-1493201547dd" (UID: "ed6f9c77-47bb-4e52-9e6e-1493201547dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:00.689520 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:00.689491 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed6f9c77-47bb-4e52-9e6e-1493201547dd-istio-podinfo" pod "ed6f9c77-47bb-4e52-9e6e-1493201547dd" (UID: "ed6f9c77-47bb-4e52-9e6e-1493201547dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:23:00.689520 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:00.689505 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed6f9c77-47bb-4e52-9e6e-1493201547dd-istio-token" pod "ed6f9c77-47bb-4e52-9e6e-1493201547dd" (UID: "ed6f9c77-47bb-4e52-9e6e-1493201547dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:00.689674 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:00.689586 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed6f9c77-47bb-4e52-9e6e-1493201547dd-kube-api-access-2wd98" pod "ed6f9c77-47bb-4e52-9e6e-1493201547dd" (UID: "ed6f9c77-47bb-4e52-9e6e-1493201547dd"). InnerVolumeSpecName "kube-api-access-2wd98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:00.693839 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:00.693813 5918 generic.go:356] "Generic (PLEG): container finished" podID="ed6f9c77-47bb-4e52-9e6e-1493201547dd" containerID="f7cb38a1c3461af578cf7006f1647421d30cfd936731adcea5240184d9b98871" exitCode=0 Oct 30 22:23:00.693947 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:00.693864 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" Oct 30 22:23:00.693947 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:00.693898 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" event={"ID":"ed6f9c77-47bb-4e52-9e6e-1493201547dd","Type":"ContainerDied","Data":"f7cb38a1c3461af578cf7006f1647421d30cfd936731adcea5240184d9b98871"} Oct 30 22:23:00.694064 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:00.693959 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76" event={"ID":"ed6f9c77-47bb-4e52-9e6e-1493201547dd","Type":"ContainerDied","Data":"2514a25795306ec0595357aaf06ccb4503f1f6665b95a5b9ba01688bad6c1015"} Oct 30 22:23:00.694064 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:00.693991 5918 scope.go:122] "RemoveContainer" containerID="f7cb38a1c3461af578cf7006f1647421d30cfd936731adcea5240184d9b98871" Oct 30 22:23:00.723086 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:00.723068 5918 scope.go:122] "RemoveContainer" containerID="f7cb38a1c3461af578cf7006f1647421d30cfd936731adcea5240184d9b98871" Oct 30 22:23:00.723507 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:23:00.723477 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7cb38a1c3461af578cf7006f1647421d30cfd936731adcea5240184d9b98871\": container with ID starting with f7cb38a1c3461af578cf7006f1647421d30cfd936731adcea5240184d9b98871 not found: ID does not exist" containerID="f7cb38a1c3461af578cf7006f1647421d30cfd936731adcea5240184d9b98871" Oct 30 22:23:00.723567 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:00.723519 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7cb38a1c3461af578cf7006f1647421d30cfd936731adcea5240184d9b98871"} err="failed to get container status \"f7cb38a1c3461af578cf7006f1647421d30cfd936731adcea5240184d9b98871\": rpc error: code = NotFound desc = could not find container \"f7cb38a1c3461af578cf7006f1647421d30cfd936731adcea5240184d9b98871\": container with ID starting with f7cb38a1c3461af578cf7006f1647421d30cfd936731adcea5240184d9b98871 not found: ID does not exist" Oct 30 22:23:00.788895 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:00.788814 5918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed6f9c77-47bb-4e52-9e6e-1493201547dd-istio-podinfo\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:00.788895 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:00.788850 5918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed6f9c77-47bb-4e52-9e6e-1493201547dd-istio-token\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:00.788895 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:00.788865 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wd98\" (UniqueName: \"kubernetes.io/projected/ed6f9c77-47bb-4e52-9e6e-1493201547dd-kube-api-access-2wd98\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:00.788895 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:00.788880 5918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed6f9c77-47bb-4e52-9e6e-1493201547dd-istio-envoy\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:00.857386 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:00.857355 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76"] Oct 30 22:23:00.862478 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:00.862453 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-dlf76"] Oct 30 22:23:01.808171 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:01.808137 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-81f37a-9454c4d75-fmvwr"] Oct 30 22:23:01.853844 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:01.853812 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5d61cd5-fed8-44a1-9b3a-2e7b942f0947" path="/var/lib/kubelet/pods/d5d61cd5-fed8-44a1-9b3a-2e7b942f0947/volumes" Oct 30 22:23:01.854342 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:01.854329 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed6f9c77-47bb-4e52-9e6e-1493201547dd" path="/var/lib/kubelet/pods/ed6f9c77-47bb-4e52-9e6e-1493201547dd/volumes" Oct 30 22:23:05.423385 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:05.423351 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-18edc4-87574c999-vcrzw"] Oct 30 22:23:05.423995 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:05.423982 5918 cpu_manager.go:414] "containerMap: removing container" podUID="44a3edcb-2c75-49eb-a134-a5d98f105534" containerName="istio-proxy" Oct 30 22:23:05.424043 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:05.423996 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44a3edcb-2c75-49eb-a134-a5d98f105534" containerName="istio-proxy" Oct 30 22:23:05.424043 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:05.424011 5918 cpu_manager.go:414] "containerMap: removing container" podUID="ed6f9c77-47bb-4e52-9e6e-1493201547dd" containerName="istio-proxy" Oct 30 22:23:05.424043 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:05.424016 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed6f9c77-47bb-4e52-9e6e-1493201547dd" containerName="istio-proxy" Oct 30 22:23:05.424043 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:05.424024 5918 cpu_manager.go:414] "containerMap: removing container" podUID="d5d61cd5-fed8-44a1-9b3a-2e7b942f0947" containerName="istio-proxy" Oct 30 22:23:05.424043 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:05.424031 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5d61cd5-fed8-44a1-9b3a-2e7b942f0947" containerName="istio-proxy" Oct 30 22:23:05.424043 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:05.424041 5918 cpu_manager.go:414] "containerMap: removing container" podUID="34335b46-c602-47cb-92f4-62df851dd01b" containerName="istio-proxy" Oct 30 22:23:05.424043 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:05.424046 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34335b46-c602-47cb-92f4-62df851dd01b" containerName="istio-proxy" Oct 30 22:23:05.424305 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:05.424085 5918 cpu_manager.go:414] "containerMap: removing container" podUID="7df7a416-e18e-4713-89b1-5b3f21e9251d" containerName="istio-proxy" Oct 30 22:23:05.424305 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:05.424090 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7df7a416-e18e-4713-89b1-5b3f21e9251d" containerName="istio-proxy" Oct 30 22:23:05.424305 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:05.424101 5918 cpu_manager.go:414] "containerMap: removing container" podUID="471d388d-0ab5-4bec-ac57-93d6f4df1bd6" containerName="payload-pre-processing" Oct 30 22:23:05.424305 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:05.424105 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="471d388d-0ab5-4bec-ac57-93d6f4df1bd6" containerName="payload-pre-processing" Oct 30 22:23:05.424305 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:05.424207 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="44a3edcb-2c75-49eb-a134-a5d98f105534" containerName="istio-proxy" Oct 30 22:23:05.424305 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:05.424232 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed6f9c77-47bb-4e52-9e6e-1493201547dd" containerName="istio-proxy" Oct 30 22:23:05.424305 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:05.424240 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5d61cd5-fed8-44a1-9b3a-2e7b942f0947" containerName="istio-proxy" Oct 30 22:23:05.424305 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:05.424246 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="34335b46-c602-47cb-92f4-62df851dd01b" containerName="istio-proxy" Oct 30 22:23:05.424305 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:05.424259 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="7df7a416-e18e-4713-89b1-5b3f21e9251d" containerName="istio-proxy" Oct 30 22:23:05.424305 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:05.424266 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="471d388d-0ab5-4bec-ac57-93d6f4df1bd6" containerName="payload-pre-processing" Oct 30 22:23:05.428992 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:05.428977 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18edc4-87574c999-vcrzw" Oct 30 22:23:05.534442 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:05.534413 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-18edc4-serving-cert\"" Oct 30 22:23:05.537826 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:05.537801 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-18edc4-87574c999-vcrzw"] Oct 30 22:23:05.561303 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:05.561274 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75b6d2cb-35f2-47fc-a7f5-aa19149ca875-metrics-certs\") pod \"maas-api-e2e-inf-a-18edc4-87574c999-vcrzw\" (UID: \"75b6d2cb-35f2-47fc-a7f5-aa19149ca875\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18edc4-87574c999-vcrzw" Oct 30 22:23:05.561435 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:05.561314 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75b6d2cb-35f2-47fc-a7f5-aa19149ca875-openshift-service-ca\") pod \"maas-api-e2e-inf-a-18edc4-87574c999-vcrzw\" (UID: \"75b6d2cb-35f2-47fc-a7f5-aa19149ca875\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18edc4-87574c999-vcrzw" Oct 30 22:23:05.561435 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:05.561353 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75b6d2cb-35f2-47fc-a7f5-aa19149ca875-maas-api-tls\") pod \"maas-api-e2e-inf-a-18edc4-87574c999-vcrzw\" (UID: \"75b6d2cb-35f2-47fc-a7f5-aa19149ca875\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18edc4-87574c999-vcrzw" Oct 30 22:23:05.561635 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:05.561612 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7p7x\" (UniqueName: \"kubernetes.io/projected/75b6d2cb-35f2-47fc-a7f5-aa19149ca875-kube-api-access-p7p7x\") pod \"maas-api-e2e-inf-a-18edc4-87574c999-vcrzw\" (UID: \"75b6d2cb-35f2-47fc-a7f5-aa19149ca875\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18edc4-87574c999-vcrzw" Oct 30 22:23:05.663111 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:05.663074 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7p7x\" (UniqueName: \"kubernetes.io/projected/75b6d2cb-35f2-47fc-a7f5-aa19149ca875-kube-api-access-p7p7x\") pod \"maas-api-e2e-inf-a-18edc4-87574c999-vcrzw\" (UID: \"75b6d2cb-35f2-47fc-a7f5-aa19149ca875\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18edc4-87574c999-vcrzw" Oct 30 22:23:05.663349 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:05.663191 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75b6d2cb-35f2-47fc-a7f5-aa19149ca875-metrics-certs\") pod \"maas-api-e2e-inf-a-18edc4-87574c999-vcrzw\" (UID: \"75b6d2cb-35f2-47fc-a7f5-aa19149ca875\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18edc4-87574c999-vcrzw" Oct 30 22:23:05.663349 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:05.663249 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75b6d2cb-35f2-47fc-a7f5-aa19149ca875-openshift-service-ca\") pod \"maas-api-e2e-inf-a-18edc4-87574c999-vcrzw\" (UID: \"75b6d2cb-35f2-47fc-a7f5-aa19149ca875\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18edc4-87574c999-vcrzw" Oct 30 22:23:05.663349 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:05.663310 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75b6d2cb-35f2-47fc-a7f5-aa19149ca875-maas-api-tls\") pod \"maas-api-e2e-inf-a-18edc4-87574c999-vcrzw\" (UID: \"75b6d2cb-35f2-47fc-a7f5-aa19149ca875\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18edc4-87574c999-vcrzw" Oct 30 22:23:05.663571 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:23:05.663552 5918 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-18edc4-serving-cert: secret "maas-api-e2e-inf-a-18edc4-serving-cert" not found Oct 30 22:23:05.663631 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:23:05.663615 5918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75b6d2cb-35f2-47fc-a7f5-aa19149ca875-maas-api-tls podName:75b6d2cb-35f2-47fc-a7f5-aa19149ca875 nodeName:}" failed. No retries permitted until 2026-10-30 22:23:06.163598842 +0000 UTC m=+1946.434296906 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/75b6d2cb-35f2-47fc-a7f5-aa19149ca875-maas-api-tls") pod "maas-api-e2e-inf-a-18edc4-87574c999-vcrzw" (UID: "75b6d2cb-35f2-47fc-a7f5-aa19149ca875") : secret "maas-api-e2e-inf-a-18edc4-serving-cert" not found Oct 30 22:23:05.664194 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:05.664171 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75b6d2cb-35f2-47fc-a7f5-aa19149ca875-openshift-service-ca\") pod \"maas-api-e2e-inf-a-18edc4-87574c999-vcrzw\" (UID: \"75b6d2cb-35f2-47fc-a7f5-aa19149ca875\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18edc4-87574c999-vcrzw" Oct 30 22:23:05.666010 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:05.665988 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75b6d2cb-35f2-47fc-a7f5-aa19149ca875-metrics-certs\") pod \"maas-api-e2e-inf-a-18edc4-87574c999-vcrzw\" (UID: \"75b6d2cb-35f2-47fc-a7f5-aa19149ca875\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18edc4-87574c999-vcrzw" Oct 30 22:23:05.732649 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:05.732614 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7p7x\" (UniqueName: \"kubernetes.io/projected/75b6d2cb-35f2-47fc-a7f5-aa19149ca875-kube-api-access-p7p7x\") pod \"maas-api-e2e-inf-a-18edc4-87574c999-vcrzw\" (UID: \"75b6d2cb-35f2-47fc-a7f5-aa19149ca875\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18edc4-87574c999-vcrzw" Oct 30 22:23:06.169572 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:06.169487 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75b6d2cb-35f2-47fc-a7f5-aa19149ca875-maas-api-tls\") pod \"maas-api-e2e-inf-a-18edc4-87574c999-vcrzw\" (UID: \"75b6d2cb-35f2-47fc-a7f5-aa19149ca875\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18edc4-87574c999-vcrzw" Oct 30 22:23:06.173054 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:06.173026 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75b6d2cb-35f2-47fc-a7f5-aa19149ca875-maas-api-tls\") pod \"maas-api-e2e-inf-a-18edc4-87574c999-vcrzw\" (UID: \"75b6d2cb-35f2-47fc-a7f5-aa19149ca875\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18edc4-87574c999-vcrzw" Oct 30 22:23:06.343046 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:06.343002 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18edc4-87574c999-vcrzw" Oct 30 22:23:06.723345 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:06.723314 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-18edc4-87574c999-vcrzw"] Oct 30 22:23:06.723744 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:23:06.723722 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b6d2cb_35f2_47fc_a7f5_aa19149ca875.slice/crio-bdfe01412757bceedb2c11f903b5ab479eafc7f0d777c5328540531c0d0434e5 WatchSource:0}: Error finding container bdfe01412757bceedb2c11f903b5ab479eafc7f0d777c5328540531c0d0434e5: Status 404 returned error can't find the container with id bdfe01412757bceedb2c11f903b5ab479eafc7f0d777c5328540531c0d0434e5 Oct 30 22:23:06.725446 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:06.725427 5918 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:23:06.734857 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:06.734822 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18edc4-87574c999-vcrzw" event={"ID":"75b6d2cb-35f2-47fc-a7f5-aa19149ca875","Type":"ContainerStarted","Data":"bdfe01412757bceedb2c11f903b5ab479eafc7f0d777c5328540531c0d0434e5"} Oct 30 22:23:06.808819 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:06.808774 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-81f37a-9454c4d75-fmvwr" podUID="f998d536-e995-48fb-9c20-c5cfbabaaa2f" containerName="payload-pre-processing" containerID="cri-o://8d143da20bcc86da56a8b143b96dd06e0d35fb07dcff4ecbda89e9dffff4771b" gracePeriod=25 Oct 30 22:23:07.085695 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:07.085598 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-81f37a-9454c4d75-fmvwr" Oct 30 22:23:07.183333 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:07.183298 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f998d536-e995-48fb-9c20-c5cfbabaaa2f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f998d536-e995-48fb-9c20-c5cfbabaaa2f-plugins-config-volume\") pod \"f998d536-e995-48fb-9c20-c5cfbabaaa2f\" (UID: \"f998d536-e995-48fb-9c20-c5cfbabaaa2f\") " Oct 30 22:23:07.183525 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:07.183516 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f998d536-e995-48fb-9c20-c5cfbabaaa2f-kube-api-access-8r827\" (UniqueName: \"kubernetes.io/projected/f998d536-e995-48fb-9c20-c5cfbabaaa2f-kube-api-access-8r827\") pod \"f998d536-e995-48fb-9c20-c5cfbabaaa2f\" (UID: \"f998d536-e995-48fb-9c20-c5cfbabaaa2f\") " Oct 30 22:23:07.183721 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:07.183696 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f998d536-e995-48fb-9c20-c5cfbabaaa2f-plugins-config-volume" pod "f998d536-e995-48fb-9c20-c5cfbabaaa2f" (UID: "f998d536-e995-48fb-9c20-c5cfbabaaa2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:07.184070 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:07.183998 5918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f998d536-e995-48fb-9c20-c5cfbabaaa2f-plugins-config-volume\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:07.185938 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:07.185915 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f998d536-e995-48fb-9c20-c5cfbabaaa2f-kube-api-access-8r827" pod "f998d536-e995-48fb-9c20-c5cfbabaaa2f" (UID: "f998d536-e995-48fb-9c20-c5cfbabaaa2f"). InnerVolumeSpecName "kube-api-access-8r827". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:07.285320 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:07.285209 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r827\" (UniqueName: \"kubernetes.io/projected/f998d536-e995-48fb-9c20-c5cfbabaaa2f-kube-api-access-8r827\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:07.745174 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:07.745136 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18edc4-87574c999-vcrzw" event={"ID":"75b6d2cb-35f2-47fc-a7f5-aa19149ca875","Type":"ContainerStarted","Data":"f5c2d7c258005b1baf656bd11ee2033dc4a2d9a9f43e5cbb23f6e7995ea9b4a2"} Oct 30 22:23:07.745721 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:07.745210 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18edc4-87574c999-vcrzw" Oct 30 22:23:07.746646 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:07.746622 5918 generic.go:356] "Generic (PLEG): container finished" podID="f998d536-e995-48fb-9c20-c5cfbabaaa2f" containerID="8d143da20bcc86da56a8b143b96dd06e0d35fb07dcff4ecbda89e9dffff4771b" exitCode=0 Oct 30 22:23:07.746730 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:07.746657 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-81f37a-9454c4d75-fmvwr" Oct 30 22:23:07.746779 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:07.746661 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-81f37a-9454c4d75-fmvwr" event={"ID":"f998d536-e995-48fb-9c20-c5cfbabaaa2f","Type":"ContainerDied","Data":"8d143da20bcc86da56a8b143b96dd06e0d35fb07dcff4ecbda89e9dffff4771b"} Oct 30 22:23:07.746815 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:07.746797 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-81f37a-9454c4d75-fmvwr" event={"ID":"f998d536-e995-48fb-9c20-c5cfbabaaa2f","Type":"ContainerDied","Data":"9da80486782d9eed4e0e734526192bd035fc18f2e8e23c3fd5def8d00ed34a23"} Oct 30 22:23:07.746848 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:07.746827 5918 scope.go:122] "RemoveContainer" containerID="8d143da20bcc86da56a8b143b96dd06e0d35fb07dcff4ecbda89e9dffff4771b" Oct 30 22:23:07.758130 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:07.758109 5918 scope.go:122] "RemoveContainer" containerID="8d143da20bcc86da56a8b143b96dd06e0d35fb07dcff4ecbda89e9dffff4771b" Oct 30 22:23:07.758409 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:23:07.758389 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d143da20bcc86da56a8b143b96dd06e0d35fb07dcff4ecbda89e9dffff4771b\": container with ID starting with 8d143da20bcc86da56a8b143b96dd06e0d35fb07dcff4ecbda89e9dffff4771b not found: ID does not exist" containerID="8d143da20bcc86da56a8b143b96dd06e0d35fb07dcff4ecbda89e9dffff4771b" Oct 30 22:23:07.758469 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:07.758417 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d143da20bcc86da56a8b143b96dd06e0d35fb07dcff4ecbda89e9dffff4771b"} err="failed to get container status \"8d143da20bcc86da56a8b143b96dd06e0d35fb07dcff4ecbda89e9dffff4771b\": rpc error: code = NotFound desc = could not find container \"8d143da20bcc86da56a8b143b96dd06e0d35fb07dcff4ecbda89e9dffff4771b\": container with ID starting with 8d143da20bcc86da56a8b143b96dd06e0d35fb07dcff4ecbda89e9dffff4771b not found: ID does not exist" Oct 30 22:23:07.874103 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:07.873978 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18edc4-87574c999-vcrzw" podStartSLOduration=1.98551572 podStartE2EDuration="2.873960401s" totalImagesPullingTime="888.444681ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:07.869683277 +0000 UTC m=+1948.140381350" watchObservedRunningTime="2026-10-30 22:23:07.873960401 +0000 UTC m=+1948.144658488" Oct 30 22:23:07.991431 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:07.991367 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-81f37a-9454c4d75-fmvwr"] Oct 30 22:23:07.996901 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:07.996864 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-81f37a-9454c4d75-fmvwr"] Oct 30 22:23:09.857566 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:09.857534 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f998d536-e995-48fb-9c20-c5cfbabaaa2f" path="/var/lib/kubelet/pods/f998d536-e995-48fb-9c20-c5cfbabaaa2f/volumes" Oct 30 22:23:13.760130 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:13.760100 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18edc4-87574c999-vcrzw" Oct 30 22:23:21.290426 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:23:21.290385 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:23:45.807770 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:45.807731 5918 patch_prober.go:29] interesting pod/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 22:23:45.808194 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:45.807800 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" podUID="147ff712-6d65-4184-9125-9ae9391707dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 22:23:46.012710 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:46.012666 5918 generic.go:356] "Generic (PLEG): container finished" podID="147ff712-6d65-4184-9125-9ae9391707dd" containerID="6004971db453dac3f21f59c48449d733113448c0d32f6acb612ea3fe9c5e71df" exitCode=0 Oct 30 22:23:46.012880 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:46.012728 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" event={"ID":"147ff712-6d65-4184-9125-9ae9391707dd","Type":"ContainerDied","Data":"6004971db453dac3f21f59c48449d733113448c0d32f6acb612ea3fe9c5e71df"} Oct 30 22:23:46.013170 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:46.013151 5918 scope.go:122] "RemoveContainer" containerID="6004971db453dac3f21f59c48449d733113448c0d32f6acb612ea3fe9c5e71df" Oct 30 22:23:46.015162 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:46.015127 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:23:46.015260 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:46.015233 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:23:46.015308 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:46.015281 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:23:47.025784 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:47.025745 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" event={"ID":"147ff712-6d65-4184-9125-9ae9391707dd","Type":"ContainerStarted","Data":"cbd7a79e8788f2ad487bca9c7fec68a1f58ad7c4181748baa790a094b0a24884"} Oct 30 22:23:47.585192 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:47.585156 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:23:47.585396 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:47.585369 5918 patch_prober.go:29] interesting pod/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 22:23:47.585458 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:47.585428 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" podUID="147ff712-6d65-4184-9125-9ae9391707dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 22:23:48.585243 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:48.585175 5918 patch_prober.go:29] interesting pod/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 22:23:48.585725 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:48.585260 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" podUID="147ff712-6d65-4184-9125-9ae9391707dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 22:23:49.585472 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:49.585380 5918 patch_prober.go:29] interesting pod/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 22:23:49.585472 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:49.585448 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" podUID="147ff712-6d65-4184-9125-9ae9391707dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 22:23:50.585691 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:50.585655 5918 patch_prober.go:29] interesting pod/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 22:23:50.586073 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:50.585743 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" podUID="147ff712-6d65-4184-9125-9ae9391707dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 22:23:51.275755 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:23:51.275712 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:23:51.585633 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:51.585541 5918 patch_prober.go:29] interesting pod/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 22:23:51.585633 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:51.585600 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" podUID="147ff712-6d65-4184-9125-9ae9391707dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 22:23:52.590119 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:52.590088 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:23:52.590545 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:52.590440 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:23:52.591090 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:23:52.591075 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:24:05.163647 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:05.163612 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9b614c-c47575765-qqmbn"] Oct 30 22:24:05.291584 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:05.291551 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-64gml"] Oct 30 22:24:10.164841 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:10.164789 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9b614c-c47575765-qqmbn" podUID="cefcee5b-c92a-4067-8c83-cea422238674" containerName="payload-processing" containerID="cri-o://d58a92385779f79ed60ddb253dd6e752f4ea2225388bbefda44e648fd240e700" gracePeriod=25 Oct 30 22:24:10.292716 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:10.292529 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-64gml" podUID="bc8c80b0-9111-4da5-908c-fe7f3f99acb8" containerName="payload-pre-processing" containerID="cri-o://b17f2c33bff5377b211e62bc8eb6ddf437b6e411617dc2a02e55b482c5f47a3a" gracePeriod=25 Oct 30 22:24:10.529773 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:10.529709 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9b614c-c47575765-qqmbn" Oct 30 22:24:10.636776 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:10.636736 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cefcee5b-c92a-4067-8c83-cea422238674-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cefcee5b-c92a-4067-8c83-cea422238674-plugins-config-volume\") pod \"cefcee5b-c92a-4067-8c83-cea422238674\" (UID: \"cefcee5b-c92a-4067-8c83-cea422238674\") " Oct 30 22:24:10.637231 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:10.637068 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cefcee5b-c92a-4067-8c83-cea422238674-kube-api-access-swcwn\" (UniqueName: \"kubernetes.io/projected/cefcee5b-c92a-4067-8c83-cea422238674-kube-api-access-swcwn\") pod \"cefcee5b-c92a-4067-8c83-cea422238674\" (UID: \"cefcee5b-c92a-4067-8c83-cea422238674\") " Oct 30 22:24:10.637231 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:10.637150 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cefcee5b-c92a-4067-8c83-cea422238674-plugins-config-volume" pod "cefcee5b-c92a-4067-8c83-cea422238674" (UID: "cefcee5b-c92a-4067-8c83-cea422238674"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:10.637770 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:10.637746 5918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cefcee5b-c92a-4067-8c83-cea422238674-plugins-config-volume\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:10.640390 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:10.640362 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cefcee5b-c92a-4067-8c83-cea422238674-kube-api-access-swcwn" pod "cefcee5b-c92a-4067-8c83-cea422238674" (UID: "cefcee5b-c92a-4067-8c83-cea422238674"). InnerVolumeSpecName "kube-api-access-swcwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:10.677988 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:10.677938 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-64gml" Oct 30 22:24:10.739367 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:10.739266 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc8c80b0-9111-4da5-908c-fe7f3f99acb8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc8c80b0-9111-4da5-908c-fe7f3f99acb8-plugins-config-volume\") pod \"bc8c80b0-9111-4da5-908c-fe7f3f99acb8\" (UID: \"bc8c80b0-9111-4da5-908c-fe7f3f99acb8\") " Oct 30 22:24:10.739568 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:10.739378 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc8c80b0-9111-4da5-908c-fe7f3f99acb8-kube-api-access-96lhv\" (UniqueName: \"kubernetes.io/projected/bc8c80b0-9111-4da5-908c-fe7f3f99acb8-kube-api-access-96lhv\") pod \"bc8c80b0-9111-4da5-908c-fe7f3f99acb8\" (UID: \"bc8c80b0-9111-4da5-908c-fe7f3f99acb8\") " Oct 30 22:24:10.739713 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:10.739689 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc8c80b0-9111-4da5-908c-fe7f3f99acb8-plugins-config-volume" pod "bc8c80b0-9111-4da5-908c-fe7f3f99acb8" (UID: "bc8c80b0-9111-4da5-908c-fe7f3f99acb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:10.740146 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:10.740124 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swcwn\" (UniqueName: \"kubernetes.io/projected/cefcee5b-c92a-4067-8c83-cea422238674-kube-api-access-swcwn\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:10.740146 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:10.740146 5918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc8c80b0-9111-4da5-908c-fe7f3f99acb8-plugins-config-volume\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:10.742057 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:10.742033 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc8c80b0-9111-4da5-908c-fe7f3f99acb8-kube-api-access-96lhv" pod "bc8c80b0-9111-4da5-908c-fe7f3f99acb8" (UID: "bc8c80b0-9111-4da5-908c-fe7f3f99acb8"). InnerVolumeSpecName "kube-api-access-96lhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:10.841704 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:10.841670 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96lhv\" (UniqueName: \"kubernetes.io/projected/bc8c80b0-9111-4da5-908c-fe7f3f99acb8-kube-api-access-96lhv\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:11.190370 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:11.190343 5918 generic.go:356] "Generic (PLEG): container finished" podID="cefcee5b-c92a-4067-8c83-cea422238674" containerID="d58a92385779f79ed60ddb253dd6e752f4ea2225388bbefda44e648fd240e700" exitCode=0 Oct 30 22:24:11.190768 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:11.190388 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9b614c-c47575765-qqmbn" Oct 30 22:24:11.190768 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:11.190420 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9b614c-c47575765-qqmbn" event={"ID":"cefcee5b-c92a-4067-8c83-cea422238674","Type":"ContainerDied","Data":"d58a92385779f79ed60ddb253dd6e752f4ea2225388bbefda44e648fd240e700"} Oct 30 22:24:11.190768 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:11.190475 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9b614c-c47575765-qqmbn" event={"ID":"cefcee5b-c92a-4067-8c83-cea422238674","Type":"ContainerDied","Data":"4586260d739d1bee18a097bf51e59c68f1da5707da478eba66926c9518f0398c"} Oct 30 22:24:11.190768 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:11.190495 5918 scope.go:122] "RemoveContainer" containerID="d58a92385779f79ed60ddb253dd6e752f4ea2225388bbefda44e648fd240e700" Oct 30 22:24:11.191874 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:11.191846 5918 generic.go:356] "Generic (PLEG): container finished" podID="bc8c80b0-9111-4da5-908c-fe7f3f99acb8" containerID="b17f2c33bff5377b211e62bc8eb6ddf437b6e411617dc2a02e55b482c5f47a3a" exitCode=0 Oct 30 22:24:11.191950 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:11.191880 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-64gml" event={"ID":"bc8c80b0-9111-4da5-908c-fe7f3f99acb8","Type":"ContainerDied","Data":"b17f2c33bff5377b211e62bc8eb6ddf437b6e411617dc2a02e55b482c5f47a3a"} Oct 30 22:24:11.191950 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:11.191925 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-64gml" event={"ID":"bc8c80b0-9111-4da5-908c-fe7f3f99acb8","Type":"ContainerDied","Data":"045d402285259811cb63f0e33ab0132d353ccad73eea05996da9bc248686f285"} Oct 30 22:24:11.191950 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:11.191891 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-64gml" Oct 30 22:24:11.202367 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:11.202350 5918 scope.go:122] "RemoveContainer" containerID="d58a92385779f79ed60ddb253dd6e752f4ea2225388bbefda44e648fd240e700" Oct 30 22:24:11.202622 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:24:11.202592 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d58a92385779f79ed60ddb253dd6e752f4ea2225388bbefda44e648fd240e700\": container with ID starting with d58a92385779f79ed60ddb253dd6e752f4ea2225388bbefda44e648fd240e700 not found: ID does not exist" containerID="d58a92385779f79ed60ddb253dd6e752f4ea2225388bbefda44e648fd240e700" Oct 30 22:24:11.202662 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:11.202630 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d58a92385779f79ed60ddb253dd6e752f4ea2225388bbefda44e648fd240e700"} err="failed to get container status \"d58a92385779f79ed60ddb253dd6e752f4ea2225388bbefda44e648fd240e700\": rpc error: code = NotFound desc = could not find container \"d58a92385779f79ed60ddb253dd6e752f4ea2225388bbefda44e648fd240e700\": container with ID starting with d58a92385779f79ed60ddb253dd6e752f4ea2225388bbefda44e648fd240e700 not found: ID does not exist" Oct 30 22:24:11.202662 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:11.202646 5918 scope.go:122] "RemoveContainer" containerID="b17f2c33bff5377b211e62bc8eb6ddf437b6e411617dc2a02e55b482c5f47a3a" Oct 30 22:24:11.211840 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:11.211822 5918 scope.go:122] "RemoveContainer" containerID="b17f2c33bff5377b211e62bc8eb6ddf437b6e411617dc2a02e55b482c5f47a3a" Oct 30 22:24:11.212109 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:24:11.212091 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b17f2c33bff5377b211e62bc8eb6ddf437b6e411617dc2a02e55b482c5f47a3a\": container with ID starting with b17f2c33bff5377b211e62bc8eb6ddf437b6e411617dc2a02e55b482c5f47a3a not found: ID does not exist" containerID="b17f2c33bff5377b211e62bc8eb6ddf437b6e411617dc2a02e55b482c5f47a3a" Oct 30 22:24:11.212151 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:11.212116 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b17f2c33bff5377b211e62bc8eb6ddf437b6e411617dc2a02e55b482c5f47a3a"} err="failed to get container status \"b17f2c33bff5377b211e62bc8eb6ddf437b6e411617dc2a02e55b482c5f47a3a\": rpc error: code = NotFound desc = could not find container \"b17f2c33bff5377b211e62bc8eb6ddf437b6e411617dc2a02e55b482c5f47a3a\": container with ID starting with b17f2c33bff5377b211e62bc8eb6ddf437b6e411617dc2a02e55b482c5f47a3a not found: ID does not exist" Oct 30 22:24:11.361608 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:11.361575 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9b614c-c47575765-qqmbn"] Oct 30 22:24:11.412359 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:11.412327 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9b614c-c47575765-qqmbn"] Oct 30 22:24:11.482642 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:11.482566 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-64gml"] Oct 30 22:24:11.490001 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:11.489970 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9b614c-56cd686967-64gml"] Oct 30 22:24:11.853117 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:11.853043 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc8c80b0-9111-4da5-908c-fe7f3f99acb8" path="/var/lib/kubelet/pods/bc8c80b0-9111-4da5-908c-fe7f3f99acb8/volumes" Oct 30 22:24:11.853551 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:11.853537 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cefcee5b-c92a-4067-8c83-cea422238674" path="/var/lib/kubelet/pods/cefcee5b-c92a-4067-8c83-cea422238674/volumes" Oct 30 22:24:21.295459 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:24:21.295410 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:24:33.221393 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:33.221361 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-f2qhs"] Oct 30 22:24:33.632998 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:33.632918 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z"] Oct 30 22:24:33.633145 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:33.633092 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z" podUID="d0d4e2b8-0f58-4b32-8227-6e3418b8521d" containerName="maas-api" containerID="cri-o://513e936f8b524119e699a9f0e5022c2943bf08b12a004a983ae4cfe053846eb3" gracePeriod=30 Oct 30 22:24:33.965006 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:33.964980 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z" Oct 30 22:24:34.053146 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:34.053101 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0d4e2b8-0f58-4b32-8227-6e3418b8521d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0d4e2b8-0f58-4b32-8227-6e3418b8521d-maas-api-tls\") pod \"d0d4e2b8-0f58-4b32-8227-6e3418b8521d\" (UID: \"d0d4e2b8-0f58-4b32-8227-6e3418b8521d\") " Oct 30 22:24:34.053370 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:34.053164 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0d4e2b8-0f58-4b32-8227-6e3418b8521d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0d4e2b8-0f58-4b32-8227-6e3418b8521d-metrics-certs\") pod \"d0d4e2b8-0f58-4b32-8227-6e3418b8521d\" (UID: \"d0d4e2b8-0f58-4b32-8227-6e3418b8521d\") " Oct 30 22:24:34.053370 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:34.053291 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0d4e2b8-0f58-4b32-8227-6e3418b8521d-kube-api-access-7wnpm\" (UniqueName: \"kubernetes.io/projected/d0d4e2b8-0f58-4b32-8227-6e3418b8521d-kube-api-access-7wnpm\") pod \"d0d4e2b8-0f58-4b32-8227-6e3418b8521d\" (UID: \"d0d4e2b8-0f58-4b32-8227-6e3418b8521d\") " Oct 30 22:24:34.053370 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:34.053355 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0d4e2b8-0f58-4b32-8227-6e3418b8521d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d4e2b8-0f58-4b32-8227-6e3418b8521d-openshift-service-ca\") pod \"d0d4e2b8-0f58-4b32-8227-6e3418b8521d\" (UID: \"d0d4e2b8-0f58-4b32-8227-6e3418b8521d\") " Oct 30 22:24:34.053837 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:34.053812 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0d4e2b8-0f58-4b32-8227-6e3418b8521d-openshift-service-ca" pod "d0d4e2b8-0f58-4b32-8227-6e3418b8521d" (UID: "d0d4e2b8-0f58-4b32-8227-6e3418b8521d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:34.054196 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:34.054174 5918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d4e2b8-0f58-4b32-8227-6e3418b8521d-openshift-service-ca\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:34.055830 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:34.055803 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0d4e2b8-0f58-4b32-8227-6e3418b8521d-metrics-certs" pod "d0d4e2b8-0f58-4b32-8227-6e3418b8521d" (UID: "d0d4e2b8-0f58-4b32-8227-6e3418b8521d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:34.055928 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:34.055897 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0d4e2b8-0f58-4b32-8227-6e3418b8521d-kube-api-access-7wnpm" pod "d0d4e2b8-0f58-4b32-8227-6e3418b8521d" (UID: "d0d4e2b8-0f58-4b32-8227-6e3418b8521d"). InnerVolumeSpecName "kube-api-access-7wnpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:34.056089 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:34.056060 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0d4e2b8-0f58-4b32-8227-6e3418b8521d-maas-api-tls" pod "d0d4e2b8-0f58-4b32-8227-6e3418b8521d" (UID: "d0d4e2b8-0f58-4b32-8227-6e3418b8521d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:34.156337 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:34.156299 5918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0d4e2b8-0f58-4b32-8227-6e3418b8521d-maas-api-tls\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:34.156337 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:34.156337 5918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0d4e2b8-0f58-4b32-8227-6e3418b8521d-metrics-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:34.156537 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:34.156353 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wnpm\" (UniqueName: \"kubernetes.io/projected/d0d4e2b8-0f58-4b32-8227-6e3418b8521d-kube-api-access-7wnpm\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:34.385477 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:34.385388 5918 generic.go:356] "Generic (PLEG): container finished" podID="d0d4e2b8-0f58-4b32-8227-6e3418b8521d" containerID="513e936f8b524119e699a9f0e5022c2943bf08b12a004a983ae4cfe053846eb3" exitCode=0 Oct 30 22:24:34.385477 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:34.385434 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z" Oct 30 22:24:34.385477 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:34.385452 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z" event={"ID":"d0d4e2b8-0f58-4b32-8227-6e3418b8521d","Type":"ContainerDied","Data":"513e936f8b524119e699a9f0e5022c2943bf08b12a004a983ae4cfe053846eb3"} Oct 30 22:24:34.386034 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:34.385505 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z" event={"ID":"d0d4e2b8-0f58-4b32-8227-6e3418b8521d","Type":"ContainerDied","Data":"7ad995170c16d3d8f7d5210892462a7413a69f5361250a0e09c034dac47786ea"} Oct 30 22:24:34.386034 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:34.385532 5918 scope.go:122] "RemoveContainer" containerID="513e936f8b524119e699a9f0e5022c2943bf08b12a004a983ae4cfe053846eb3" Oct 30 22:24:34.396546 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:34.396529 5918 scope.go:122] "RemoveContainer" containerID="513e936f8b524119e699a9f0e5022c2943bf08b12a004a983ae4cfe053846eb3" Oct 30 22:24:34.396773 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:24:34.396757 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"513e936f8b524119e699a9f0e5022c2943bf08b12a004a983ae4cfe053846eb3\": container with ID starting with 513e936f8b524119e699a9f0e5022c2943bf08b12a004a983ae4cfe053846eb3 not found: ID does not exist" containerID="513e936f8b524119e699a9f0e5022c2943bf08b12a004a983ae4cfe053846eb3" Oct 30 22:24:34.396826 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:34.396782 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"513e936f8b524119e699a9f0e5022c2943bf08b12a004a983ae4cfe053846eb3"} err="failed to get container status \"513e936f8b524119e699a9f0e5022c2943bf08b12a004a983ae4cfe053846eb3\": rpc error: code = NotFound desc = could not find container \"513e936f8b524119e699a9f0e5022c2943bf08b12a004a983ae4cfe053846eb3\": container with ID starting with 513e936f8b524119e699a9f0e5022c2943bf08b12a004a983ae4cfe053846eb3 not found: ID does not exist" Oct 30 22:24:34.559742 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:34.559709 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z"] Oct 30 22:24:34.610338 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:34.610309 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-41e4de-d94b9f59f-rdt5z"] Oct 30 22:24:35.853104 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:35.853073 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0d4e2b8-0f58-4b32-8227-6e3418b8521d" path="/var/lib/kubelet/pods/d0d4e2b8-0f58-4b32-8227-6e3418b8521d/volumes" Oct 30 22:24:38.222353 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:38.222309 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-f2qhs" podUID="b159c891-8623-463e-b65f-fc1f9d2dd2cb" containerName="payload-pre-processing" containerID="cri-o://1972fa778b46df012159e1d23a7e1568db444bab9bea0fed1af2c9bbea2dee62" gracePeriod=25 Oct 30 22:24:38.421454 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:38.421421 5918 generic.go:356] "Generic (PLEG): container finished" podID="b159c891-8623-463e-b65f-fc1f9d2dd2cb" containerID="1972fa778b46df012159e1d23a7e1568db444bab9bea0fed1af2c9bbea2dee62" exitCode=0 Oct 30 22:24:38.421705 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:38.421464 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-f2qhs" event={"ID":"b159c891-8623-463e-b65f-fc1f9d2dd2cb","Type":"ContainerDied","Data":"1972fa778b46df012159e1d23a7e1568db444bab9bea0fed1af2c9bbea2dee62"} Oct 30 22:24:38.487329 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:38.487272 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-f2qhs" Oct 30 22:24:38.618072 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:38.618032 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b159c891-8623-463e-b65f-fc1f9d2dd2cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b159c891-8623-463e-b65f-fc1f9d2dd2cb-plugins-config-volume\") pod \"b159c891-8623-463e-b65f-fc1f9d2dd2cb\" (UID: \"b159c891-8623-463e-b65f-fc1f9d2dd2cb\") " Oct 30 22:24:38.618258 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:38.618110 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b159c891-8623-463e-b65f-fc1f9d2dd2cb-kube-api-access-dwkz5\" (UniqueName: \"kubernetes.io/projected/b159c891-8623-463e-b65f-fc1f9d2dd2cb-kube-api-access-dwkz5\") pod \"b159c891-8623-463e-b65f-fc1f9d2dd2cb\" (UID: \"b159c891-8623-463e-b65f-fc1f9d2dd2cb\") " Oct 30 22:24:38.618495 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:38.618467 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b159c891-8623-463e-b65f-fc1f9d2dd2cb-plugins-config-volume" pod "b159c891-8623-463e-b65f-fc1f9d2dd2cb" (UID: "b159c891-8623-463e-b65f-fc1f9d2dd2cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:38.618862 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:38.618839 5918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b159c891-8623-463e-b65f-fc1f9d2dd2cb-plugins-config-volume\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:38.620700 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:38.620676 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b159c891-8623-463e-b65f-fc1f9d2dd2cb-kube-api-access-dwkz5" pod "b159c891-8623-463e-b65f-fc1f9d2dd2cb" (UID: "b159c891-8623-463e-b65f-fc1f9d2dd2cb"). InnerVolumeSpecName "kube-api-access-dwkz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:38.719739 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:38.719701 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwkz5\" (UniqueName: \"kubernetes.io/projected/b159c891-8623-463e-b65f-fc1f9d2dd2cb-kube-api-access-dwkz5\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:39.430183 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:39.430146 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-f2qhs" event={"ID":"b159c891-8623-463e-b65f-fc1f9d2dd2cb","Type":"ContainerDied","Data":"4f4ea24d68f14a73aaea75e7572cffbd76bdae4e5f4cb46649fa93bd38f37b9b"} Oct 30 22:24:39.430729 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:39.430190 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-f2qhs" Oct 30 22:24:39.430729 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:39.430205 5918 scope.go:122] "RemoveContainer" containerID="1972fa778b46df012159e1d23a7e1568db444bab9bea0fed1af2c9bbea2dee62" Oct 30 22:24:39.600356 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:39.600322 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-f2qhs"] Oct 30 22:24:39.650867 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:39.650834 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7cfd50-754495c479-f2qhs"] Oct 30 22:24:39.853104 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:39.853033 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b159c891-8623-463e-b65f-fc1f9d2dd2cb" path="/var/lib/kubelet/pods/b159c891-8623-463e-b65f-fc1f9d2dd2cb/volumes" Oct 30 22:24:40.411688 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:40.411656 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-81f37a-895cb88f-wwbk6"] Oct 30 22:24:40.411868 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:40.411845 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81f37a-895cb88f-wwbk6" podUID="36861cad-68a4-4d18-b2b6-de583e5dab2b" containerName="maas-api" containerID="cri-o://78abbc7625dd5d27f50f93634900bc2afe45a6a8b26f97b619bf8670312c6ed5" gracePeriod=30 Oct 30 22:24:40.747978 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:40.747946 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81f37a-895cb88f-wwbk6" Oct 30 22:24:40.849631 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:40.849594 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36861cad-68a4-4d18-b2b6-de583e5dab2b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36861cad-68a4-4d18-b2b6-de583e5dab2b-maas-api-tls\") pod \"36861cad-68a4-4d18-b2b6-de583e5dab2b\" (UID: \"36861cad-68a4-4d18-b2b6-de583e5dab2b\") " Oct 30 22:24:40.849827 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:40.849675 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36861cad-68a4-4d18-b2b6-de583e5dab2b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/36861cad-68a4-4d18-b2b6-de583e5dab2b-metrics-certs\") pod \"36861cad-68a4-4d18-b2b6-de583e5dab2b\" (UID: \"36861cad-68a4-4d18-b2b6-de583e5dab2b\") " Oct 30 22:24:40.849827 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:40.849754 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36861cad-68a4-4d18-b2b6-de583e5dab2b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36861cad-68a4-4d18-b2b6-de583e5dab2b-openshift-service-ca\") pod \"36861cad-68a4-4d18-b2b6-de583e5dab2b\" (UID: \"36861cad-68a4-4d18-b2b6-de583e5dab2b\") " Oct 30 22:24:40.849827 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:40.849798 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36861cad-68a4-4d18-b2b6-de583e5dab2b-kube-api-access-f7nts\" (UniqueName: \"kubernetes.io/projected/36861cad-68a4-4d18-b2b6-de583e5dab2b-kube-api-access-f7nts\") pod \"36861cad-68a4-4d18-b2b6-de583e5dab2b\" (UID: \"36861cad-68a4-4d18-b2b6-de583e5dab2b\") " Oct 30 22:24:40.851570 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:40.851545 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36861cad-68a4-4d18-b2b6-de583e5dab2b-openshift-service-ca" pod "36861cad-68a4-4d18-b2b6-de583e5dab2b" (UID: "36861cad-68a4-4d18-b2b6-de583e5dab2b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:40.852942 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:40.852917 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36861cad-68a4-4d18-b2b6-de583e5dab2b-kube-api-access-f7nts" pod "36861cad-68a4-4d18-b2b6-de583e5dab2b" (UID: "36861cad-68a4-4d18-b2b6-de583e5dab2b"). InnerVolumeSpecName "kube-api-access-f7nts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:40.853270 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:40.853245 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36861cad-68a4-4d18-b2b6-de583e5dab2b-metrics-certs" pod "36861cad-68a4-4d18-b2b6-de583e5dab2b" (UID: "36861cad-68a4-4d18-b2b6-de583e5dab2b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:40.853725 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:40.853700 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36861cad-68a4-4d18-b2b6-de583e5dab2b-maas-api-tls" pod "36861cad-68a4-4d18-b2b6-de583e5dab2b" (UID: "36861cad-68a4-4d18-b2b6-de583e5dab2b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:40.952560 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:40.952414 5918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36861cad-68a4-4d18-b2b6-de583e5dab2b-maas-api-tls\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:40.952560 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:40.952450 5918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36861cad-68a4-4d18-b2b6-de583e5dab2b-metrics-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:40.952560 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:40.952464 5918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36861cad-68a4-4d18-b2b6-de583e5dab2b-openshift-service-ca\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:40.952560 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:40.952478 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7nts\" (UniqueName: \"kubernetes.io/projected/36861cad-68a4-4d18-b2b6-de583e5dab2b-kube-api-access-f7nts\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:41.445206 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:41.445176 5918 generic.go:356] "Generic (PLEG): container finished" podID="36861cad-68a4-4d18-b2b6-de583e5dab2b" containerID="78abbc7625dd5d27f50f93634900bc2afe45a6a8b26f97b619bf8670312c6ed5" exitCode=0 Oct 30 22:24:41.445387 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:41.445235 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81f37a-895cb88f-wwbk6" Oct 30 22:24:41.445387 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:41.445263 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81f37a-895cb88f-wwbk6" event={"ID":"36861cad-68a4-4d18-b2b6-de583e5dab2b","Type":"ContainerDied","Data":"78abbc7625dd5d27f50f93634900bc2afe45a6a8b26f97b619bf8670312c6ed5"} Oct 30 22:24:41.445387 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:41.445323 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81f37a-895cb88f-wwbk6" event={"ID":"36861cad-68a4-4d18-b2b6-de583e5dab2b","Type":"ContainerDied","Data":"d6952ca848ba1016231f57bbfdcd03acc028b041672f969a1c31c88e01732537"} Oct 30 22:24:41.445387 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:41.445351 5918 scope.go:122] "RemoveContainer" containerID="78abbc7625dd5d27f50f93634900bc2afe45a6a8b26f97b619bf8670312c6ed5" Oct 30 22:24:41.456257 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:41.456236 5918 scope.go:122] "RemoveContainer" containerID="78abbc7625dd5d27f50f93634900bc2afe45a6a8b26f97b619bf8670312c6ed5" Oct 30 22:24:41.456524 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:24:41.456504 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78abbc7625dd5d27f50f93634900bc2afe45a6a8b26f97b619bf8670312c6ed5\": container with ID starting with 78abbc7625dd5d27f50f93634900bc2afe45a6a8b26f97b619bf8670312c6ed5 not found: ID does not exist" containerID="78abbc7625dd5d27f50f93634900bc2afe45a6a8b26f97b619bf8670312c6ed5" Oct 30 22:24:41.456595 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:41.456532 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78abbc7625dd5d27f50f93634900bc2afe45a6a8b26f97b619bf8670312c6ed5"} err="failed to get container status \"78abbc7625dd5d27f50f93634900bc2afe45a6a8b26f97b619bf8670312c6ed5\": rpc error: code = NotFound desc = could not find container \"78abbc7625dd5d27f50f93634900bc2afe45a6a8b26f97b619bf8670312c6ed5\": container with ID starting with 78abbc7625dd5d27f50f93634900bc2afe45a6a8b26f97b619bf8670312c6ed5 not found: ID does not exist" Oct 30 22:24:41.613610 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:41.613578 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-18edc4-87574c999-vcrzw"] Oct 30 22:24:41.613807 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:41.613787 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18edc4-87574c999-vcrzw" podUID="75b6d2cb-35f2-47fc-a7f5-aa19149ca875" containerName="maas-api" containerID="cri-o://f5c2d7c258005b1baf656bd11ee2033dc4a2d9a9f43e5cbb23f6e7995ea9b4a2" gracePeriod=30 Oct 30 22:24:41.619575 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:41.619548 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-81f37a-895cb88f-wwbk6"] Oct 30 22:24:41.664114 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:41.664088 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-81f37a-895cb88f-wwbk6"] Oct 30 22:24:41.889039 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:41.888987 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36861cad-68a4-4d18-b2b6-de583e5dab2b" path="/var/lib/kubelet/pods/36861cad-68a4-4d18-b2b6-de583e5dab2b/volumes" Oct 30 22:24:41.952092 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:41.952064 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18edc4-87574c999-vcrzw" Oct 30 22:24:42.066075 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:42.065979 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75b6d2cb-35f2-47fc-a7f5-aa19149ca875-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75b6d2cb-35f2-47fc-a7f5-aa19149ca875-maas-api-tls\") pod \"75b6d2cb-35f2-47fc-a7f5-aa19149ca875\" (UID: \"75b6d2cb-35f2-47fc-a7f5-aa19149ca875\") " Oct 30 22:24:42.066298 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:42.066076 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75b6d2cb-35f2-47fc-a7f5-aa19149ca875-kube-api-access-p7p7x\" (UniqueName: \"kubernetes.io/projected/75b6d2cb-35f2-47fc-a7f5-aa19149ca875-kube-api-access-p7p7x\") pod \"75b6d2cb-35f2-47fc-a7f5-aa19149ca875\" (UID: \"75b6d2cb-35f2-47fc-a7f5-aa19149ca875\") " Oct 30 22:24:42.066298 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:42.066187 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75b6d2cb-35f2-47fc-a7f5-aa19149ca875-metrics-certs\" (UniqueName: \"kubernetes.io/secret/75b6d2cb-35f2-47fc-a7f5-aa19149ca875-metrics-certs\") pod \"75b6d2cb-35f2-47fc-a7f5-aa19149ca875\" (UID: \"75b6d2cb-35f2-47fc-a7f5-aa19149ca875\") " Oct 30 22:24:42.066414 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:42.066316 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75b6d2cb-35f2-47fc-a7f5-aa19149ca875-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75b6d2cb-35f2-47fc-a7f5-aa19149ca875-openshift-service-ca\") pod \"75b6d2cb-35f2-47fc-a7f5-aa19149ca875\" (UID: \"75b6d2cb-35f2-47fc-a7f5-aa19149ca875\") " Oct 30 22:24:42.070390 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:42.069792 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75b6d2cb-35f2-47fc-a7f5-aa19149ca875-openshift-service-ca" pod "75b6d2cb-35f2-47fc-a7f5-aa19149ca875" (UID: "75b6d2cb-35f2-47fc-a7f5-aa19149ca875"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:42.079765 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:42.079711 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75b6d2cb-35f2-47fc-a7f5-aa19149ca875-maas-api-tls" pod "75b6d2cb-35f2-47fc-a7f5-aa19149ca875" (UID: "75b6d2cb-35f2-47fc-a7f5-aa19149ca875"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:42.080378 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:42.080322 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b6d2cb-35f2-47fc-a7f5-aa19149ca875-kube-api-access-p7p7x" pod "75b6d2cb-35f2-47fc-a7f5-aa19149ca875" (UID: "75b6d2cb-35f2-47fc-a7f5-aa19149ca875"). InnerVolumeSpecName "kube-api-access-p7p7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:42.083191 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:42.083147 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75b6d2cb-35f2-47fc-a7f5-aa19149ca875-metrics-certs" pod "75b6d2cb-35f2-47fc-a7f5-aa19149ca875" (UID: "75b6d2cb-35f2-47fc-a7f5-aa19149ca875"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:42.168038 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:42.167998 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7p7x\" (UniqueName: \"kubernetes.io/projected/75b6d2cb-35f2-47fc-a7f5-aa19149ca875-kube-api-access-p7p7x\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:42.168038 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:42.168027 5918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75b6d2cb-35f2-47fc-a7f5-aa19149ca875-metrics-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:42.168038 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:42.168037 5918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75b6d2cb-35f2-47fc-a7f5-aa19149ca875-openshift-service-ca\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:42.168038 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:42.168047 5918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75b6d2cb-35f2-47fc-a7f5-aa19149ca875-maas-api-tls\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:42.455314 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:42.455282 5918 generic.go:356] "Generic (PLEG): container finished" podID="75b6d2cb-35f2-47fc-a7f5-aa19149ca875" containerID="f5c2d7c258005b1baf656bd11ee2033dc4a2d9a9f43e5cbb23f6e7995ea9b4a2" exitCode=0 Oct 30 22:24:42.455502 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:42.455336 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18edc4-87574c999-vcrzw" Oct 30 22:24:42.455502 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:42.455380 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18edc4-87574c999-vcrzw" event={"ID":"75b6d2cb-35f2-47fc-a7f5-aa19149ca875","Type":"ContainerDied","Data":"f5c2d7c258005b1baf656bd11ee2033dc4a2d9a9f43e5cbb23f6e7995ea9b4a2"} Oct 30 22:24:42.455502 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:42.455437 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-18edc4-87574c999-vcrzw" event={"ID":"75b6d2cb-35f2-47fc-a7f5-aa19149ca875","Type":"ContainerDied","Data":"bdfe01412757bceedb2c11f903b5ab479eafc7f0d777c5328540531c0d0434e5"} Oct 30 22:24:42.455502 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:42.455466 5918 scope.go:122] "RemoveContainer" containerID="f5c2d7c258005b1baf656bd11ee2033dc4a2d9a9f43e5cbb23f6e7995ea9b4a2" Oct 30 22:24:42.473050 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:42.472093 5918 scope.go:122] "RemoveContainer" containerID="f5c2d7c258005b1baf656bd11ee2033dc4a2d9a9f43e5cbb23f6e7995ea9b4a2" Oct 30 22:24:42.473050 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:24:42.472549 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5c2d7c258005b1baf656bd11ee2033dc4a2d9a9f43e5cbb23f6e7995ea9b4a2\": container with ID starting with f5c2d7c258005b1baf656bd11ee2033dc4a2d9a9f43e5cbb23f6e7995ea9b4a2 not found: ID does not exist" containerID="f5c2d7c258005b1baf656bd11ee2033dc4a2d9a9f43e5cbb23f6e7995ea9b4a2" Oct 30 22:24:42.473050 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:42.472595 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5c2d7c258005b1baf656bd11ee2033dc4a2d9a9f43e5cbb23f6e7995ea9b4a2"} err="failed to get container status \"f5c2d7c258005b1baf656bd11ee2033dc4a2d9a9f43e5cbb23f6e7995ea9b4a2\": rpc error: code = NotFound desc = could not find container \"f5c2d7c258005b1baf656bd11ee2033dc4a2d9a9f43e5cbb23f6e7995ea9b4a2\": container with ID starting with f5c2d7c258005b1baf656bd11ee2033dc4a2d9a9f43e5cbb23f6e7995ea9b4a2 not found: ID does not exist" Oct 30 22:24:42.625607 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:42.625570 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-18edc4-87574c999-vcrzw"] Oct 30 22:24:42.635542 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:42.635516 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-18edc4-87574c999-vcrzw"] Oct 30 22:24:42.868608 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:42.868528 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2"] Oct 30 22:24:42.868749 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:42.868694 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2" podUID="ace7d02d-5b11-4848-844c-e96dfab5e47a" containerName="maas-api" containerID="cri-o://af3b912dd08e0aa55662373e42cecde8ad3f245da2658bd7ac376063ac13a344" gracePeriod=30 Oct 30 22:24:43.170680 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:43.170118 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2" Oct 30 22:24:43.291000 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:43.290899 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ace7d02d-5b11-4848-844c-e96dfab5e47a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ace7d02d-5b11-4848-844c-e96dfab5e47a-maas-api-tls\") pod \"ace7d02d-5b11-4848-844c-e96dfab5e47a\" (UID: \"ace7d02d-5b11-4848-844c-e96dfab5e47a\") " Oct 30 22:24:43.291000 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:43.290949 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ace7d02d-5b11-4848-844c-e96dfab5e47a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ace7d02d-5b11-4848-844c-e96dfab5e47a-openshift-service-ca\") pod \"ace7d02d-5b11-4848-844c-e96dfab5e47a\" (UID: \"ace7d02d-5b11-4848-844c-e96dfab5e47a\") " Oct 30 22:24:43.291292 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:43.291014 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ace7d02d-5b11-4848-844c-e96dfab5e47a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ace7d02d-5b11-4848-844c-e96dfab5e47a-metrics-certs\") pod \"ace7d02d-5b11-4848-844c-e96dfab5e47a\" (UID: \"ace7d02d-5b11-4848-844c-e96dfab5e47a\") " Oct 30 22:24:43.291292 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:43.291047 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ace7d02d-5b11-4848-844c-e96dfab5e47a-kube-api-access-4vqtf\" (UniqueName: \"kubernetes.io/projected/ace7d02d-5b11-4848-844c-e96dfab5e47a-kube-api-access-4vqtf\") pod \"ace7d02d-5b11-4848-844c-e96dfab5e47a\" (UID: \"ace7d02d-5b11-4848-844c-e96dfab5e47a\") " Oct 30 22:24:43.291724 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:43.291676 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ace7d02d-5b11-4848-844c-e96dfab5e47a-openshift-service-ca" pod "ace7d02d-5b11-4848-844c-e96dfab5e47a" (UID: "ace7d02d-5b11-4848-844c-e96dfab5e47a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:43.295370 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:43.291932 5918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ace7d02d-5b11-4848-844c-e96dfab5e47a-openshift-service-ca\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:43.295792 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:43.295675 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ace7d02d-5b11-4848-844c-e96dfab5e47a-kube-api-access-4vqtf" pod "ace7d02d-5b11-4848-844c-e96dfab5e47a" (UID: "ace7d02d-5b11-4848-844c-e96dfab5e47a"). InnerVolumeSpecName "kube-api-access-4vqtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:43.295792 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:43.295762 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ace7d02d-5b11-4848-844c-e96dfab5e47a-metrics-certs" pod "ace7d02d-5b11-4848-844c-e96dfab5e47a" (UID: "ace7d02d-5b11-4848-844c-e96dfab5e47a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:43.296115 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:43.296068 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ace7d02d-5b11-4848-844c-e96dfab5e47a-maas-api-tls" pod "ace7d02d-5b11-4848-844c-e96dfab5e47a" (UID: "ace7d02d-5b11-4848-844c-e96dfab5e47a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:43.393191 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:43.393102 5918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ace7d02d-5b11-4848-844c-e96dfab5e47a-maas-api-tls\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:43.393191 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:43.393134 5918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ace7d02d-5b11-4848-844c-e96dfab5e47a-metrics-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:43.393191 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:43.393148 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vqtf\" (UniqueName: \"kubernetes.io/projected/ace7d02d-5b11-4848-844c-e96dfab5e47a-kube-api-access-4vqtf\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:43.466259 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:43.466205 5918 generic.go:356] "Generic (PLEG): container finished" podID="ace7d02d-5b11-4848-844c-e96dfab5e47a" containerID="af3b912dd08e0aa55662373e42cecde8ad3f245da2658bd7ac376063ac13a344" exitCode=0 Oct 30 22:24:43.466462 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:43.466282 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2" Oct 30 22:24:43.466462 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:43.466318 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2" event={"ID":"ace7d02d-5b11-4848-844c-e96dfab5e47a","Type":"ContainerDied","Data":"af3b912dd08e0aa55662373e42cecde8ad3f245da2658bd7ac376063ac13a344"} Oct 30 22:24:43.466462 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:43.466384 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2" event={"ID":"ace7d02d-5b11-4848-844c-e96dfab5e47a","Type":"ContainerDied","Data":"e4ce65d7ae87b151be93828cc406d1e7868427a46a1ef4556e4edb8108c79663"} Oct 30 22:24:43.466462 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:43.466416 5918 scope.go:122] "RemoveContainer" containerID="af3b912dd08e0aa55662373e42cecde8ad3f245da2658bd7ac376063ac13a344" Oct 30 22:24:43.478929 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:43.478908 5918 scope.go:122] "RemoveContainer" containerID="af3b912dd08e0aa55662373e42cecde8ad3f245da2658bd7ac376063ac13a344" Oct 30 22:24:43.479293 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:24:43.479271 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af3b912dd08e0aa55662373e42cecde8ad3f245da2658bd7ac376063ac13a344\": container with ID starting with af3b912dd08e0aa55662373e42cecde8ad3f245da2658bd7ac376063ac13a344 not found: ID does not exist" containerID="af3b912dd08e0aa55662373e42cecde8ad3f245da2658bd7ac376063ac13a344" Oct 30 22:24:43.479383 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:43.479308 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af3b912dd08e0aa55662373e42cecde8ad3f245da2658bd7ac376063ac13a344"} err="failed to get container status \"af3b912dd08e0aa55662373e42cecde8ad3f245da2658bd7ac376063ac13a344\": rpc error: code = NotFound desc = could not find container \"af3b912dd08e0aa55662373e42cecde8ad3f245da2658bd7ac376063ac13a344\": container with ID starting with af3b912dd08e0aa55662373e42cecde8ad3f245da2658bd7ac376063ac13a344 not found: ID does not exist" Oct 30 22:24:43.646375 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:43.646297 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2"] Oct 30 22:24:43.652609 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:43.652584 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b55a43-54d66b74c6-rtnz2"] Oct 30 22:24:43.853373 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:43.853342 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75b6d2cb-35f2-47fc-a7f5-aa19149ca875" path="/var/lib/kubelet/pods/75b6d2cb-35f2-47fc-a7f5-aa19149ca875/volumes" Oct 30 22:24:43.853840 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:43.853818 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ace7d02d-5b11-4848-844c-e96dfab5e47a" path="/var/lib/kubelet/pods/ace7d02d-5b11-4848-844c-e96dfab5e47a/volumes" Oct 30 22:24:50.000862 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:50.000826 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn"] Oct 30 22:24:50.001368 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:50.001009 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn" podUID="a491d2d9-f2c7-48aa-9091-7df25f1e0b09" containerName="maas-api" containerID="cri-o://512b299b5cdb1a27f8434db87dc30b0926929a46fd9d353a12cb7437496a1eb4" gracePeriod=30 Oct 30 22:24:50.329494 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:50.329461 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn" Oct 30 22:24:50.391352 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:50.391312 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a491d2d9-f2c7-48aa-9091-7df25f1e0b09-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a491d2d9-f2c7-48aa-9091-7df25f1e0b09-metrics-certs\") pod \"a491d2d9-f2c7-48aa-9091-7df25f1e0b09\" (UID: \"a491d2d9-f2c7-48aa-9091-7df25f1e0b09\") " Oct 30 22:24:50.391526 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:50.391373 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a491d2d9-f2c7-48aa-9091-7df25f1e0b09-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a491d2d9-f2c7-48aa-9091-7df25f1e0b09-openshift-service-ca\") pod \"a491d2d9-f2c7-48aa-9091-7df25f1e0b09\" (UID: \"a491d2d9-f2c7-48aa-9091-7df25f1e0b09\") " Oct 30 22:24:50.391526 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:50.391440 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a491d2d9-f2c7-48aa-9091-7df25f1e0b09-kube-api-access-g5dxw\" (UniqueName: \"kubernetes.io/projected/a491d2d9-f2c7-48aa-9091-7df25f1e0b09-kube-api-access-g5dxw\") pod \"a491d2d9-f2c7-48aa-9091-7df25f1e0b09\" (UID: \"a491d2d9-f2c7-48aa-9091-7df25f1e0b09\") " Oct 30 22:24:50.391526 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:50.391503 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a491d2d9-f2c7-48aa-9091-7df25f1e0b09-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a491d2d9-f2c7-48aa-9091-7df25f1e0b09-maas-api-tls\") pod \"a491d2d9-f2c7-48aa-9091-7df25f1e0b09\" (UID: \"a491d2d9-f2c7-48aa-9091-7df25f1e0b09\") " Oct 30 22:24:50.391817 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:50.391779 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a491d2d9-f2c7-48aa-9091-7df25f1e0b09-openshift-service-ca" pod "a491d2d9-f2c7-48aa-9091-7df25f1e0b09" (UID: "a491d2d9-f2c7-48aa-9091-7df25f1e0b09"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:50.392286 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:50.392233 5918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a491d2d9-f2c7-48aa-9091-7df25f1e0b09-openshift-service-ca\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:50.393844 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:50.393817 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a491d2d9-f2c7-48aa-9091-7df25f1e0b09-kube-api-access-g5dxw" pod "a491d2d9-f2c7-48aa-9091-7df25f1e0b09" (UID: "a491d2d9-f2c7-48aa-9091-7df25f1e0b09"). InnerVolumeSpecName "kube-api-access-g5dxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:50.393933 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:50.393865 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a491d2d9-f2c7-48aa-9091-7df25f1e0b09-maas-api-tls" pod "a491d2d9-f2c7-48aa-9091-7df25f1e0b09" (UID: "a491d2d9-f2c7-48aa-9091-7df25f1e0b09"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:50.394026 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:50.394009 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a491d2d9-f2c7-48aa-9091-7df25f1e0b09-metrics-certs" pod "a491d2d9-f2c7-48aa-9091-7df25f1e0b09" (UID: "a491d2d9-f2c7-48aa-9091-7df25f1e0b09"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:50.495241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:50.494296 5918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a491d2d9-f2c7-48aa-9091-7df25f1e0b09-metrics-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:50.495241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:50.494336 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5dxw\" (UniqueName: \"kubernetes.io/projected/a491d2d9-f2c7-48aa-9091-7df25f1e0b09-kube-api-access-g5dxw\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:50.495241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:50.494354 5918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a491d2d9-f2c7-48aa-9091-7df25f1e0b09-maas-api-tls\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:50.516872 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:50.516838 5918 generic.go:356] "Generic (PLEG): container finished" podID="a491d2d9-f2c7-48aa-9091-7df25f1e0b09" containerID="512b299b5cdb1a27f8434db87dc30b0926929a46fd9d353a12cb7437496a1eb4" exitCode=0 Oct 30 22:24:50.517026 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:50.516889 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn" Oct 30 22:24:50.517026 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:50.516894 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn" event={"ID":"a491d2d9-f2c7-48aa-9091-7df25f1e0b09","Type":"ContainerDied","Data":"512b299b5cdb1a27f8434db87dc30b0926929a46fd9d353a12cb7437496a1eb4"} Oct 30 22:24:50.517026 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:50.517019 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn" event={"ID":"a491d2d9-f2c7-48aa-9091-7df25f1e0b09","Type":"ContainerDied","Data":"18a75e688e3e8e65b82e9b25a78f3368230f0bd9d11f8a32948babbff943a872"} Oct 30 22:24:50.517161 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:50.517050 5918 scope.go:122] "RemoveContainer" containerID="512b299b5cdb1a27f8434db87dc30b0926929a46fd9d353a12cb7437496a1eb4" Oct 30 22:24:50.528922 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:50.528901 5918 scope.go:122] "RemoveContainer" containerID="512b299b5cdb1a27f8434db87dc30b0926929a46fd9d353a12cb7437496a1eb4" Oct 30 22:24:50.529314 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:24:50.529285 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"512b299b5cdb1a27f8434db87dc30b0926929a46fd9d353a12cb7437496a1eb4\": container with ID starting with 512b299b5cdb1a27f8434db87dc30b0926929a46fd9d353a12cb7437496a1eb4 not found: ID does not exist" containerID="512b299b5cdb1a27f8434db87dc30b0926929a46fd9d353a12cb7437496a1eb4" Oct 30 22:24:50.529430 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:50.529324 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"512b299b5cdb1a27f8434db87dc30b0926929a46fd9d353a12cb7437496a1eb4"} err="failed to get container status \"512b299b5cdb1a27f8434db87dc30b0926929a46fd9d353a12cb7437496a1eb4\": rpc error: code = NotFound desc = could not find container \"512b299b5cdb1a27f8434db87dc30b0926929a46fd9d353a12cb7437496a1eb4\": container with ID starting with 512b299b5cdb1a27f8434db87dc30b0926929a46fd9d353a12cb7437496a1eb4 not found: ID does not exist" Oct 30 22:24:50.686924 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:50.686895 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn"] Oct 30 22:24:50.696206 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:50.696163 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-gfrmn"] Oct 30 22:24:51.280450 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:24:51.280409 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:24:51.882396 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:51.882348 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a491d2d9-f2c7-48aa-9091-7df25f1e0b09" path="/var/lib/kubelet/pods/a491d2d9-f2c7-48aa-9091-7df25f1e0b09/volumes" Oct 30 22:24:52.991261 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:52.991227 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2"] Oct 30 22:24:52.991756 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:52.991478 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" podUID="147ff712-6d65-4184-9125-9ae9391707dd" containerName="istio-proxy" containerID="cri-o://cbd7a79e8788f2ad487bca9c7fec68a1f58ad7c4181748baa790a094b0a24884" gracePeriod=30 Oct 30 22:24:58.078199 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.078169 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-j52jw"] Oct 30 22:24:58.199767 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.199729 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-8p6dn"] Oct 30 22:24:58.266876 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.266818 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:24:58.402986 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.402951 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/147ff712-6d65-4184-9125-9ae9391707dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/147ff712-6d65-4184-9125-9ae9391707dd-istio-podinfo\") pod \"147ff712-6d65-4184-9125-9ae9391707dd\" (UID: \"147ff712-6d65-4184-9125-9ae9391707dd\") " Oct 30 22:24:58.402986 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.403003 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/147ff712-6d65-4184-9125-9ae9391707dd-kube-api-access-9w7mw\" (UniqueName: \"kubernetes.io/projected/147ff712-6d65-4184-9125-9ae9391707dd-kube-api-access-9w7mw\") pod \"147ff712-6d65-4184-9125-9ae9391707dd\" (UID: \"147ff712-6d65-4184-9125-9ae9391707dd\") " Oct 30 22:24:58.403207 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.403083 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147ff712-6d65-4184-9125-9ae9391707dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/147ff712-6d65-4184-9125-9ae9391707dd-istio-envoy\") pod \"147ff712-6d65-4184-9125-9ae9391707dd\" (UID: \"147ff712-6d65-4184-9125-9ae9391707dd\") " Oct 30 22:24:58.403207 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.403122 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147ff712-6d65-4184-9125-9ae9391707dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/147ff712-6d65-4184-9125-9ae9391707dd-istio-data\") pod \"147ff712-6d65-4184-9125-9ae9391707dd\" (UID: \"147ff712-6d65-4184-9125-9ae9391707dd\") " Oct 30 22:24:58.403207 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.403170 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/147ff712-6d65-4184-9125-9ae9391707dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/147ff712-6d65-4184-9125-9ae9391707dd-istiod-ca-cert\") pod \"147ff712-6d65-4184-9125-9ae9391707dd\" (UID: \"147ff712-6d65-4184-9125-9ae9391707dd\") " Oct 30 22:24:58.403354 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.403289 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147ff712-6d65-4184-9125-9ae9391707dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/147ff712-6d65-4184-9125-9ae9391707dd-workload-certs\") pod \"147ff712-6d65-4184-9125-9ae9391707dd\" (UID: \"147ff712-6d65-4184-9125-9ae9391707dd\") " Oct 30 22:24:58.403354 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.403321 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147ff712-6d65-4184-9125-9ae9391707dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/147ff712-6d65-4184-9125-9ae9391707dd-credential-socket\") pod \"147ff712-6d65-4184-9125-9ae9391707dd\" (UID: \"147ff712-6d65-4184-9125-9ae9391707dd\") " Oct 30 22:24:58.403430 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.403382 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147ff712-6d65-4184-9125-9ae9391707dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/147ff712-6d65-4184-9125-9ae9391707dd-workload-socket\") pod \"147ff712-6d65-4184-9125-9ae9391707dd\" (UID: \"147ff712-6d65-4184-9125-9ae9391707dd\") " Oct 30 22:24:58.403990 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.403526 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/147ff712-6d65-4184-9125-9ae9391707dd-istio-token\" (UniqueName: \"kubernetes.io/projected/147ff712-6d65-4184-9125-9ae9391707dd-istio-token\") pod \"147ff712-6d65-4184-9125-9ae9391707dd\" (UID: \"147ff712-6d65-4184-9125-9ae9391707dd\") " Oct 30 22:24:58.403990 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.403739 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147ff712-6d65-4184-9125-9ae9391707dd-istio-data" pod "147ff712-6d65-4184-9125-9ae9391707dd" (UID: "147ff712-6d65-4184-9125-9ae9391707dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:58.403990 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.403750 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147ff712-6d65-4184-9125-9ae9391707dd-workload-certs" pod "147ff712-6d65-4184-9125-9ae9391707dd" (UID: "147ff712-6d65-4184-9125-9ae9391707dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:58.404263 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.404149 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/147ff712-6d65-4184-9125-9ae9391707dd-istiod-ca-cert" pod "147ff712-6d65-4184-9125-9ae9391707dd" (UID: "147ff712-6d65-4184-9125-9ae9391707dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:58.404735 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.404456 5918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/147ff712-6d65-4184-9125-9ae9391707dd-istio-data\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:58.404735 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.404483 5918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/147ff712-6d65-4184-9125-9ae9391707dd-istiod-ca-cert\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:58.404735 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.404499 5918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/147ff712-6d65-4184-9125-9ae9391707dd-workload-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:58.404735 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.404456 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147ff712-6d65-4184-9125-9ae9391707dd-credential-socket" pod "147ff712-6d65-4184-9125-9ae9391707dd" (UID: "147ff712-6d65-4184-9125-9ae9391707dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:58.405076 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.405032 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147ff712-6d65-4184-9125-9ae9391707dd-workload-socket" pod "147ff712-6d65-4184-9125-9ae9391707dd" (UID: "147ff712-6d65-4184-9125-9ae9391707dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:58.409125 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.408263 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147ff712-6d65-4184-9125-9ae9391707dd-kube-api-access-9w7mw" pod "147ff712-6d65-4184-9125-9ae9391707dd" (UID: "147ff712-6d65-4184-9125-9ae9391707dd"). InnerVolumeSpecName "kube-api-access-9w7mw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:58.409125 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.408821 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147ff712-6d65-4184-9125-9ae9391707dd-istio-envoy" pod "147ff712-6d65-4184-9125-9ae9391707dd" (UID: "147ff712-6d65-4184-9125-9ae9391707dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:58.409125 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.408933 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147ff712-6d65-4184-9125-9ae9391707dd-istio-token" pod "147ff712-6d65-4184-9125-9ae9391707dd" (UID: "147ff712-6d65-4184-9125-9ae9391707dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:58.409398 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.409161 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/147ff712-6d65-4184-9125-9ae9391707dd-istio-podinfo" pod "147ff712-6d65-4184-9125-9ae9391707dd" (UID: "147ff712-6d65-4184-9125-9ae9391707dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:24:58.507861 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.507813 5918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/147ff712-6d65-4184-9125-9ae9391707dd-workload-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:58.507861 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.507850 5918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/147ff712-6d65-4184-9125-9ae9391707dd-istio-token\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:58.507861 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.507865 5918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/147ff712-6d65-4184-9125-9ae9391707dd-istio-podinfo\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:58.508123 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.507879 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w7mw\" (UniqueName: \"kubernetes.io/projected/147ff712-6d65-4184-9125-9ae9391707dd-kube-api-access-9w7mw\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:58.508123 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.507894 5918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/147ff712-6d65-4184-9125-9ae9391707dd-istio-envoy\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:58.508123 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.507906 5918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/147ff712-6d65-4184-9125-9ae9391707dd-credential-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:58.585055 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.584970 5918 generic.go:356] "Generic (PLEG): container finished" podID="147ff712-6d65-4184-9125-9ae9391707dd" containerID="cbd7a79e8788f2ad487bca9c7fec68a1f58ad7c4181748baa790a094b0a24884" exitCode=0 Oct 30 22:24:58.585055 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.585041 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" Oct 30 22:24:58.585369 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.585052 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" event={"ID":"147ff712-6d65-4184-9125-9ae9391707dd","Type":"ContainerDied","Data":"cbd7a79e8788f2ad487bca9c7fec68a1f58ad7c4181748baa790a094b0a24884"} Oct 30 22:24:58.585369 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.585125 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2" event={"ID":"147ff712-6d65-4184-9125-9ae9391707dd","Type":"ContainerDied","Data":"520dcb6ccbcc682e6f2e8fb1d0dc7abdab470317354eccc2a707394ecd6a8693"} Oct 30 22:24:58.585369 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.585150 5918 scope.go:122] "RemoveContainer" containerID="cbd7a79e8788f2ad487bca9c7fec68a1f58ad7c4181748baa790a094b0a24884" Oct 30 22:24:58.596473 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.596453 5918 scope.go:122] "RemoveContainer" containerID="6004971db453dac3f21f59c48449d733113448c0d32f6acb612ea3fe9c5e71df" Oct 30 22:24:58.605775 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.605758 5918 scope.go:122] "RemoveContainer" containerID="cbd7a79e8788f2ad487bca9c7fec68a1f58ad7c4181748baa790a094b0a24884" Oct 30 22:24:58.606073 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:24:58.606042 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbd7a79e8788f2ad487bca9c7fec68a1f58ad7c4181748baa790a094b0a24884\": container with ID starting with cbd7a79e8788f2ad487bca9c7fec68a1f58ad7c4181748baa790a094b0a24884 not found: ID does not exist" containerID="cbd7a79e8788f2ad487bca9c7fec68a1f58ad7c4181748baa790a094b0a24884" Oct 30 22:24:58.606184 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.606080 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbd7a79e8788f2ad487bca9c7fec68a1f58ad7c4181748baa790a094b0a24884"} err="failed to get container status \"cbd7a79e8788f2ad487bca9c7fec68a1f58ad7c4181748baa790a094b0a24884\": rpc error: code = NotFound desc = could not find container \"cbd7a79e8788f2ad487bca9c7fec68a1f58ad7c4181748baa790a094b0a24884\": container with ID starting with cbd7a79e8788f2ad487bca9c7fec68a1f58ad7c4181748baa790a094b0a24884 not found: ID does not exist" Oct 30 22:24:58.606184 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.606100 5918 scope.go:122] "RemoveContainer" containerID="6004971db453dac3f21f59c48449d733113448c0d32f6acb612ea3fe9c5e71df" Oct 30 22:24:58.606461 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:24:58.606442 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6004971db453dac3f21f59c48449d733113448c0d32f6acb612ea3fe9c5e71df\": container with ID starting with 6004971db453dac3f21f59c48449d733113448c0d32f6acb612ea3fe9c5e71df not found: ID does not exist" containerID="6004971db453dac3f21f59c48449d733113448c0d32f6acb612ea3fe9c5e71df" Oct 30 22:24:58.606519 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.606468 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6004971db453dac3f21f59c48449d733113448c0d32f6acb612ea3fe9c5e71df"} err="failed to get container status \"6004971db453dac3f21f59c48449d733113448c0d32f6acb612ea3fe9c5e71df\": rpc error: code = NotFound desc = could not find container \"6004971db453dac3f21f59c48449d733113448c0d32f6acb612ea3fe9c5e71df\": container with ID starting with 6004971db453dac3f21f59c48449d733113448c0d32f6acb612ea3fe9c5e71df not found: ID does not exist" Oct 30 22:24:58.759647 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.759609 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2"] Oct 30 22:24:58.772547 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:58.772513 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-18edc4-openshift-default-7f884f6765-6hqk2"] Oct 30 22:24:59.855303 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:24:59.855267 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="147ff712-6d65-4184-9125-9ae9391707dd" path="/var/lib/kubelet/pods/147ff712-6d65-4184-9125-9ae9391707dd/volumes" Oct 30 22:25:02.795074 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.795033 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-n54lj"] Oct 30 22:25:02.795749 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.795729 5918 cpu_manager.go:414] "containerMap: removing container" podUID="ace7d02d-5b11-4848-844c-e96dfab5e47a" containerName="maas-api" Oct 30 22:25:02.795804 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.795752 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ace7d02d-5b11-4848-844c-e96dfab5e47a" containerName="maas-api" Oct 30 22:25:02.795804 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.795761 5918 cpu_manager.go:414] "containerMap: removing container" podUID="b159c891-8623-463e-b65f-fc1f9d2dd2cb" containerName="payload-pre-processing" Oct 30 22:25:02.795804 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.795767 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b159c891-8623-463e-b65f-fc1f9d2dd2cb" containerName="payload-pre-processing" Oct 30 22:25:02.795804 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.795782 5918 cpu_manager.go:414] "containerMap: removing container" podUID="d0d4e2b8-0f58-4b32-8227-6e3418b8521d" containerName="maas-api" Oct 30 22:25:02.795804 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.795787 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0d4e2b8-0f58-4b32-8227-6e3418b8521d" containerName="maas-api" Oct 30 22:25:02.795804 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.795800 5918 cpu_manager.go:414] "containerMap: removing container" podUID="147ff712-6d65-4184-9125-9ae9391707dd" containerName="istio-proxy" Oct 30 22:25:02.795804 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.795805 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="147ff712-6d65-4184-9125-9ae9391707dd" containerName="istio-proxy" Oct 30 22:25:02.796320 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.795818 5918 cpu_manager.go:414] "containerMap: removing container" podUID="147ff712-6d65-4184-9125-9ae9391707dd" containerName="istio-proxy" Oct 30 22:25:02.796320 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.795823 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="147ff712-6d65-4184-9125-9ae9391707dd" containerName="istio-proxy" Oct 30 22:25:02.796320 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.795835 5918 cpu_manager.go:414] "containerMap: removing container" podUID="bc8c80b0-9111-4da5-908c-fe7f3f99acb8" containerName="payload-pre-processing" Oct 30 22:25:02.796320 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.795839 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc8c80b0-9111-4da5-908c-fe7f3f99acb8" containerName="payload-pre-processing" Oct 30 22:25:02.796320 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.795845 5918 cpu_manager.go:414] "containerMap: removing container" podUID="75b6d2cb-35f2-47fc-a7f5-aa19149ca875" containerName="maas-api" Oct 30 22:25:02.796320 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.795849 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75b6d2cb-35f2-47fc-a7f5-aa19149ca875" containerName="maas-api" Oct 30 22:25:02.796320 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.795867 5918 cpu_manager.go:414] "containerMap: removing container" podUID="f998d536-e995-48fb-9c20-c5cfbabaaa2f" containerName="payload-pre-processing" Oct 30 22:25:02.796320 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.795871 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f998d536-e995-48fb-9c20-c5cfbabaaa2f" containerName="payload-pre-processing" Oct 30 22:25:02.796320 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.795879 5918 cpu_manager.go:414] "containerMap: removing container" podUID="cefcee5b-c92a-4067-8c83-cea422238674" containerName="payload-processing" Oct 30 22:25:02.796320 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.795883 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cefcee5b-c92a-4067-8c83-cea422238674" containerName="payload-processing" Oct 30 22:25:02.796320 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.795894 5918 cpu_manager.go:414] "containerMap: removing container" podUID="36861cad-68a4-4d18-b2b6-de583e5dab2b" containerName="maas-api" Oct 30 22:25:02.796320 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.795898 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36861cad-68a4-4d18-b2b6-de583e5dab2b" containerName="maas-api" Oct 30 22:25:02.796320 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.795906 5918 cpu_manager.go:414] "containerMap: removing container" podUID="a491d2d9-f2c7-48aa-9091-7df25f1e0b09" containerName="maas-api" Oct 30 22:25:02.796320 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.795910 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a491d2d9-f2c7-48aa-9091-7df25f1e0b09" containerName="maas-api" Oct 30 22:25:02.796320 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.795993 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc8c80b0-9111-4da5-908c-fe7f3f99acb8" containerName="payload-pre-processing" Oct 30 22:25:02.796320 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.796001 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="75b6d2cb-35f2-47fc-a7f5-aa19149ca875" containerName="maas-api" Oct 30 22:25:02.796320 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.796010 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f998d536-e995-48fb-9c20-c5cfbabaaa2f" containerName="payload-pre-processing" Oct 30 22:25:02.796320 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.796016 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="cefcee5b-c92a-4067-8c83-cea422238674" containerName="payload-processing" Oct 30 22:25:02.796320 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.796022 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="36861cad-68a4-4d18-b2b6-de583e5dab2b" containerName="maas-api" Oct 30 22:25:02.796320 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.796028 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a491d2d9-f2c7-48aa-9091-7df25f1e0b09" containerName="maas-api" Oct 30 22:25:02.796320 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.796034 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ace7d02d-5b11-4848-844c-e96dfab5e47a" containerName="maas-api" Oct 30 22:25:02.796320 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.796038 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="b159c891-8623-463e-b65f-fc1f9d2dd2cb" containerName="payload-pre-processing" Oct 30 22:25:02.796320 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.796044 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0d4e2b8-0f58-4b32-8227-6e3418b8521d" containerName="maas-api" Oct 30 22:25:02.796320 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.796051 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="147ff712-6d65-4184-9125-9ae9391707dd" containerName="istio-proxy" Oct 30 22:25:02.796320 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.796057 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="147ff712-6d65-4184-9125-9ae9391707dd" containerName="istio-proxy" Oct 30 22:25:02.799396 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.799364 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-n54lj" Oct 30 22:25:02.871788 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.871756 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-n54lj"] Oct 30 22:25:02.910326 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.910295 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-422c1029-dockercfg-bmcgl\"" Oct 30 22:25:02.910541 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.910398 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-422c1029\"" Oct 30 22:25:02.915002 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.914964 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-ngjmd"] Oct 30 22:25:02.920925 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.920900 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-ngjmd" Oct 30 22:25:02.964225 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.963744 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n92m\" (UniqueName: \"kubernetes.io/projected/69e2e901-0602-41fe-9321-12f0e2093e75-kube-api-access-8n92m\") pod \"payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-n54lj\" (UID: \"69e2e901-0602-41fe-9321-12f0e2093e75\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-n54lj" Oct 30 22:25:02.964225 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.963969 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69e2e901-0602-41fe-9321-12f0e2093e75-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-n54lj\" (UID: \"69e2e901-0602-41fe-9321-12f0e2093e75\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-n54lj" Oct 30 22:25:02.999286 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:02.996645 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-ngjmd"] Oct 30 22:25:03.065305 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.065187 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n92m\" (UniqueName: \"kubernetes.io/projected/69e2e901-0602-41fe-9321-12f0e2093e75-kube-api-access-8n92m\") pod \"payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-n54lj\" (UID: \"69e2e901-0602-41fe-9321-12f0e2093e75\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-n54lj" Oct 30 22:25:03.065479 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.065342 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a036216b-8457-4f47-8dc6-1399b627ff7f-plugins-config-volume\") pod \"payload-processing-e2e-mt-422c1029-769464bdcc-ngjmd\" (UID: \"a036216b-8457-4f47-8dc6-1399b627ff7f\") " pod="openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-ngjmd" Oct 30 22:25:03.065479 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.065386 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2wnr\" (UniqueName: \"kubernetes.io/projected/a036216b-8457-4f47-8dc6-1399b627ff7f-kube-api-access-b2wnr\") pod \"payload-processing-e2e-mt-422c1029-769464bdcc-ngjmd\" (UID: \"a036216b-8457-4f47-8dc6-1399b627ff7f\") " pod="openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-ngjmd" Oct 30 22:25:03.065666 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.065519 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69e2e901-0602-41fe-9321-12f0e2093e75-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-n54lj\" (UID: \"69e2e901-0602-41fe-9321-12f0e2093e75\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-n54lj" Oct 30 22:25:03.066146 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.066127 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69e2e901-0602-41fe-9321-12f0e2093e75-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-n54lj\" (UID: \"69e2e901-0602-41fe-9321-12f0e2093e75\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-n54lj" Oct 30 22:25:03.078767 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.078732 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-j52jw" podUID="a821fb11-c9cd-4e19-8710-3d01c7c42955" containerName="payload-processing" containerID="cri-o://b756979dfc3580baac429da6c0dad365f31ed7d8b98499a037589208125e6f79" gracePeriod=25 Oct 30 22:25:03.135159 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.135134 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n92m\" (UniqueName: \"kubernetes.io/projected/69e2e901-0602-41fe-9321-12f0e2093e75-kube-api-access-8n92m\") pod \"payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-n54lj\" (UID: \"69e2e901-0602-41fe-9321-12f0e2093e75\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-n54lj" Oct 30 22:25:03.166473 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.166437 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2wnr\" (UniqueName: \"kubernetes.io/projected/a036216b-8457-4f47-8dc6-1399b627ff7f-kube-api-access-b2wnr\") pod \"payload-processing-e2e-mt-422c1029-769464bdcc-ngjmd\" (UID: \"a036216b-8457-4f47-8dc6-1399b627ff7f\") " pod="openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-ngjmd" Oct 30 22:25:03.166735 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.166718 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a036216b-8457-4f47-8dc6-1399b627ff7f-plugins-config-volume\") pod \"payload-processing-e2e-mt-422c1029-769464bdcc-ngjmd\" (UID: \"a036216b-8457-4f47-8dc6-1399b627ff7f\") " pod="openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-ngjmd" Oct 30 22:25:03.167383 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.167364 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a036216b-8457-4f47-8dc6-1399b627ff7f-plugins-config-volume\") pod \"payload-processing-e2e-mt-422c1029-769464bdcc-ngjmd\" (UID: \"a036216b-8457-4f47-8dc6-1399b627ff7f\") " pod="openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-ngjmd" Oct 30 22:25:03.200880 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.200849 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-8p6dn" podUID="035926cf-f020-4c5b-ad47-7b69e12cf03f" containerName="payload-pre-processing" containerID="cri-o://485b973e528b4d6c5ce7684f83e9be5b7dac16ac34fffcc2a07bbbf7d49b3f08" gracePeriod=25 Oct 30 22:25:03.231481 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.231453 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2wnr\" (UniqueName: \"kubernetes.io/projected/a036216b-8457-4f47-8dc6-1399b627ff7f-kube-api-access-b2wnr\") pod \"payload-processing-e2e-mt-422c1029-769464bdcc-ngjmd\" (UID: \"a036216b-8457-4f47-8dc6-1399b627ff7f\") " pod="openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-ngjmd" Oct 30 22:25:03.253349 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.253322 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-ngjmd" Oct 30 22:25:03.364140 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.364115 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-j52jw" Oct 30 22:25:03.412926 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.412904 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-n54lj" Oct 30 22:25:03.473720 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.473695 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a821fb11-c9cd-4e19-8710-3d01c7c42955-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a821fb11-c9cd-4e19-8710-3d01c7c42955-plugins-config-volume\") pod \"a821fb11-c9cd-4e19-8710-3d01c7c42955\" (UID: \"a821fb11-c9cd-4e19-8710-3d01c7c42955\") " Oct 30 22:25:03.473886 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.473871 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a821fb11-c9cd-4e19-8710-3d01c7c42955-kube-api-access-fdl5x\" (UniqueName: \"kubernetes.io/projected/a821fb11-c9cd-4e19-8710-3d01c7c42955-kube-api-access-fdl5x\") pod \"a821fb11-c9cd-4e19-8710-3d01c7c42955\" (UID: \"a821fb11-c9cd-4e19-8710-3d01c7c42955\") " Oct 30 22:25:03.476147 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.476114 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a821fb11-c9cd-4e19-8710-3d01c7c42955-plugins-config-volume" pod "a821fb11-c9cd-4e19-8710-3d01c7c42955" (UID: "a821fb11-c9cd-4e19-8710-3d01c7c42955"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:03.478543 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.477664 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a821fb11-c9cd-4e19-8710-3d01c7c42955-kube-api-access-fdl5x" pod "a821fb11-c9cd-4e19-8710-3d01c7c42955" (UID: "a821fb11-c9cd-4e19-8710-3d01c7c42955"). InnerVolumeSpecName "kube-api-access-fdl5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:03.500143 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.500117 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-8p6dn" Oct 30 22:25:03.574886 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.574797 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/035926cf-f020-4c5b-ad47-7b69e12cf03f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/035926cf-f020-4c5b-ad47-7b69e12cf03f-plugins-config-volume\") pod \"035926cf-f020-4c5b-ad47-7b69e12cf03f\" (UID: \"035926cf-f020-4c5b-ad47-7b69e12cf03f\") " Oct 30 22:25:03.575062 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.575046 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/035926cf-f020-4c5b-ad47-7b69e12cf03f-kube-api-access-44gm5\" (UniqueName: \"kubernetes.io/projected/035926cf-f020-4c5b-ad47-7b69e12cf03f-kube-api-access-44gm5\") pod \"035926cf-f020-4c5b-ad47-7b69e12cf03f\" (UID: \"035926cf-f020-4c5b-ad47-7b69e12cf03f\") " Oct 30 22:25:03.575158 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.575133 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/035926cf-f020-4c5b-ad47-7b69e12cf03f-plugins-config-volume" pod "035926cf-f020-4c5b-ad47-7b69e12cf03f" (UID: "035926cf-f020-4c5b-ad47-7b69e12cf03f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:03.575516 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.575498 5918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a821fb11-c9cd-4e19-8710-3d01c7c42955-plugins-config-volume\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:03.575609 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.575519 5918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/035926cf-f020-4c5b-ad47-7b69e12cf03f-plugins-config-volume\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:03.575609 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.575534 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdl5x\" (UniqueName: \"kubernetes.io/projected/a821fb11-c9cd-4e19-8710-3d01c7c42955-kube-api-access-fdl5x\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:03.577300 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.577281 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/035926cf-f020-4c5b-ad47-7b69e12cf03f-kube-api-access-44gm5" pod "035926cf-f020-4c5b-ad47-7b69e12cf03f" (UID: "035926cf-f020-4c5b-ad47-7b69e12cf03f"). InnerVolumeSpecName "kube-api-access-44gm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:03.618997 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.618966 5918 generic.go:356] "Generic (PLEG): container finished" podID="035926cf-f020-4c5b-ad47-7b69e12cf03f" containerID="485b973e528b4d6c5ce7684f83e9be5b7dac16ac34fffcc2a07bbbf7d49b3f08" exitCode=0 Oct 30 22:25:03.619166 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.619028 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-8p6dn" Oct 30 22:25:03.619166 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.619066 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-8p6dn" event={"ID":"035926cf-f020-4c5b-ad47-7b69e12cf03f","Type":"ContainerDied","Data":"485b973e528b4d6c5ce7684f83e9be5b7dac16ac34fffcc2a07bbbf7d49b3f08"} Oct 30 22:25:03.619166 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.619126 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-8p6dn" event={"ID":"035926cf-f020-4c5b-ad47-7b69e12cf03f","Type":"ContainerDied","Data":"956353c5f047a3bed5851e386efbfc29f75a864068adfc7a82d728f20f7aff21"} Oct 30 22:25:03.619166 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.619157 5918 scope.go:122] "RemoveContainer" containerID="485b973e528b4d6c5ce7684f83e9be5b7dac16ac34fffcc2a07bbbf7d49b3f08" Oct 30 22:25:03.620887 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.620864 5918 generic.go:356] "Generic (PLEG): container finished" podID="a821fb11-c9cd-4e19-8710-3d01c7c42955" containerID="b756979dfc3580baac429da6c0dad365f31ed7d8b98499a037589208125e6f79" exitCode=0 Oct 30 22:25:03.620972 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.620952 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-j52jw" Oct 30 22:25:03.621036 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.620953 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-j52jw" event={"ID":"a821fb11-c9cd-4e19-8710-3d01c7c42955","Type":"ContainerDied","Data":"b756979dfc3580baac429da6c0dad365f31ed7d8b98499a037589208125e6f79"} Oct 30 22:25:03.621036 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.621019 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-j52jw" event={"ID":"a821fb11-c9cd-4e19-8710-3d01c7c42955","Type":"ContainerDied","Data":"e1f928cf4724ea1eaa25ce1232a602c4001c96f479630266d1557a789a65dddd"} Oct 30 22:25:03.636079 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.636053 5918 scope.go:122] "RemoveContainer" containerID="485b973e528b4d6c5ce7684f83e9be5b7dac16ac34fffcc2a07bbbf7d49b3f08" Oct 30 22:25:03.636617 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:25:03.636587 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"485b973e528b4d6c5ce7684f83e9be5b7dac16ac34fffcc2a07bbbf7d49b3f08\": container with ID starting with 485b973e528b4d6c5ce7684f83e9be5b7dac16ac34fffcc2a07bbbf7d49b3f08 not found: ID does not exist" containerID="485b973e528b4d6c5ce7684f83e9be5b7dac16ac34fffcc2a07bbbf7d49b3f08" Oct 30 22:25:03.636738 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.636664 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"485b973e528b4d6c5ce7684f83e9be5b7dac16ac34fffcc2a07bbbf7d49b3f08"} err="failed to get container status \"485b973e528b4d6c5ce7684f83e9be5b7dac16ac34fffcc2a07bbbf7d49b3f08\": rpc error: code = NotFound desc = could not find container \"485b973e528b4d6c5ce7684f83e9be5b7dac16ac34fffcc2a07bbbf7d49b3f08\": container with ID starting with 485b973e528b4d6c5ce7684f83e9be5b7dac16ac34fffcc2a07bbbf7d49b3f08 not found: ID does not exist" Oct 30 22:25:03.636738 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.636695 5918 scope.go:122] "RemoveContainer" containerID="b756979dfc3580baac429da6c0dad365f31ed7d8b98499a037589208125e6f79" Oct 30 22:25:03.649835 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.649793 5918 scope.go:122] "RemoveContainer" containerID="b756979dfc3580baac429da6c0dad365f31ed7d8b98499a037589208125e6f79" Oct 30 22:25:03.650249 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:25:03.650197 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b756979dfc3580baac429da6c0dad365f31ed7d8b98499a037589208125e6f79\": container with ID starting with b756979dfc3580baac429da6c0dad365f31ed7d8b98499a037589208125e6f79 not found: ID does not exist" containerID="b756979dfc3580baac429da6c0dad365f31ed7d8b98499a037589208125e6f79" Oct 30 22:25:03.650366 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.650243 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b756979dfc3580baac429da6c0dad365f31ed7d8b98499a037589208125e6f79"} err="failed to get container status \"b756979dfc3580baac429da6c0dad365f31ed7d8b98499a037589208125e6f79\": rpc error: code = NotFound desc = could not find container \"b756979dfc3580baac429da6c0dad365f31ed7d8b98499a037589208125e6f79\": container with ID starting with b756979dfc3580baac429da6c0dad365f31ed7d8b98499a037589208125e6f79 not found: ID does not exist" Oct 30 22:25:03.662190 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.662168 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-ngjmd"] Oct 30 22:25:03.662941 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:25:03.662921 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda036216b_8457_4f47_8dc6_1399b627ff7f.slice/crio-f9ac13b06a10212e283c94cd88ab1851b1cbb66fa77804b3bc7454a71a30dafe WatchSource:0}: Error finding container f9ac13b06a10212e283c94cd88ab1851b1cbb66fa77804b3bc7454a71a30dafe: Status 404 returned error can't find the container with id f9ac13b06a10212e283c94cd88ab1851b1cbb66fa77804b3bc7454a71a30dafe Oct 30 22:25:03.677157 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.677131 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44gm5\" (UniqueName: \"kubernetes.io/projected/035926cf-f020-4c5b-ad47-7b69e12cf03f-kube-api-access-44gm5\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:03.803739 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.803715 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-n54lj"] Oct 30 22:25:03.804605 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:25:03.804581 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69e2e901_0602_41fe_9321_12f0e2093e75.slice/crio-c8bbb75766980739639bb552dfde5797f7a6a4f21656d2844ca9c2bd78bd86e8 WatchSource:0}: Error finding container c8bbb75766980739639bb552dfde5797f7a6a4f21656d2844ca9c2bd78bd86e8: Status 404 returned error can't find the container with id c8bbb75766980739639bb552dfde5797f7a6a4f21656d2844ca9c2bd78bd86e8 Oct 30 22:25:03.856590 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.856565 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-j52jw"] Oct 30 22:25:03.859134 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.859110 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-24aa4c-6b5788c7cc-j52jw"] Oct 30 22:25:03.980634 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.980590 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-8p6dn"] Oct 30 22:25:03.983459 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:03.983432 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-24aa4c-78fb8645f5-8p6dn"] Oct 30 22:25:04.632021 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:04.631980 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-n54lj" event={"ID":"69e2e901-0602-41fe-9321-12f0e2093e75","Type":"ContainerStarted","Data":"b075569687c7aef0c284a1577eacca921db6faac1d60dd4e25576d100677518f"} Oct 30 22:25:04.632271 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:04.632161 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-n54lj" Oct 30 22:25:04.632271 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:04.632205 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-n54lj" event={"ID":"69e2e901-0602-41fe-9321-12f0e2093e75","Type":"ContainerStarted","Data":"c8bbb75766980739639bb552dfde5797f7a6a4f21656d2844ca9c2bd78bd86e8"} Oct 30 22:25:04.634615 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:04.634584 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-ngjmd" event={"ID":"a036216b-8457-4f47-8dc6-1399b627ff7f","Type":"ContainerStarted","Data":"246f4c179d8dd99c0e8c96b15363c425f1d512c15d9442650180e8afa92c9cd6"} Oct 30 22:25:04.634742 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:04.634636 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-ngjmd" event={"ID":"a036216b-8457-4f47-8dc6-1399b627ff7f","Type":"ContainerStarted","Data":"f9ac13b06a10212e283c94cd88ab1851b1cbb66fa77804b3bc7454a71a30dafe"} Oct 30 22:25:04.634742 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:04.634725 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-ngjmd" Oct 30 22:25:04.752946 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:04.752896 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-n54lj" podStartSLOduration=2.752883134 podStartE2EDuration="2.752883134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:04.751244067 +0000 UTC m=+2065.021942150" watchObservedRunningTime="2026-10-30 22:25:04.752883134 +0000 UTC m=+2065.023581218" Oct 30 22:25:04.918871 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:04.918766 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-ngjmd" podStartSLOduration=2.918747631 podStartE2EDuration="2.918747631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:04.917584786 +0000 UTC m=+2065.188282870" watchObservedRunningTime="2026-10-30 22:25:04.918747631 +0000 UTC m=+2065.189445716" Oct 30 22:25:05.852321 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:05.852283 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="035926cf-f020-4c5b-ad47-7b69e12cf03f" path="/var/lib/kubelet/pods/035926cf-f020-4c5b-ad47-7b69e12cf03f/volumes" Oct 30 22:25:05.852730 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:05.852716 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a821fb11-c9cd-4e19-8710-3d01c7c42955" path="/var/lib/kubelet/pods/a821fb11-c9cd-4e19-8710-3d01c7c42955/volumes" Oct 30 22:25:09.559480 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:09.559442 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r"] Oct 30 22:25:09.560000 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:09.559699 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" podUID="1181a098-41e1-4d6a-8582-bda860f8bd99" containerName="istio-proxy" containerID="cri-o://f4347d240efac04d580af138555dd862c6e7309d2e629df6e8b8151255541969" gracePeriod=30 Oct 30 22:25:10.035106 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:10.035064 5918 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:25:10.035274 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:10.035131 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" podUID="1181a098-41e1-4d6a-8582-bda860f8bd99" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:25:14.702841 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:14.702810 5918 generic.go:356] "Generic (PLEG): container finished" podID="1181a098-41e1-4d6a-8582-bda860f8bd99" containerID="f4347d240efac04d580af138555dd862c6e7309d2e629df6e8b8151255541969" exitCode=0 Oct 30 22:25:14.703195 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:14.702888 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" event={"ID":"1181a098-41e1-4d6a-8582-bda860f8bd99","Type":"ContainerDied","Data":"f4347d240efac04d580af138555dd862c6e7309d2e629df6e8b8151255541969"} Oct 30 22:25:14.822809 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:14.822782 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" Oct 30 22:25:14.927741 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:14.927716 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1181a098-41e1-4d6a-8582-bda860f8bd99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1181a098-41e1-4d6a-8582-bda860f8bd99-workload-socket\") pod \"1181a098-41e1-4d6a-8582-bda860f8bd99\" (UID: \"1181a098-41e1-4d6a-8582-bda860f8bd99\") " Oct 30 22:25:14.927904 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:14.927755 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1181a098-41e1-4d6a-8582-bda860f8bd99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1181a098-41e1-4d6a-8582-bda860f8bd99-credential-socket\") pod \"1181a098-41e1-4d6a-8582-bda860f8bd99\" (UID: \"1181a098-41e1-4d6a-8582-bda860f8bd99\") " Oct 30 22:25:14.927904 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:14.927786 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1181a098-41e1-4d6a-8582-bda860f8bd99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1181a098-41e1-4d6a-8582-bda860f8bd99-istio-data\") pod \"1181a098-41e1-4d6a-8582-bda860f8bd99\" (UID: \"1181a098-41e1-4d6a-8582-bda860f8bd99\") " Oct 30 22:25:14.927904 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:14.927858 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1181a098-41e1-4d6a-8582-bda860f8bd99-kube-api-access-vq8qd\" (UniqueName: \"kubernetes.io/projected/1181a098-41e1-4d6a-8582-bda860f8bd99-kube-api-access-vq8qd\") pod \"1181a098-41e1-4d6a-8582-bda860f8bd99\" (UID: \"1181a098-41e1-4d6a-8582-bda860f8bd99\") " Oct 30 22:25:14.927904 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:14.927886 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1181a098-41e1-4d6a-8582-bda860f8bd99-istio-token\" (UniqueName: \"kubernetes.io/projected/1181a098-41e1-4d6a-8582-bda860f8bd99-istio-token\") pod \"1181a098-41e1-4d6a-8582-bda860f8bd99\" (UID: \"1181a098-41e1-4d6a-8582-bda860f8bd99\") " Oct 30 22:25:14.928120 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:14.927911 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1181a098-41e1-4d6a-8582-bda860f8bd99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1181a098-41e1-4d6a-8582-bda860f8bd99-istio-envoy\") pod \"1181a098-41e1-4d6a-8582-bda860f8bd99\" (UID: \"1181a098-41e1-4d6a-8582-bda860f8bd99\") " Oct 30 22:25:14.928120 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:14.927929 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1181a098-41e1-4d6a-8582-bda860f8bd99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1181a098-41e1-4d6a-8582-bda860f8bd99-istiod-ca-cert\") pod \"1181a098-41e1-4d6a-8582-bda860f8bd99\" (UID: \"1181a098-41e1-4d6a-8582-bda860f8bd99\") " Oct 30 22:25:14.928120 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:14.928021 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1181a098-41e1-4d6a-8582-bda860f8bd99-workload-socket" pod "1181a098-41e1-4d6a-8582-bda860f8bd99" (UID: "1181a098-41e1-4d6a-8582-bda860f8bd99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.928120 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:14.928028 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1181a098-41e1-4d6a-8582-bda860f8bd99-istio-data" pod "1181a098-41e1-4d6a-8582-bda860f8bd99" (UID: "1181a098-41e1-4d6a-8582-bda860f8bd99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.928120 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:14.928030 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1181a098-41e1-4d6a-8582-bda860f8bd99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1181a098-41e1-4d6a-8582-bda860f8bd99-istio-podinfo\") pod \"1181a098-41e1-4d6a-8582-bda860f8bd99\" (UID: \"1181a098-41e1-4d6a-8582-bda860f8bd99\") " Oct 30 22:25:14.928386 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:14.928125 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1181a098-41e1-4d6a-8582-bda860f8bd99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1181a098-41e1-4d6a-8582-bda860f8bd99-workload-certs\") pod \"1181a098-41e1-4d6a-8582-bda860f8bd99\" (UID: \"1181a098-41e1-4d6a-8582-bda860f8bd99\") " Oct 30 22:25:14.928386 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:14.928134 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1181a098-41e1-4d6a-8582-bda860f8bd99-credential-socket" pod "1181a098-41e1-4d6a-8582-bda860f8bd99" (UID: "1181a098-41e1-4d6a-8582-bda860f8bd99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.928497 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:14.928465 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1181a098-41e1-4d6a-8582-bda860f8bd99-workload-certs" pod "1181a098-41e1-4d6a-8582-bda860f8bd99" (UID: "1181a098-41e1-4d6a-8582-bda860f8bd99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.928553 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:14.928530 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1181a098-41e1-4d6a-8582-bda860f8bd99-istiod-ca-cert" pod "1181a098-41e1-4d6a-8582-bda860f8bd99" (UID: "1181a098-41e1-4d6a-8582-bda860f8bd99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:14.929376 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:14.929352 5918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1181a098-41e1-4d6a-8582-bda860f8bd99-workload-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.929516 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:14.929378 5918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1181a098-41e1-4d6a-8582-bda860f8bd99-workload-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.929516 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:14.929392 5918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1181a098-41e1-4d6a-8582-bda860f8bd99-credential-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.929516 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:14.929406 5918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1181a098-41e1-4d6a-8582-bda860f8bd99-istio-data\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.929516 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:14.929415 5918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1181a098-41e1-4d6a-8582-bda860f8bd99-istiod-ca-cert\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:14.931020 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:14.930997 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1181a098-41e1-4d6a-8582-bda860f8bd99-kube-api-access-vq8qd" pod "1181a098-41e1-4d6a-8582-bda860f8bd99" (UID: "1181a098-41e1-4d6a-8582-bda860f8bd99"). InnerVolumeSpecName "kube-api-access-vq8qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:14.931020 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:14.931005 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1181a098-41e1-4d6a-8582-bda860f8bd99-istio-envoy" pod "1181a098-41e1-4d6a-8582-bda860f8bd99" (UID: "1181a098-41e1-4d6a-8582-bda860f8bd99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:14.931208 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:14.931156 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1181a098-41e1-4d6a-8582-bda860f8bd99-istio-podinfo" pod "1181a098-41e1-4d6a-8582-bda860f8bd99" (UID: "1181a098-41e1-4d6a-8582-bda860f8bd99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:25:14.931313 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:14.931298 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1181a098-41e1-4d6a-8582-bda860f8bd99-istio-token" pod "1181a098-41e1-4d6a-8582-bda860f8bd99" (UID: "1181a098-41e1-4d6a-8582-bda860f8bd99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:15.032913 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:15.032791 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq8qd\" (UniqueName: \"kubernetes.io/projected/1181a098-41e1-4d6a-8582-bda860f8bd99-kube-api-access-vq8qd\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:15.032913 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:15.032827 5918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1181a098-41e1-4d6a-8582-bda860f8bd99-istio-token\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:15.032913 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:15.032857 5918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1181a098-41e1-4d6a-8582-bda860f8bd99-istio-envoy\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:15.032913 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:15.032872 5918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1181a098-41e1-4d6a-8582-bda860f8bd99-istio-podinfo\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:15.645047 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:15.645013 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-n54lj" Oct 30 22:25:15.645247 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:15.645128 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-ngjmd" Oct 30 22:25:15.710895 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:15.710865 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" event={"ID":"1181a098-41e1-4d6a-8582-bda860f8bd99","Type":"ContainerDied","Data":"2870d8d975b270efdb62cb0c626adc9a2b03fd7ee3180b77406a5b04dce7245c"} Oct 30 22:25:15.710895 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:15.710904 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r" Oct 30 22:25:15.711365 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:15.710929 5918 scope.go:122] "RemoveContainer" containerID="f4347d240efac04d580af138555dd862c6e7309d2e629df6e8b8151255541969" Oct 30 22:25:16.026343 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:16.026314 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r"] Oct 30 22:25:16.076582 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:16.076551 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-mp86r"] Oct 30 22:25:17.852721 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:17.852689 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1181a098-41e1-4d6a-8582-bda860f8bd99" path="/var/lib/kubelet/pods/1181a098-41e1-4d6a-8582-bda860f8bd99/volumes" Oct 30 22:25:21.276465 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:25:21.276384 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:25:29.117415 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:29.117379 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm"] Oct 30 22:25:29.117846 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:29.117595 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" podUID="ada89063-30f1-4149-bd94-688bd43ed3dc" containerName="istio-proxy" containerID="cri-o://9f232437abd50869fae45c238591fb400ef615409e862d5f03ce1b1568f23ce5" gracePeriod=30 Oct 30 22:25:32.138087 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:32.138048 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh"] Oct 30 22:25:32.138574 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:32.138276 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh" podUID="1c6eadf8-b382-4d7e-a27b-9f02938df064" containerName="maas-api" containerID="cri-o://c7e3037a3755c892fc83d5f69879bbacbb31e962e8ed504e310e366585951e38" gracePeriod=30 Oct 30 22:25:32.450326 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:32.450297 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh" Oct 30 22:25:32.589744 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:32.589703 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c6eadf8-b382-4d7e-a27b-9f02938df064-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6eadf8-b382-4d7e-a27b-9f02938df064-metrics-certs\") pod \"1c6eadf8-b382-4d7e-a27b-9f02938df064\" (UID: \"1c6eadf8-b382-4d7e-a27b-9f02938df064\") " Oct 30 22:25:32.590571 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:32.590544 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c6eadf8-b382-4d7e-a27b-9f02938df064-kube-api-access-2rp2r\" (UniqueName: \"kubernetes.io/projected/1c6eadf8-b382-4d7e-a27b-9f02938df064-kube-api-access-2rp2r\") pod \"1c6eadf8-b382-4d7e-a27b-9f02938df064\" (UID: \"1c6eadf8-b382-4d7e-a27b-9f02938df064\") " Oct 30 22:25:32.591132 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:32.591112 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c6eadf8-b382-4d7e-a27b-9f02938df064-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c6eadf8-b382-4d7e-a27b-9f02938df064-maas-api-tls\") pod \"1c6eadf8-b382-4d7e-a27b-9f02938df064\" (UID: \"1c6eadf8-b382-4d7e-a27b-9f02938df064\") " Oct 30 22:25:32.591773 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:32.591750 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c6eadf8-b382-4d7e-a27b-9f02938df064-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c6eadf8-b382-4d7e-a27b-9f02938df064-openshift-service-ca\") pod \"1c6eadf8-b382-4d7e-a27b-9f02938df064\" (UID: \"1c6eadf8-b382-4d7e-a27b-9f02938df064\") " Oct 30 22:25:32.592410 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:32.592283 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c6eadf8-b382-4d7e-a27b-9f02938df064-openshift-service-ca" pod "1c6eadf8-b382-4d7e-a27b-9f02938df064" (UID: "1c6eadf8-b382-4d7e-a27b-9f02938df064"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:32.593363 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:32.593341 5918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c6eadf8-b382-4d7e-a27b-9f02938df064-openshift-service-ca\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:32.606586 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:32.606207 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c6eadf8-b382-4d7e-a27b-9f02938df064-kube-api-access-2rp2r" pod "1c6eadf8-b382-4d7e-a27b-9f02938df064" (UID: "1c6eadf8-b382-4d7e-a27b-9f02938df064"). InnerVolumeSpecName "kube-api-access-2rp2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:32.606867 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:32.606826 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c6eadf8-b382-4d7e-a27b-9f02938df064-maas-api-tls" pod "1c6eadf8-b382-4d7e-a27b-9f02938df064" (UID: "1c6eadf8-b382-4d7e-a27b-9f02938df064"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:25:32.611374 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:32.611327 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c6eadf8-b382-4d7e-a27b-9f02938df064-metrics-certs" pod "1c6eadf8-b382-4d7e-a27b-9f02938df064" (UID: "1c6eadf8-b382-4d7e-a27b-9f02938df064"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:25:32.703441 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:32.703406 5918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6eadf8-b382-4d7e-a27b-9f02938df064-metrics-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:32.703441 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:32.703438 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rp2r\" (UniqueName: \"kubernetes.io/projected/1c6eadf8-b382-4d7e-a27b-9f02938df064-kube-api-access-2rp2r\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:32.703668 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:32.703450 5918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c6eadf8-b382-4d7e-a27b-9f02938df064-maas-api-tls\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:32.825300 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:32.825265 5918 generic.go:356] "Generic (PLEG): container finished" podID="1c6eadf8-b382-4d7e-a27b-9f02938df064" containerID="c7e3037a3755c892fc83d5f69879bbacbb31e962e8ed504e310e366585951e38" exitCode=0 Oct 30 22:25:32.825468 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:32.825310 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh" Oct 30 22:25:32.825468 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:32.825342 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh" event={"ID":"1c6eadf8-b382-4d7e-a27b-9f02938df064","Type":"ContainerDied","Data":"c7e3037a3755c892fc83d5f69879bbacbb31e962e8ed504e310e366585951e38"} Oct 30 22:25:32.825468 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:32.825401 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh" event={"ID":"1c6eadf8-b382-4d7e-a27b-9f02938df064","Type":"ContainerDied","Data":"fe76e433c2e0a9f4168b93130129879f1b43551e1e0d0340a510b6e8662163eb"} Oct 30 22:25:32.825468 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:32.825429 5918 scope.go:122] "RemoveContainer" containerID="c7e3037a3755c892fc83d5f69879bbacbb31e962e8ed504e310e366585951e38" Oct 30 22:25:32.836396 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:32.836368 5918 scope.go:122] "RemoveContainer" containerID="c7e3037a3755c892fc83d5f69879bbacbb31e962e8ed504e310e366585951e38" Oct 30 22:25:32.836711 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:25:32.836690 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7e3037a3755c892fc83d5f69879bbacbb31e962e8ed504e310e366585951e38\": container with ID starting with c7e3037a3755c892fc83d5f69879bbacbb31e962e8ed504e310e366585951e38 not found: ID does not exist" containerID="c7e3037a3755c892fc83d5f69879bbacbb31e962e8ed504e310e366585951e38" Oct 30 22:25:32.836794 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:32.836721 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7e3037a3755c892fc83d5f69879bbacbb31e962e8ed504e310e366585951e38"} err="failed to get container status \"c7e3037a3755c892fc83d5f69879bbacbb31e962e8ed504e310e366585951e38\": rpc error: code = NotFound desc = could not find container \"c7e3037a3755c892fc83d5f69879bbacbb31e962e8ed504e310e366585951e38\": container with ID starting with c7e3037a3755c892fc83d5f69879bbacbb31e962e8ed504e310e366585951e38 not found: ID does not exist" Oct 30 22:25:32.998820 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:32.998741 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh"] Oct 30 22:25:33.005584 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:33.005559 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-516ccb-695b54f8b8-qwmjh"] Oct 30 22:25:33.856946 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:33.856901 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c6eadf8-b382-4d7e-a27b-9f02938df064" path="/var/lib/kubelet/pods/1c6eadf8-b382-4d7e-a27b-9f02938df064/volumes" Oct 30 22:25:34.380588 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:34.380566 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" Oct 30 22:25:34.528709 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:34.528679 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ada89063-30f1-4149-bd94-688bd43ed3dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ada89063-30f1-4149-bd94-688bd43ed3dc-istiod-ca-cert\") pod \"ada89063-30f1-4149-bd94-688bd43ed3dc\" (UID: \"ada89063-30f1-4149-bd94-688bd43ed3dc\") " Oct 30 22:25:34.528888 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:34.528741 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ada89063-30f1-4149-bd94-688bd43ed3dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada89063-30f1-4149-bd94-688bd43ed3dc-credential-socket\") pod \"ada89063-30f1-4149-bd94-688bd43ed3dc\" (UID: \"ada89063-30f1-4149-bd94-688bd43ed3dc\") " Oct 30 22:25:34.528888 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:34.528772 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ada89063-30f1-4149-bd94-688bd43ed3dc-istio-token\" (UniqueName: \"kubernetes.io/projected/ada89063-30f1-4149-bd94-688bd43ed3dc-istio-token\") pod \"ada89063-30f1-4149-bd94-688bd43ed3dc\" (UID: \"ada89063-30f1-4149-bd94-688bd43ed3dc\") " Oct 30 22:25:34.528888 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:34.528813 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ada89063-30f1-4149-bd94-688bd43ed3dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ada89063-30f1-4149-bd94-688bd43ed3dc-istio-envoy\") pod \"ada89063-30f1-4149-bd94-688bd43ed3dc\" (UID: \"ada89063-30f1-4149-bd94-688bd43ed3dc\") " Oct 30 22:25:34.528888 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:34.528834 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ada89063-30f1-4149-bd94-688bd43ed3dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada89063-30f1-4149-bd94-688bd43ed3dc-workload-socket\") pod \"ada89063-30f1-4149-bd94-688bd43ed3dc\" (UID: \"ada89063-30f1-4149-bd94-688bd43ed3dc\") " Oct 30 22:25:34.529098 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:34.528909 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ada89063-30f1-4149-bd94-688bd43ed3dc-kube-api-access-8rvmq\" (UniqueName: \"kubernetes.io/projected/ada89063-30f1-4149-bd94-688bd43ed3dc-kube-api-access-8rvmq\") pod \"ada89063-30f1-4149-bd94-688bd43ed3dc\" (UID: \"ada89063-30f1-4149-bd94-688bd43ed3dc\") " Oct 30 22:25:34.529098 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:34.529043 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ada89063-30f1-4149-bd94-688bd43ed3dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ada89063-30f1-4149-bd94-688bd43ed3dc-workload-certs\") pod \"ada89063-30f1-4149-bd94-688bd43ed3dc\" (UID: \"ada89063-30f1-4149-bd94-688bd43ed3dc\") " Oct 30 22:25:34.529098 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:34.529075 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada89063-30f1-4149-bd94-688bd43ed3dc-credential-socket" pod "ada89063-30f1-4149-bd94-688bd43ed3dc" (UID: "ada89063-30f1-4149-bd94-688bd43ed3dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:34.529305 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:34.529078 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ada89063-30f1-4149-bd94-688bd43ed3dc-istiod-ca-cert" pod "ada89063-30f1-4149-bd94-688bd43ed3dc" (UID: "ada89063-30f1-4149-bd94-688bd43ed3dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:34.529305 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:34.529138 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ada89063-30f1-4149-bd94-688bd43ed3dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ada89063-30f1-4149-bd94-688bd43ed3dc-istio-data\") pod \"ada89063-30f1-4149-bd94-688bd43ed3dc\" (UID: \"ada89063-30f1-4149-bd94-688bd43ed3dc\") " Oct 30 22:25:34.529305 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:34.529176 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ada89063-30f1-4149-bd94-688bd43ed3dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ada89063-30f1-4149-bd94-688bd43ed3dc-istio-podinfo\") pod \"ada89063-30f1-4149-bd94-688bd43ed3dc\" (UID: \"ada89063-30f1-4149-bd94-688bd43ed3dc\") " Oct 30 22:25:34.529305 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:34.529202 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada89063-30f1-4149-bd94-688bd43ed3dc-workload-socket" pod "ada89063-30f1-4149-bd94-688bd43ed3dc" (UID: "ada89063-30f1-4149-bd94-688bd43ed3dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:34.529305 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:34.529261 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada89063-30f1-4149-bd94-688bd43ed3dc-workload-certs" pod "ada89063-30f1-4149-bd94-688bd43ed3dc" (UID: "ada89063-30f1-4149-bd94-688bd43ed3dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:34.529932 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:34.529784 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada89063-30f1-4149-bd94-688bd43ed3dc-istio-data" pod "ada89063-30f1-4149-bd94-688bd43ed3dc" (UID: "ada89063-30f1-4149-bd94-688bd43ed3dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:34.529932 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:34.529907 5918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ada89063-30f1-4149-bd94-688bd43ed3dc-workload-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:34.530064 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:34.529955 5918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ada89063-30f1-4149-bd94-688bd43ed3dc-istiod-ca-cert\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:34.530064 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:34.529972 5918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada89063-30f1-4149-bd94-688bd43ed3dc-credential-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:34.530064 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:34.529987 5918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada89063-30f1-4149-bd94-688bd43ed3dc-workload-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:34.531273 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:34.531253 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ada89063-30f1-4149-bd94-688bd43ed3dc-istio-token" pod "ada89063-30f1-4149-bd94-688bd43ed3dc" (UID: "ada89063-30f1-4149-bd94-688bd43ed3dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:34.531385 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:34.531369 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada89063-30f1-4149-bd94-688bd43ed3dc-istio-envoy" pod "ada89063-30f1-4149-bd94-688bd43ed3dc" (UID: "ada89063-30f1-4149-bd94-688bd43ed3dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:34.531850 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:34.531822 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ada89063-30f1-4149-bd94-688bd43ed3dc-istio-podinfo" pod "ada89063-30f1-4149-bd94-688bd43ed3dc" (UID: "ada89063-30f1-4149-bd94-688bd43ed3dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:25:34.531850 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:34.531846 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ada89063-30f1-4149-bd94-688bd43ed3dc-kube-api-access-8rvmq" pod "ada89063-30f1-4149-bd94-688bd43ed3dc" (UID: "ada89063-30f1-4149-bd94-688bd43ed3dc"). InnerVolumeSpecName "kube-api-access-8rvmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:34.631707 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:34.631676 5918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ada89063-30f1-4149-bd94-688bd43ed3dc-istio-data\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:34.631707 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:34.631704 5918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ada89063-30f1-4149-bd94-688bd43ed3dc-istio-podinfo\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:34.631906 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:34.631716 5918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ada89063-30f1-4149-bd94-688bd43ed3dc-istio-token\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:34.631906 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:34.631726 5918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ada89063-30f1-4149-bd94-688bd43ed3dc-istio-envoy\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:34.631906 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:34.631735 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rvmq\" (UniqueName: \"kubernetes.io/projected/ada89063-30f1-4149-bd94-688bd43ed3dc-kube-api-access-8rvmq\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:34.844507 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:34.844425 5918 generic.go:356] "Generic (PLEG): container finished" podID="ada89063-30f1-4149-bd94-688bd43ed3dc" containerID="9f232437abd50869fae45c238591fb400ef615409e862d5f03ce1b1568f23ce5" exitCode=0 Oct 30 22:25:34.844507 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:34.844470 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" Oct 30 22:25:34.844732 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:34.844512 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" event={"ID":"ada89063-30f1-4149-bd94-688bd43ed3dc","Type":"ContainerDied","Data":"9f232437abd50869fae45c238591fb400ef615409e862d5f03ce1b1568f23ce5"} Oct 30 22:25:34.844732 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:34.844566 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm" event={"ID":"ada89063-30f1-4149-bd94-688bd43ed3dc","Type":"ContainerDied","Data":"ab8790630fe6c13a9f0e4a49844bab1f0c48b3541fbe530f349c86c077282406"} Oct 30 22:25:34.844732 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:34.844592 5918 scope.go:122] "RemoveContainer" containerID="9f232437abd50869fae45c238591fb400ef615409e862d5f03ce1b1568f23ce5" Oct 30 22:25:34.855723 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:34.855703 5918 scope.go:122] "RemoveContainer" containerID="9f232437abd50869fae45c238591fb400ef615409e862d5f03ce1b1568f23ce5" Oct 30 22:25:34.856018 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:25:34.855999 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f232437abd50869fae45c238591fb400ef615409e862d5f03ce1b1568f23ce5\": container with ID starting with 9f232437abd50869fae45c238591fb400ef615409e862d5f03ce1b1568f23ce5 not found: ID does not exist" containerID="9f232437abd50869fae45c238591fb400ef615409e862d5f03ce1b1568f23ce5" Oct 30 22:25:34.856080 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:34.856026 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f232437abd50869fae45c238591fb400ef615409e862d5f03ce1b1568f23ce5"} err="failed to get container status \"9f232437abd50869fae45c238591fb400ef615409e862d5f03ce1b1568f23ce5\": rpc error: code = NotFound desc = could not find container \"9f232437abd50869fae45c238591fb400ef615409e862d5f03ce1b1568f23ce5\": container with ID starting with 9f232437abd50869fae45c238591fb400ef615409e862d5f03ce1b1568f23ce5 not found: ID does not exist" Oct 30 22:25:35.020459 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:35.020374 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm"] Oct 30 22:25:35.073666 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:35.073625 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-ws8qm"] Oct 30 22:25:35.852873 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:35.852843 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ada89063-30f1-4149-bd94-688bd43ed3dc" path="/var/lib/kubelet/pods/ada89063-30f1-4149-bd94-688bd43ed3dc/volumes" Oct 30 22:25:51.274784 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:25:51.274740 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:25:53.966119 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:53.966087 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4"] Oct 30 22:25:53.966914 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:53.966893 5918 cpu_manager.go:414] "containerMap: removing container" podUID="1c6eadf8-b382-4d7e-a27b-9f02938df064" containerName="maas-api" Oct 30 22:25:53.967018 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:53.966915 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c6eadf8-b382-4d7e-a27b-9f02938df064" containerName="maas-api" Oct 30 22:25:53.967018 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:53.966927 5918 cpu_manager.go:414] "containerMap: removing container" podUID="a821fb11-c9cd-4e19-8710-3d01c7c42955" containerName="payload-processing" Oct 30 22:25:53.967018 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:53.966936 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a821fb11-c9cd-4e19-8710-3d01c7c42955" containerName="payload-processing" Oct 30 22:25:53.967018 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:53.966956 5918 cpu_manager.go:414] "containerMap: removing container" podUID="ada89063-30f1-4149-bd94-688bd43ed3dc" containerName="istio-proxy" Oct 30 22:25:53.967018 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:53.966964 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ada89063-30f1-4149-bd94-688bd43ed3dc" containerName="istio-proxy" Oct 30 22:25:53.967018 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:53.966995 5918 cpu_manager.go:414] "containerMap: removing container" podUID="1181a098-41e1-4d6a-8582-bda860f8bd99" containerName="istio-proxy" Oct 30 22:25:53.967018 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:53.967002 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1181a098-41e1-4d6a-8582-bda860f8bd99" containerName="istio-proxy" Oct 30 22:25:53.967421 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:53.967028 5918 cpu_manager.go:414] "containerMap: removing container" podUID="035926cf-f020-4c5b-ad47-7b69e12cf03f" containerName="payload-pre-processing" Oct 30 22:25:53.967421 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:53.967036 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="035926cf-f020-4c5b-ad47-7b69e12cf03f" containerName="payload-pre-processing" Oct 30 22:25:53.967421 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:53.967233 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c6eadf8-b382-4d7e-a27b-9f02938df064" containerName="maas-api" Oct 30 22:25:53.967421 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:53.967245 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a821fb11-c9cd-4e19-8710-3d01c7c42955" containerName="payload-processing" Oct 30 22:25:53.967421 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:53.967259 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ada89063-30f1-4149-bd94-688bd43ed3dc" containerName="istio-proxy" Oct 30 22:25:53.967421 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:53.967273 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="1181a098-41e1-4d6a-8582-bda860f8bd99" containerName="istio-proxy" Oct 30 22:25:53.967421 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:53.967285 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="035926cf-f020-4c5b-ad47-7b69e12cf03f" containerName="payload-pre-processing" Oct 30 22:25:53.972131 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:53.972110 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4" Oct 30 22:25:54.029915 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:54.029879 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4"] Oct 30 22:25:54.077152 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:54.077115 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-422c1029-serving-cert\"" Oct 30 22:25:54.092883 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:54.092845 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f39ab9c9-c404-49e7-8273-4e6e50656378-openshift-service-ca\") pod \"maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4\" (UID: \"f39ab9c9-c404-49e7-8273-4e6e50656378\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4" Oct 30 22:25:54.093057 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:54.093030 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f39ab9c9-c404-49e7-8273-4e6e50656378-metrics-certs\") pod \"maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4\" (UID: \"f39ab9c9-c404-49e7-8273-4e6e50656378\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4" Oct 30 22:25:54.093182 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:54.093167 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f39ab9c9-c404-49e7-8273-4e6e50656378-maas-api-tls\") pod \"maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4\" (UID: \"f39ab9c9-c404-49e7-8273-4e6e50656378\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4" Oct 30 22:25:54.093257 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:54.093200 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8xfs\" (UniqueName: \"kubernetes.io/projected/f39ab9c9-c404-49e7-8273-4e6e50656378-kube-api-access-t8xfs\") pod \"maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4\" (UID: \"f39ab9c9-c404-49e7-8273-4e6e50656378\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4" Oct 30 22:25:54.194060 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:54.194029 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f39ab9c9-c404-49e7-8273-4e6e50656378-maas-api-tls\") pod \"maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4\" (UID: \"f39ab9c9-c404-49e7-8273-4e6e50656378\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4" Oct 30 22:25:54.194266 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:54.194074 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8xfs\" (UniqueName: \"kubernetes.io/projected/f39ab9c9-c404-49e7-8273-4e6e50656378-kube-api-access-t8xfs\") pod \"maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4\" (UID: \"f39ab9c9-c404-49e7-8273-4e6e50656378\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4" Oct 30 22:25:54.194266 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:54.194117 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f39ab9c9-c404-49e7-8273-4e6e50656378-openshift-service-ca\") pod \"maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4\" (UID: \"f39ab9c9-c404-49e7-8273-4e6e50656378\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4" Oct 30 22:25:54.194266 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:25:54.194188 5918 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-serving-cert: secret "maas-api-e2e-mt-422c1029-serving-cert" not found Oct 30 22:25:54.194266 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:54.194206 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f39ab9c9-c404-49e7-8273-4e6e50656378-metrics-certs\") pod \"maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4\" (UID: \"f39ab9c9-c404-49e7-8273-4e6e50656378\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4" Oct 30 22:25:54.194514 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:25:54.194305 5918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f39ab9c9-c404-49e7-8273-4e6e50656378-maas-api-tls podName:f39ab9c9-c404-49e7-8273-4e6e50656378 nodeName:}" failed. No retries permitted until 2026-10-30 22:25:54.694285111 +0000 UTC m=+2114.964983191 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f39ab9c9-c404-49e7-8273-4e6e50656378-maas-api-tls") pod "maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4" (UID: "f39ab9c9-c404-49e7-8273-4e6e50656378") : secret "maas-api-e2e-mt-422c1029-serving-cert" not found Oct 30 22:25:54.194855 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:54.194834 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f39ab9c9-c404-49e7-8273-4e6e50656378-openshift-service-ca\") pod \"maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4\" (UID: \"f39ab9c9-c404-49e7-8273-4e6e50656378\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4" Oct 30 22:25:54.196837 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:54.196819 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f39ab9c9-c404-49e7-8273-4e6e50656378-metrics-certs\") pod \"maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4\" (UID: \"f39ab9c9-c404-49e7-8273-4e6e50656378\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4" Oct 30 22:25:54.258868 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:54.258790 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8xfs\" (UniqueName: \"kubernetes.io/projected/f39ab9c9-c404-49e7-8273-4e6e50656378-kube-api-access-t8xfs\") pod \"maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4\" (UID: \"f39ab9c9-c404-49e7-8273-4e6e50656378\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4" Oct 30 22:25:54.705143 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:54.705103 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f39ab9c9-c404-49e7-8273-4e6e50656378-maas-api-tls\") pod \"maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4\" (UID: \"f39ab9c9-c404-49e7-8273-4e6e50656378\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4" Oct 30 22:25:54.708056 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:54.708025 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f39ab9c9-c404-49e7-8273-4e6e50656378-maas-api-tls\") pod \"maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4\" (UID: \"f39ab9c9-c404-49e7-8273-4e6e50656378\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4" Oct 30 22:25:54.886015 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:54.885985 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4" Oct 30 22:25:55.334835 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:55.334807 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4"] Oct 30 22:25:55.337137 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:25:55.337108 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf39ab9c9_c404_49e7_8273_4e6e50656378.slice/crio-0a44a7e6ef47e85595c82584fb921d99d7d70adb7b35c48d2af5208d56646bb1 WatchSource:0}: Error finding container 0a44a7e6ef47e85595c82584fb921d99d7d70adb7b35c48d2af5208d56646bb1: Status 404 returned error can't find the container with id 0a44a7e6ef47e85595c82584fb921d99d7d70adb7b35c48d2af5208d56646bb1 Oct 30 22:25:55.986758 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:55.986716 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4" event={"ID":"f39ab9c9-c404-49e7-8273-4e6e50656378","Type":"ContainerStarted","Data":"0a44a7e6ef47e85595c82584fb921d99d7d70adb7b35c48d2af5208d56646bb1"} Oct 30 22:25:56.994543 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:56.994505 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4" event={"ID":"f39ab9c9-c404-49e7-8273-4e6e50656378","Type":"ContainerStarted","Data":"9f0f9590386a53172cde3857c16f2c6250716bd5d91305686d398dc0655c6b65"} Oct 30 22:25:56.995012 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:56.994602 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4" Oct 30 22:25:57.121463 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:25:57.121404 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4" podStartSLOduration=3.222889315 podStartE2EDuration="4.121386598s" totalImagesPullingTime="898.497283ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:57.12077599 +0000 UTC m=+2117.391474074" watchObservedRunningTime="2026-10-30 22:25:57.121386598 +0000 UTC m=+2117.392084679" Oct 30 22:26:03.006490 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:03.006457 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4" Oct 30 22:26:04.670426 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:04.670385 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw"] Oct 30 22:26:04.670891 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:04.670611 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw" podUID="ae6513bd-92b2-4a98-85ee-a02ab40fe705" containerName="maas-api" containerID="cri-o://9c5baf4c8a21f142df081ceb272f4cdbe19333e85855f5896df0c24ff20f8468" gracePeriod=30 Oct 30 22:26:04.779264 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:04.779155 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z"] Oct 30 22:26:04.779514 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:04.779480 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" podUID="c9afe2de-950a-495b-b234-4bab76e0dadc" containerName="istio-proxy" containerID="cri-o://d7d6602f2e7119147d31da9ee8a6157bc52439fa2e568c387071659a14ae00d1" gracePeriod=30 Oct 30 22:26:05.002517 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:05.002489 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw" Oct 30 22:26:05.047534 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:05.047485 5918 generic.go:356] "Generic (PLEG): container finished" podID="ae6513bd-92b2-4a98-85ee-a02ab40fe705" containerID="9c5baf4c8a21f142df081ceb272f4cdbe19333e85855f5896df0c24ff20f8468" exitCode=0 Oct 30 22:26:05.047716 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:05.047550 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw" Oct 30 22:26:05.047716 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:05.047565 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw" event={"ID":"ae6513bd-92b2-4a98-85ee-a02ab40fe705","Type":"ContainerDied","Data":"9c5baf4c8a21f142df081ceb272f4cdbe19333e85855f5896df0c24ff20f8468"} Oct 30 22:26:05.047716 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:05.047625 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw" event={"ID":"ae6513bd-92b2-4a98-85ee-a02ab40fe705","Type":"ContainerDied","Data":"e26c58431e904b07cf41bfe14df1a1698ef802200ca7941580f431afa32bbcb5"} Oct 30 22:26:05.047716 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:05.047648 5918 scope.go:122] "RemoveContainer" containerID="9c5baf4c8a21f142df081ceb272f4cdbe19333e85855f5896df0c24ff20f8468" Oct 30 22:26:05.048363 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:05.048339 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae6513bd-92b2-4a98-85ee-a02ab40fe705-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae6513bd-92b2-4a98-85ee-a02ab40fe705-maas-api-tls\") pod \"ae6513bd-92b2-4a98-85ee-a02ab40fe705\" (UID: \"ae6513bd-92b2-4a98-85ee-a02ab40fe705\") " Oct 30 22:26:05.048515 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:05.048500 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae6513bd-92b2-4a98-85ee-a02ab40fe705-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae6513bd-92b2-4a98-85ee-a02ab40fe705-metrics-certs\") pod \"ae6513bd-92b2-4a98-85ee-a02ab40fe705\" (UID: \"ae6513bd-92b2-4a98-85ee-a02ab40fe705\") " Oct 30 22:26:05.048587 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:05.048576 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae6513bd-92b2-4a98-85ee-a02ab40fe705-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae6513bd-92b2-4a98-85ee-a02ab40fe705-openshift-service-ca\") pod \"ae6513bd-92b2-4a98-85ee-a02ab40fe705\" (UID: \"ae6513bd-92b2-4a98-85ee-a02ab40fe705\") " Oct 30 22:26:05.048720 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:05.048696 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae6513bd-92b2-4a98-85ee-a02ab40fe705-kube-api-access-g5mbn\" (UniqueName: \"kubernetes.io/projected/ae6513bd-92b2-4a98-85ee-a02ab40fe705-kube-api-access-g5mbn\") pod \"ae6513bd-92b2-4a98-85ee-a02ab40fe705\" (UID: \"ae6513bd-92b2-4a98-85ee-a02ab40fe705\") " Oct 30 22:26:05.049046 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:05.049023 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae6513bd-92b2-4a98-85ee-a02ab40fe705-openshift-service-ca" pod "ae6513bd-92b2-4a98-85ee-a02ab40fe705" (UID: "ae6513bd-92b2-4a98-85ee-a02ab40fe705"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:05.049230 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:05.049169 5918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae6513bd-92b2-4a98-85ee-a02ab40fe705-openshift-service-ca\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:05.051248 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:05.051162 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae6513bd-92b2-4a98-85ee-a02ab40fe705-metrics-certs" pod "ae6513bd-92b2-4a98-85ee-a02ab40fe705" (UID: "ae6513bd-92b2-4a98-85ee-a02ab40fe705"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:26:05.051248 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:05.051202 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae6513bd-92b2-4a98-85ee-a02ab40fe705-maas-api-tls" pod "ae6513bd-92b2-4a98-85ee-a02ab40fe705" (UID: "ae6513bd-92b2-4a98-85ee-a02ab40fe705"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:26:05.051406 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:05.051387 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae6513bd-92b2-4a98-85ee-a02ab40fe705-kube-api-access-g5mbn" pod "ae6513bd-92b2-4a98-85ee-a02ab40fe705" (UID: "ae6513bd-92b2-4a98-85ee-a02ab40fe705"). InnerVolumeSpecName "kube-api-access-g5mbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:05.078067 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:05.078043 5918 scope.go:122] "RemoveContainer" containerID="9c5baf4c8a21f142df081ceb272f4cdbe19333e85855f5896df0c24ff20f8468" Oct 30 22:26:05.078455 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:26:05.078429 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c5baf4c8a21f142df081ceb272f4cdbe19333e85855f5896df0c24ff20f8468\": container with ID starting with 9c5baf4c8a21f142df081ceb272f4cdbe19333e85855f5896df0c24ff20f8468 not found: ID does not exist" containerID="9c5baf4c8a21f142df081ceb272f4cdbe19333e85855f5896df0c24ff20f8468" Oct 30 22:26:05.078561 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:05.078460 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c5baf4c8a21f142df081ceb272f4cdbe19333e85855f5896df0c24ff20f8468"} err="failed to get container status \"9c5baf4c8a21f142df081ceb272f4cdbe19333e85855f5896df0c24ff20f8468\": rpc error: code = NotFound desc = could not find container \"9c5baf4c8a21f142df081ceb272f4cdbe19333e85855f5896df0c24ff20f8468\": container with ID starting with 9c5baf4c8a21f142df081ceb272f4cdbe19333e85855f5896df0c24ff20f8468 not found: ID does not exist" Oct 30 22:26:05.150963 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:05.150925 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5mbn\" (UniqueName: \"kubernetes.io/projected/ae6513bd-92b2-4a98-85ee-a02ab40fe705-kube-api-access-g5mbn\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:05.150963 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:05.150968 5918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae6513bd-92b2-4a98-85ee-a02ab40fe705-maas-api-tls\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:05.151178 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:05.150984 5918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae6513bd-92b2-4a98-85ee-a02ab40fe705-metrics-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:05.473564 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:05.473529 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw"] Oct 30 22:26:05.479424 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:05.479387 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4511b0-57c965cdcb-27rzw"] Oct 30 22:26:05.856000 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:05.855961 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae6513bd-92b2-4a98-85ee-a02ab40fe705" path="/var/lib/kubelet/pods/ae6513bd-92b2-4a98-85ee-a02ab40fe705/volumes" Oct 30 22:26:10.063154 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.063092 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:26:10.084117 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.084090 5918 generic.go:356] "Generic (PLEG): container finished" podID="c9afe2de-950a-495b-b234-4bab76e0dadc" containerID="d7d6602f2e7119147d31da9ee8a6157bc52439fa2e568c387071659a14ae00d1" exitCode=0 Oct 30 22:26:10.084279 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.084141 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" Oct 30 22:26:10.084279 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.084175 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" event={"ID":"c9afe2de-950a-495b-b234-4bab76e0dadc","Type":"ContainerDied","Data":"d7d6602f2e7119147d31da9ee8a6157bc52439fa2e568c387071659a14ae00d1"} Oct 30 22:26:10.084279 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.084246 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z" event={"ID":"c9afe2de-950a-495b-b234-4bab76e0dadc","Type":"ContainerDied","Data":"8e0dc47261f5fc0d5884d4e085e6ca182b323657070ab586d4d090b33a2763b3"} Oct 30 22:26:10.084279 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.084274 5918 scope.go:122] "RemoveContainer" containerID="d7d6602f2e7119147d31da9ee8a6157bc52439fa2e568c387071659a14ae00d1" Oct 30 22:26:10.096890 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.096868 5918 scope.go:122] "RemoveContainer" containerID="b2dd8535cb8c97525862cf1e145697db1fa661b8f3cc41fb51a2e927ceacc805" Oct 30 22:26:10.106278 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.106252 5918 scope.go:122] "RemoveContainer" containerID="d7d6602f2e7119147d31da9ee8a6157bc52439fa2e568c387071659a14ae00d1" Oct 30 22:26:10.106557 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:26:10.106537 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7d6602f2e7119147d31da9ee8a6157bc52439fa2e568c387071659a14ae00d1\": container with ID starting with d7d6602f2e7119147d31da9ee8a6157bc52439fa2e568c387071659a14ae00d1 not found: ID does not exist" containerID="d7d6602f2e7119147d31da9ee8a6157bc52439fa2e568c387071659a14ae00d1" Oct 30 22:26:10.106636 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.106566 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7d6602f2e7119147d31da9ee8a6157bc52439fa2e568c387071659a14ae00d1"} err="failed to get container status \"d7d6602f2e7119147d31da9ee8a6157bc52439fa2e568c387071659a14ae00d1\": rpc error: code = NotFound desc = could not find container \"d7d6602f2e7119147d31da9ee8a6157bc52439fa2e568c387071659a14ae00d1\": container with ID starting with d7d6602f2e7119147d31da9ee8a6157bc52439fa2e568c387071659a14ae00d1 not found: ID does not exist" Oct 30 22:26:10.106636 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.106605 5918 scope.go:122] "RemoveContainer" containerID="b2dd8535cb8c97525862cf1e145697db1fa661b8f3cc41fb51a2e927ceacc805" Oct 30 22:26:10.106870 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:26:10.106852 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2dd8535cb8c97525862cf1e145697db1fa661b8f3cc41fb51a2e927ceacc805\": container with ID starting with b2dd8535cb8c97525862cf1e145697db1fa661b8f3cc41fb51a2e927ceacc805 not found: ID does not exist" containerID="b2dd8535cb8c97525862cf1e145697db1fa661b8f3cc41fb51a2e927ceacc805" Oct 30 22:26:10.106921 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.106876 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2dd8535cb8c97525862cf1e145697db1fa661b8f3cc41fb51a2e927ceacc805"} err="failed to get container status \"b2dd8535cb8c97525862cf1e145697db1fa661b8f3cc41fb51a2e927ceacc805\": rpc error: code = NotFound desc = could not find container \"b2dd8535cb8c97525862cf1e145697db1fa661b8f3cc41fb51a2e927ceacc805\": container with ID starting with b2dd8535cb8c97525862cf1e145697db1fa661b8f3cc41fb51a2e927ceacc805 not found: ID does not exist" Oct 30 22:26:10.116271 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.115000 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9afe2de-950a-495b-b234-4bab76e0dadc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9afe2de-950a-495b-b234-4bab76e0dadc-istio-data\") pod \"c9afe2de-950a-495b-b234-4bab76e0dadc\" (UID: \"c9afe2de-950a-495b-b234-4bab76e0dadc\") " Oct 30 22:26:10.116271 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.115053 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9afe2de-950a-495b-b234-4bab76e0dadc-istio-token\" (UniqueName: \"kubernetes.io/projected/c9afe2de-950a-495b-b234-4bab76e0dadc-istio-token\") pod \"c9afe2de-950a-495b-b234-4bab76e0dadc\" (UID: \"c9afe2de-950a-495b-b234-4bab76e0dadc\") " Oct 30 22:26:10.116271 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.115116 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9afe2de-950a-495b-b234-4bab76e0dadc-kube-api-access-pwdpr\" (UniqueName: \"kubernetes.io/projected/c9afe2de-950a-495b-b234-4bab76e0dadc-kube-api-access-pwdpr\") pod \"c9afe2de-950a-495b-b234-4bab76e0dadc\" (UID: \"c9afe2de-950a-495b-b234-4bab76e0dadc\") " Oct 30 22:26:10.116271 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.115183 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9afe2de-950a-495b-b234-4bab76e0dadc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9afe2de-950a-495b-b234-4bab76e0dadc-credential-socket\") pod \"c9afe2de-950a-495b-b234-4bab76e0dadc\" (UID: \"c9afe2de-950a-495b-b234-4bab76e0dadc\") " Oct 30 22:26:10.116271 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.115292 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9afe2de-950a-495b-b234-4bab76e0dadc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9afe2de-950a-495b-b234-4bab76e0dadc-istio-envoy\") pod \"c9afe2de-950a-495b-b234-4bab76e0dadc\" (UID: \"c9afe2de-950a-495b-b234-4bab76e0dadc\") " Oct 30 22:26:10.116271 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.115358 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9afe2de-950a-495b-b234-4bab76e0dadc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9afe2de-950a-495b-b234-4bab76e0dadc-istio-podinfo\") pod \"c9afe2de-950a-495b-b234-4bab76e0dadc\" (UID: \"c9afe2de-950a-495b-b234-4bab76e0dadc\") " Oct 30 22:26:10.116271 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.115407 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9afe2de-950a-495b-b234-4bab76e0dadc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9afe2de-950a-495b-b234-4bab76e0dadc-workload-socket\") pod \"c9afe2de-950a-495b-b234-4bab76e0dadc\" (UID: \"c9afe2de-950a-495b-b234-4bab76e0dadc\") " Oct 30 22:26:10.116271 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.115482 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9afe2de-950a-495b-b234-4bab76e0dadc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9afe2de-950a-495b-b234-4bab76e0dadc-workload-certs\") pod \"c9afe2de-950a-495b-b234-4bab76e0dadc\" (UID: \"c9afe2de-950a-495b-b234-4bab76e0dadc\") " Oct 30 22:26:10.116271 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.115550 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9afe2de-950a-495b-b234-4bab76e0dadc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9afe2de-950a-495b-b234-4bab76e0dadc-istiod-ca-cert\") pod \"c9afe2de-950a-495b-b234-4bab76e0dadc\" (UID: \"c9afe2de-950a-495b-b234-4bab76e0dadc\") " Oct 30 22:26:10.116795 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.116659 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9afe2de-950a-495b-b234-4bab76e0dadc-istiod-ca-cert" pod "c9afe2de-950a-495b-b234-4bab76e0dadc" (UID: "c9afe2de-950a-495b-b234-4bab76e0dadc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:10.123070 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.116940 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9afe2de-950a-495b-b234-4bab76e0dadc-istio-data" pod "c9afe2de-950a-495b-b234-4bab76e0dadc" (UID: "c9afe2de-950a-495b-b234-4bab76e0dadc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:10.123070 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.121081 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9afe2de-950a-495b-b234-4bab76e0dadc-credential-socket" pod "c9afe2de-950a-495b-b234-4bab76e0dadc" (UID: "c9afe2de-950a-495b-b234-4bab76e0dadc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:10.123070 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.121307 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9afe2de-950a-495b-b234-4bab76e0dadc-workload-socket" pod "c9afe2de-950a-495b-b234-4bab76e0dadc" (UID: "c9afe2de-950a-495b-b234-4bab76e0dadc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:10.123070 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.121527 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9afe2de-950a-495b-b234-4bab76e0dadc-workload-certs" pod "c9afe2de-950a-495b-b234-4bab76e0dadc" (UID: "c9afe2de-950a-495b-b234-4bab76e0dadc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:10.123945 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.123455 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9afe2de-950a-495b-b234-4bab76e0dadc-istio-envoy" pod "c9afe2de-950a-495b-b234-4bab76e0dadc" (UID: "c9afe2de-950a-495b-b234-4bab76e0dadc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:10.123945 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.123744 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9afe2de-950a-495b-b234-4bab76e0dadc-istio-token" pod "c9afe2de-950a-495b-b234-4bab76e0dadc" (UID: "c9afe2de-950a-495b-b234-4bab76e0dadc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:10.125258 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.124369 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9afe2de-950a-495b-b234-4bab76e0dadc-kube-api-access-pwdpr" pod "c9afe2de-950a-495b-b234-4bab76e0dadc" (UID: "c9afe2de-950a-495b-b234-4bab76e0dadc"). InnerVolumeSpecName "kube-api-access-pwdpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:10.126646 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.126621 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9afe2de-950a-495b-b234-4bab76e0dadc-istio-podinfo" pod "c9afe2de-950a-495b-b234-4bab76e0dadc" (UID: "c9afe2de-950a-495b-b234-4bab76e0dadc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:26:10.217285 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.217261 5918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9afe2de-950a-495b-b234-4bab76e0dadc-istio-data\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:10.217285 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.217284 5918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9afe2de-950a-495b-b234-4bab76e0dadc-istio-token\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:10.217437 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.217295 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwdpr\" (UniqueName: \"kubernetes.io/projected/c9afe2de-950a-495b-b234-4bab76e0dadc-kube-api-access-pwdpr\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:10.217437 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.217306 5918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9afe2de-950a-495b-b234-4bab76e0dadc-credential-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:10.217437 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.217315 5918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9afe2de-950a-495b-b234-4bab76e0dadc-istio-envoy\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:10.217437 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.217323 5918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9afe2de-950a-495b-b234-4bab76e0dadc-istio-podinfo\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:10.217437 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.217332 5918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9afe2de-950a-495b-b234-4bab76e0dadc-workload-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:10.217437 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.217340 5918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9afe2de-950a-495b-b234-4bab76e0dadc-workload-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:10.217437 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.217347 5918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9afe2de-950a-495b-b234-4bab76e0dadc-istiod-ca-cert\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:10.517237 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.517179 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z"] Oct 30 22:26:10.523159 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:10.523120 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-24aa4c-openshift-default-6947699cb7-zcw2z"] Oct 30 22:26:11.851849 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:11.851818 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9afe2de-950a-495b-b234-4bab76e0dadc" path="/var/lib/kubelet/pods/c9afe2de-950a-495b-b234-4bab76e0dadc/volumes" Oct 30 22:26:15.743682 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:15.743645 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr"] Oct 30 22:26:15.744160 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:15.743903 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" podUID="0143cf33-1443-4232-8b8e-2f7e448b78c8" containerName="istio-proxy" containerID="cri-o://298f3a6f3559b9399a139a58466a15e2b0f9295da73accc8d9e12e5f55a9b2a3" gracePeriod=30 Oct 30 22:26:21.029891 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.029864 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:26:21.153226 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.153175 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0143cf33-1443-4232-8b8e-2f7e448b78c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0143cf33-1443-4232-8b8e-2f7e448b78c8-istio-podinfo\") pod \"0143cf33-1443-4232-8b8e-2f7e448b78c8\" (UID: \"0143cf33-1443-4232-8b8e-2f7e448b78c8\") " Oct 30 22:26:21.153383 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.153279 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0143cf33-1443-4232-8b8e-2f7e448b78c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0143cf33-1443-4232-8b8e-2f7e448b78c8-workload-certs\") pod \"0143cf33-1443-4232-8b8e-2f7e448b78c8\" (UID: \"0143cf33-1443-4232-8b8e-2f7e448b78c8\") " Oct 30 22:26:21.153383 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.153330 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0143cf33-1443-4232-8b8e-2f7e448b78c8-kube-api-access-lpl8h\" (UniqueName: \"kubernetes.io/projected/0143cf33-1443-4232-8b8e-2f7e448b78c8-kube-api-access-lpl8h\") pod \"0143cf33-1443-4232-8b8e-2f7e448b78c8\" (UID: \"0143cf33-1443-4232-8b8e-2f7e448b78c8\") " Oct 30 22:26:21.153383 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.153380 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0143cf33-1443-4232-8b8e-2f7e448b78c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0143cf33-1443-4232-8b8e-2f7e448b78c8-workload-socket\") pod \"0143cf33-1443-4232-8b8e-2f7e448b78c8\" (UID: \"0143cf33-1443-4232-8b8e-2f7e448b78c8\") " Oct 30 22:26:21.153585 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.153445 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0143cf33-1443-4232-8b8e-2f7e448b78c8-istio-token\" (UniqueName: \"kubernetes.io/projected/0143cf33-1443-4232-8b8e-2f7e448b78c8-istio-token\") pod \"0143cf33-1443-4232-8b8e-2f7e448b78c8\" (UID: \"0143cf33-1443-4232-8b8e-2f7e448b78c8\") " Oct 30 22:26:21.153585 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.153483 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0143cf33-1443-4232-8b8e-2f7e448b78c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0143cf33-1443-4232-8b8e-2f7e448b78c8-istio-data\") pod \"0143cf33-1443-4232-8b8e-2f7e448b78c8\" (UID: \"0143cf33-1443-4232-8b8e-2f7e448b78c8\") " Oct 30 22:26:21.153585 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.153566 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0143cf33-1443-4232-8b8e-2f7e448b78c8-workload-certs" pod "0143cf33-1443-4232-8b8e-2f7e448b78c8" (UID: "0143cf33-1443-4232-8b8e-2f7e448b78c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:21.153696 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.153598 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0143cf33-1443-4232-8b8e-2f7e448b78c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0143cf33-1443-4232-8b8e-2f7e448b78c8-istiod-ca-cert\") pod \"0143cf33-1443-4232-8b8e-2f7e448b78c8\" (UID: \"0143cf33-1443-4232-8b8e-2f7e448b78c8\") " Oct 30 22:26:21.153696 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.153646 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0143cf33-1443-4232-8b8e-2f7e448b78c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0143cf33-1443-4232-8b8e-2f7e448b78c8-istio-envoy\") pod \"0143cf33-1443-4232-8b8e-2f7e448b78c8\" (UID: \"0143cf33-1443-4232-8b8e-2f7e448b78c8\") " Oct 30 22:26:21.153696 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.153675 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0143cf33-1443-4232-8b8e-2f7e448b78c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0143cf33-1443-4232-8b8e-2f7e448b78c8-credential-socket\") pod \"0143cf33-1443-4232-8b8e-2f7e448b78c8\" (UID: \"0143cf33-1443-4232-8b8e-2f7e448b78c8\") " Oct 30 22:26:21.153696 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.153683 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0143cf33-1443-4232-8b8e-2f7e448b78c8-workload-socket" pod "0143cf33-1443-4232-8b8e-2f7e448b78c8" (UID: "0143cf33-1443-4232-8b8e-2f7e448b78c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:21.153908 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.153765 5918 generic.go:356] "Generic (PLEG): container finished" podID="0143cf33-1443-4232-8b8e-2f7e448b78c8" containerID="298f3a6f3559b9399a139a58466a15e2b0f9295da73accc8d9e12e5f55a9b2a3" exitCode=0 Oct 30 22:26:21.153908 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.153786 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0143cf33-1443-4232-8b8e-2f7e448b78c8-istio-data" pod "0143cf33-1443-4232-8b8e-2f7e448b78c8" (UID: "0143cf33-1443-4232-8b8e-2f7e448b78c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:21.153908 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.153832 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" Oct 30 22:26:21.153908 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.153886 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" event={"ID":"0143cf33-1443-4232-8b8e-2f7e448b78c8","Type":"ContainerDied","Data":"298f3a6f3559b9399a139a58466a15e2b0f9295da73accc8d9e12e5f55a9b2a3"} Oct 30 22:26:21.153908 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.153906 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0143cf33-1443-4232-8b8e-2f7e448b78c8-istiod-ca-cert" pod "0143cf33-1443-4232-8b8e-2f7e448b78c8" (UID: "0143cf33-1443-4232-8b8e-2f7e448b78c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:21.154156 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.153937 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr" event={"ID":"0143cf33-1443-4232-8b8e-2f7e448b78c8","Type":"ContainerDied","Data":"0c410ad0f1c205974423da976be1fc12ff3da30181f758179c779e689fce3128"} Oct 30 22:26:21.154156 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.153994 5918 scope.go:122] "RemoveContainer" containerID="298f3a6f3559b9399a139a58466a15e2b0f9295da73accc8d9e12e5f55a9b2a3" Oct 30 22:26:21.154156 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.154029 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0143cf33-1443-4232-8b8e-2f7e448b78c8-credential-socket" pod "0143cf33-1443-4232-8b8e-2f7e448b78c8" (UID: "0143cf33-1443-4232-8b8e-2f7e448b78c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:21.154345 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.154312 5918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0143cf33-1443-4232-8b8e-2f7e448b78c8-workload-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:21.154345 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.154334 5918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0143cf33-1443-4232-8b8e-2f7e448b78c8-istio-data\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:21.154450 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.154348 5918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0143cf33-1443-4232-8b8e-2f7e448b78c8-istiod-ca-cert\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:21.154450 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.154362 5918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0143cf33-1443-4232-8b8e-2f7e448b78c8-credential-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:21.154450 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.154374 5918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0143cf33-1443-4232-8b8e-2f7e448b78c8-workload-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:21.156354 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.156331 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0143cf33-1443-4232-8b8e-2f7e448b78c8-istio-podinfo" pod "0143cf33-1443-4232-8b8e-2f7e448b78c8" (UID: "0143cf33-1443-4232-8b8e-2f7e448b78c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:26:21.156354 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.156333 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0143cf33-1443-4232-8b8e-2f7e448b78c8-istio-token" pod "0143cf33-1443-4232-8b8e-2f7e448b78c8" (UID: "0143cf33-1443-4232-8b8e-2f7e448b78c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:21.156502 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.156353 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0143cf33-1443-4232-8b8e-2f7e448b78c8-kube-api-access-lpl8h" pod "0143cf33-1443-4232-8b8e-2f7e448b78c8" (UID: "0143cf33-1443-4232-8b8e-2f7e448b78c8"). InnerVolumeSpecName "kube-api-access-lpl8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:21.157794 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.157773 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0143cf33-1443-4232-8b8e-2f7e448b78c8-istio-envoy" pod "0143cf33-1443-4232-8b8e-2f7e448b78c8" (UID: "0143cf33-1443-4232-8b8e-2f7e448b78c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:21.187180 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.187146 5918 scope.go:122] "RemoveContainer" containerID="ce65856e6076097a65eae2b4ba21cbd54253e07cd10d5936e91128498208645f" Oct 30 22:26:21.197284 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.197262 5918 scope.go:122] "RemoveContainer" containerID="298f3a6f3559b9399a139a58466a15e2b0f9295da73accc8d9e12e5f55a9b2a3" Oct 30 22:26:21.197651 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:26:21.197622 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"298f3a6f3559b9399a139a58466a15e2b0f9295da73accc8d9e12e5f55a9b2a3\": container with ID starting with 298f3a6f3559b9399a139a58466a15e2b0f9295da73accc8d9e12e5f55a9b2a3 not found: ID does not exist" containerID="298f3a6f3559b9399a139a58466a15e2b0f9295da73accc8d9e12e5f55a9b2a3" Oct 30 22:26:21.197737 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.197666 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"298f3a6f3559b9399a139a58466a15e2b0f9295da73accc8d9e12e5f55a9b2a3"} err="failed to get container status \"298f3a6f3559b9399a139a58466a15e2b0f9295da73accc8d9e12e5f55a9b2a3\": rpc error: code = NotFound desc = could not find container \"298f3a6f3559b9399a139a58466a15e2b0f9295da73accc8d9e12e5f55a9b2a3\": container with ID starting with 298f3a6f3559b9399a139a58466a15e2b0f9295da73accc8d9e12e5f55a9b2a3 not found: ID does not exist" Oct 30 22:26:21.197737 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.197690 5918 scope.go:122] "RemoveContainer" containerID="ce65856e6076097a65eae2b4ba21cbd54253e07cd10d5936e91128498208645f" Oct 30 22:26:21.197966 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:26:21.197944 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce65856e6076097a65eae2b4ba21cbd54253e07cd10d5936e91128498208645f\": container with ID starting with ce65856e6076097a65eae2b4ba21cbd54253e07cd10d5936e91128498208645f not found: ID does not exist" containerID="ce65856e6076097a65eae2b4ba21cbd54253e07cd10d5936e91128498208645f" Oct 30 22:26:21.198056 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.197973 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce65856e6076097a65eae2b4ba21cbd54253e07cd10d5936e91128498208645f"} err="failed to get container status \"ce65856e6076097a65eae2b4ba21cbd54253e07cd10d5936e91128498208645f\": rpc error: code = NotFound desc = could not find container \"ce65856e6076097a65eae2b4ba21cbd54253e07cd10d5936e91128498208645f\": container with ID starting with ce65856e6076097a65eae2b4ba21cbd54253e07cd10d5936e91128498208645f not found: ID does not exist" Oct 30 22:26:21.255502 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.255470 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpl8h\" (UniqueName: \"kubernetes.io/projected/0143cf33-1443-4232-8b8e-2f7e448b78c8-kube-api-access-lpl8h\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:21.255502 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.255497 5918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0143cf33-1443-4232-8b8e-2f7e448b78c8-istio-token\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:21.255502 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.255509 5918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0143cf33-1443-4232-8b8e-2f7e448b78c8-istio-envoy\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:21.255710 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.255518 5918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0143cf33-1443-4232-8b8e-2f7e448b78c8-istio-podinfo\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:21.276149 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:26:21.276063 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:26:21.589932 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.589849 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr"] Oct 30 22:26:21.640578 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.640544 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-4511b0-openshift-default-b5d9b4dc-4z9mr"] Oct 30 22:26:21.853421 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:21.853329 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0143cf33-1443-4232-8b8e-2f7e448b78c8" path="/var/lib/kubelet/pods/0143cf33-1443-4232-8b8e-2f7e448b78c8/volumes" Oct 30 22:26:23.318914 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:23.318885 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj"] Oct 30 22:26:23.319413 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:23.319080 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" podUID="39983d32-e560-445c-8d17-d70986d3f6e7" containerName="istio-proxy" containerID="cri-o://2375a87f710334934f99f193e11c19fff952c5c2d25c27302a9140c032b2e5cc" gracePeriod=30 Oct 30 22:26:24.318538 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:24.318502 5918 patch_prober.go:29] interesting pod/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:26:24.318717 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:24.318559 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" podUID="39983d32-e560-445c-8d17-d70986d3f6e7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:26:28.572344 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:28.572315 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" Oct 30 22:26:28.744091 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:28.744060 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39983d32-e560-445c-8d17-d70986d3f6e7-istio-token\" (UniqueName: \"kubernetes.io/projected/39983d32-e560-445c-8d17-d70986d3f6e7-istio-token\") pod \"39983d32-e560-445c-8d17-d70986d3f6e7\" (UID: \"39983d32-e560-445c-8d17-d70986d3f6e7\") " Oct 30 22:26:28.744305 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:28.744111 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39983d32-e560-445c-8d17-d70986d3f6e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39983d32-e560-445c-8d17-d70986d3f6e7-istiod-ca-cert\") pod \"39983d32-e560-445c-8d17-d70986d3f6e7\" (UID: \"39983d32-e560-445c-8d17-d70986d3f6e7\") " Oct 30 22:26:28.744305 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:28.744159 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39983d32-e560-445c-8d17-d70986d3f6e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39983d32-e560-445c-8d17-d70986d3f6e7-credential-socket\") pod \"39983d32-e560-445c-8d17-d70986d3f6e7\" (UID: \"39983d32-e560-445c-8d17-d70986d3f6e7\") " Oct 30 22:26:28.744305 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:28.744242 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39983d32-e560-445c-8d17-d70986d3f6e7-kube-api-access-qcx7q\" (UniqueName: \"kubernetes.io/projected/39983d32-e560-445c-8d17-d70986d3f6e7-kube-api-access-qcx7q\") pod \"39983d32-e560-445c-8d17-d70986d3f6e7\" (UID: \"39983d32-e560-445c-8d17-d70986d3f6e7\") " Oct 30 22:26:28.744506 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:28.744356 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39983d32-e560-445c-8d17-d70986d3f6e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39983d32-e560-445c-8d17-d70986d3f6e7-workload-socket\") pod \"39983d32-e560-445c-8d17-d70986d3f6e7\" (UID: \"39983d32-e560-445c-8d17-d70986d3f6e7\") " Oct 30 22:26:28.744506 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:28.744462 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39983d32-e560-445c-8d17-d70986d3f6e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39983d32-e560-445c-8d17-d70986d3f6e7-workload-certs\") pod \"39983d32-e560-445c-8d17-d70986d3f6e7\" (UID: \"39983d32-e560-445c-8d17-d70986d3f6e7\") " Oct 30 22:26:28.744506 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:28.744494 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39983d32-e560-445c-8d17-d70986d3f6e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39983d32-e560-445c-8d17-d70986d3f6e7-istio-data\") pod \"39983d32-e560-445c-8d17-d70986d3f6e7\" (UID: \"39983d32-e560-445c-8d17-d70986d3f6e7\") " Oct 30 22:26:28.744662 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:28.744523 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39983d32-e560-445c-8d17-d70986d3f6e7-credential-socket" pod "39983d32-e560-445c-8d17-d70986d3f6e7" (UID: "39983d32-e560-445c-8d17-d70986d3f6e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:28.744662 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:28.744575 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39983d32-e560-445c-8d17-d70986d3f6e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39983d32-e560-445c-8d17-d70986d3f6e7-istio-podinfo\") pod \"39983d32-e560-445c-8d17-d70986d3f6e7\" (UID: \"39983d32-e560-445c-8d17-d70986d3f6e7\") " Oct 30 22:26:28.744662 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:28.744609 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39983d32-e560-445c-8d17-d70986d3f6e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39983d32-e560-445c-8d17-d70986d3f6e7-istio-envoy\") pod \"39983d32-e560-445c-8d17-d70986d3f6e7\" (UID: \"39983d32-e560-445c-8d17-d70986d3f6e7\") " Oct 30 22:26:28.744812 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:28.744646 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39983d32-e560-445c-8d17-d70986d3f6e7-workload-socket" pod "39983d32-e560-445c-8d17-d70986d3f6e7" (UID: "39983d32-e560-445c-8d17-d70986d3f6e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:28.744812 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:28.744719 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39983d32-e560-445c-8d17-d70986d3f6e7-workload-certs" pod "39983d32-e560-445c-8d17-d70986d3f6e7" (UID: "39983d32-e560-445c-8d17-d70986d3f6e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:28.744812 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:28.744733 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39983d32-e560-445c-8d17-d70986d3f6e7-istiod-ca-cert" pod "39983d32-e560-445c-8d17-d70986d3f6e7" (UID: "39983d32-e560-445c-8d17-d70986d3f6e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:28.744991 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:28.744807 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39983d32-e560-445c-8d17-d70986d3f6e7-istio-data" pod "39983d32-e560-445c-8d17-d70986d3f6e7" (UID: "39983d32-e560-445c-8d17-d70986d3f6e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:28.745189 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:28.745133 5918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39983d32-e560-445c-8d17-d70986d3f6e7-workload-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:28.745189 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:28.745166 5918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39983d32-e560-445c-8d17-d70986d3f6e7-istio-data\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:28.745407 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:28.745231 5918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39983d32-e560-445c-8d17-d70986d3f6e7-istiod-ca-cert\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:28.745407 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:28.745290 5918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39983d32-e560-445c-8d17-d70986d3f6e7-credential-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:28.745407 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:28.745388 5918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39983d32-e560-445c-8d17-d70986d3f6e7-workload-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:28.746711 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:28.746686 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39983d32-e560-445c-8d17-d70986d3f6e7-kube-api-access-qcx7q" pod "39983d32-e560-445c-8d17-d70986d3f6e7" (UID: "39983d32-e560-445c-8d17-d70986d3f6e7"). InnerVolumeSpecName "kube-api-access-qcx7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:28.746802 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:28.746789 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39983d32-e560-445c-8d17-d70986d3f6e7-istio-token" pod "39983d32-e560-445c-8d17-d70986d3f6e7" (UID: "39983d32-e560-445c-8d17-d70986d3f6e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:28.747186 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:28.747168 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39983d32-e560-445c-8d17-d70986d3f6e7-istio-envoy" pod "39983d32-e560-445c-8d17-d70986d3f6e7" (UID: "39983d32-e560-445c-8d17-d70986d3f6e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:28.747265 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:28.747181 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39983d32-e560-445c-8d17-d70986d3f6e7-istio-podinfo" pod "39983d32-e560-445c-8d17-d70986d3f6e7" (UID: "39983d32-e560-445c-8d17-d70986d3f6e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:26:28.846899 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:28.846868 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcx7q\" (UniqueName: \"kubernetes.io/projected/39983d32-e560-445c-8d17-d70986d3f6e7-kube-api-access-qcx7q\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:28.846899 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:28.846896 5918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39983d32-e560-445c-8d17-d70986d3f6e7-istio-podinfo\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:28.846899 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:28.846905 5918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39983d32-e560-445c-8d17-d70986d3f6e7-istio-envoy\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:28.847134 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:28.846915 5918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39983d32-e560-445c-8d17-d70986d3f6e7-istio-token\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:29.203986 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:29.203950 5918 generic.go:356] "Generic (PLEG): container finished" podID="39983d32-e560-445c-8d17-d70986d3f6e7" containerID="2375a87f710334934f99f193e11c19fff952c5c2d25c27302a9140c032b2e5cc" exitCode=0 Oct 30 22:26:29.204171 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:29.204004 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" Oct 30 22:26:29.204171 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:29.204031 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" event={"ID":"39983d32-e560-445c-8d17-d70986d3f6e7","Type":"ContainerDied","Data":"2375a87f710334934f99f193e11c19fff952c5c2d25c27302a9140c032b2e5cc"} Oct 30 22:26:29.204171 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:29.204082 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj" event={"ID":"39983d32-e560-445c-8d17-d70986d3f6e7","Type":"ContainerDied","Data":"9d3fefb69c3640c84be323fd0ef109db6c029c3f7af47724a2efc93e0b902dd2"} Oct 30 22:26:29.204171 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:29.204112 5918 scope.go:122] "RemoveContainer" containerID="2375a87f710334934f99f193e11c19fff952c5c2d25c27302a9140c032b2e5cc" Oct 30 22:26:29.215779 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:29.215760 5918 scope.go:122] "RemoveContainer" containerID="2375a87f710334934f99f193e11c19fff952c5c2d25c27302a9140c032b2e5cc" Oct 30 22:26:29.216065 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:26:29.216047 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2375a87f710334934f99f193e11c19fff952c5c2d25c27302a9140c032b2e5cc\": container with ID starting with 2375a87f710334934f99f193e11c19fff952c5c2d25c27302a9140c032b2e5cc not found: ID does not exist" containerID="2375a87f710334934f99f193e11c19fff952c5c2d25c27302a9140c032b2e5cc" Oct 30 22:26:29.216110 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:29.216074 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2375a87f710334934f99f193e11c19fff952c5c2d25c27302a9140c032b2e5cc"} err="failed to get container status \"2375a87f710334934f99f193e11c19fff952c5c2d25c27302a9140c032b2e5cc\": rpc error: code = NotFound desc = could not find container \"2375a87f710334934f99f193e11c19fff952c5c2d25c27302a9140c032b2e5cc\": container with ID starting with 2375a87f710334934f99f193e11c19fff952c5c2d25c27302a9140c032b2e5cc not found: ID does not exist" Oct 30 22:26:29.317046 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:29.317016 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ff338b-6799d5d75c-bhrjf"] Oct 30 22:26:29.388093 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:29.388060 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj"] Oct 30 22:26:29.397009 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:29.396976 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce513-openshift-default-6645cfc867-p8gmj"] Oct 30 22:26:29.852289 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:29.852260 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39983d32-e560-445c-8d17-d70986d3f6e7" path="/var/lib/kubelet/pods/39983d32-e560-445c-8d17-d70986d3f6e7/volumes" Oct 30 22:26:34.317735 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:34.317653 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-ff338b-6799d5d75c-bhrjf" podUID="23bb4082-3d3d-415c-afb6-9f0d547a064f" containerName="payload-processing" containerID="cri-o://07c684721254c83f16bdec17426357d9c96cc233ddd9cf180af1558492f1524f" gracePeriod=25 Oct 30 22:26:34.587931 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:34.587898 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ff338b-6799d5d75c-bhrjf" Oct 30 22:26:34.621854 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:34.621824 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23bb4082-3d3d-415c-afb6-9f0d547a064f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23bb4082-3d3d-415c-afb6-9f0d547a064f-plugins-config-volume\") pod \"23bb4082-3d3d-415c-afb6-9f0d547a064f\" (UID: \"23bb4082-3d3d-415c-afb6-9f0d547a064f\") " Oct 30 22:26:34.622062 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:34.622043 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23bb4082-3d3d-415c-afb6-9f0d547a064f-kube-api-access-kqtcw\" (UniqueName: \"kubernetes.io/projected/23bb4082-3d3d-415c-afb6-9f0d547a064f-kube-api-access-kqtcw\") pod \"23bb4082-3d3d-415c-afb6-9f0d547a064f\" (UID: \"23bb4082-3d3d-415c-afb6-9f0d547a064f\") " Oct 30 22:26:34.622350 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:34.622320 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23bb4082-3d3d-415c-afb6-9f0d547a064f-plugins-config-volume" pod "23bb4082-3d3d-415c-afb6-9f0d547a064f" (UID: "23bb4082-3d3d-415c-afb6-9f0d547a064f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:34.622634 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:34.622609 5918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23bb4082-3d3d-415c-afb6-9f0d547a064f-plugins-config-volume\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:34.624493 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:34.624464 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23bb4082-3d3d-415c-afb6-9f0d547a064f-kube-api-access-kqtcw" pod "23bb4082-3d3d-415c-afb6-9f0d547a064f" (UID: "23bb4082-3d3d-415c-afb6-9f0d547a064f"). InnerVolumeSpecName "kube-api-access-kqtcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:34.724477 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:34.724427 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqtcw\" (UniqueName: \"kubernetes.io/projected/23bb4082-3d3d-415c-afb6-9f0d547a064f-kube-api-access-kqtcw\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:35.243811 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:35.243778 5918 generic.go:356] "Generic (PLEG): container finished" podID="23bb4082-3d3d-415c-afb6-9f0d547a064f" containerID="07c684721254c83f16bdec17426357d9c96cc233ddd9cf180af1558492f1524f" exitCode=0 Oct 30 22:26:35.244006 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:35.243845 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ff338b-6799d5d75c-bhrjf" Oct 30 22:26:35.244006 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:35.243884 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ff338b-6799d5d75c-bhrjf" event={"ID":"23bb4082-3d3d-415c-afb6-9f0d547a064f","Type":"ContainerDied","Data":"07c684721254c83f16bdec17426357d9c96cc233ddd9cf180af1558492f1524f"} Oct 30 22:26:35.244006 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:35.243956 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ff338b-6799d5d75c-bhrjf" event={"ID":"23bb4082-3d3d-415c-afb6-9f0d547a064f","Type":"ContainerDied","Data":"5e353b71a3e407684a91ba8524a1d5f8bd51757fa91dc601bce66303a7aa58ec"} Oct 30 22:26:35.244006 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:35.243990 5918 scope.go:122] "RemoveContainer" containerID="07c684721254c83f16bdec17426357d9c96cc233ddd9cf180af1558492f1524f" Oct 30 22:26:35.257951 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:35.257841 5918 scope.go:122] "RemoveContainer" containerID="07c684721254c83f16bdec17426357d9c96cc233ddd9cf180af1558492f1524f" Oct 30 22:26:35.258363 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:26:35.258336 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07c684721254c83f16bdec17426357d9c96cc233ddd9cf180af1558492f1524f\": container with ID starting with 07c684721254c83f16bdec17426357d9c96cc233ddd9cf180af1558492f1524f not found: ID does not exist" containerID="07c684721254c83f16bdec17426357d9c96cc233ddd9cf180af1558492f1524f" Oct 30 22:26:35.258473 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:35.258371 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07c684721254c83f16bdec17426357d9c96cc233ddd9cf180af1558492f1524f"} err="failed to get container status \"07c684721254c83f16bdec17426357d9c96cc233ddd9cf180af1558492f1524f\": rpc error: code = NotFound desc = could not find container \"07c684721254c83f16bdec17426357d9c96cc233ddd9cf180af1558492f1524f\": container with ID starting with 07c684721254c83f16bdec17426357d9c96cc233ddd9cf180af1558492f1524f not found: ID does not exist" Oct 30 22:26:35.414156 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:35.414121 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ff338b-6799d5d75c-bhrjf"] Oct 30 22:26:35.464516 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:35.464485 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ff338b-6799d5d75c-bhrjf"] Oct 30 22:26:35.851324 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:35.851295 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23bb4082-3d3d-415c-afb6-9f0d547a064f" path="/var/lib/kubelet/pods/23bb4082-3d3d-415c-afb6-9f0d547a064f/volumes" Oct 30 22:26:47.902991 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:47.902955 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp"] Oct 30 22:26:47.903526 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:47.903509 5918 cpu_manager.go:414] "containerMap: removing container" podUID="23bb4082-3d3d-415c-afb6-9f0d547a064f" containerName="payload-processing" Oct 30 22:26:47.903574 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:47.903529 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23bb4082-3d3d-415c-afb6-9f0d547a064f" containerName="payload-processing" Oct 30 22:26:47.903574 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:47.903564 5918 cpu_manager.go:414] "containerMap: removing container" podUID="39983d32-e560-445c-8d17-d70986d3f6e7" containerName="istio-proxy" Oct 30 22:26:47.903574 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:47.903571 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39983d32-e560-445c-8d17-d70986d3f6e7" containerName="istio-proxy" Oct 30 22:26:47.903688 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:47.903587 5918 cpu_manager.go:414] "containerMap: removing container" podUID="0143cf33-1443-4232-8b8e-2f7e448b78c8" containerName="istio-proxy" Oct 30 22:26:47.903688 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:47.903593 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0143cf33-1443-4232-8b8e-2f7e448b78c8" containerName="istio-proxy" Oct 30 22:26:47.903688 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:47.903606 5918 cpu_manager.go:414] "containerMap: removing container" podUID="c9afe2de-950a-495b-b234-4bab76e0dadc" containerName="istio-proxy" Oct 30 22:26:47.903688 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:47.903611 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9afe2de-950a-495b-b234-4bab76e0dadc" containerName="istio-proxy" Oct 30 22:26:47.903688 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:47.903621 5918 cpu_manager.go:414] "containerMap: removing container" podUID="ae6513bd-92b2-4a98-85ee-a02ab40fe705" containerName="maas-api" Oct 30 22:26:47.903688 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:47.903625 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae6513bd-92b2-4a98-85ee-a02ab40fe705" containerName="maas-api" Oct 30 22:26:47.903688 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:47.903634 5918 cpu_manager.go:414] "containerMap: removing container" podUID="0143cf33-1443-4232-8b8e-2f7e448b78c8" containerName="istio-proxy" Oct 30 22:26:47.903688 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:47.903638 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0143cf33-1443-4232-8b8e-2f7e448b78c8" containerName="istio-proxy" Oct 30 22:26:47.903688 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:47.903645 5918 cpu_manager.go:414] "containerMap: removing container" podUID="c9afe2de-950a-495b-b234-4bab76e0dadc" containerName="istio-proxy" Oct 30 22:26:47.903688 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:47.903649 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9afe2de-950a-495b-b234-4bab76e0dadc" containerName="istio-proxy" Oct 30 22:26:47.904015 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:47.903721 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="23bb4082-3d3d-415c-afb6-9f0d547a064f" containerName="payload-processing" Oct 30 22:26:47.904015 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:47.903736 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="39983d32-e560-445c-8d17-d70986d3f6e7" containerName="istio-proxy" Oct 30 22:26:47.904015 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:47.903744 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="0143cf33-1443-4232-8b8e-2f7e448b78c8" containerName="istio-proxy" Oct 30 22:26:47.904015 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:47.903752 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9afe2de-950a-495b-b234-4bab76e0dadc" containerName="istio-proxy" Oct 30 22:26:47.904015 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:47.903758 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae6513bd-92b2-4a98-85ee-a02ab40fe705" containerName="maas-api" Oct 30 22:26:47.904015 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:47.903764 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="0143cf33-1443-4232-8b8e-2f7e448b78c8" containerName="istio-proxy" Oct 30 22:26:47.904015 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:47.903769 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9afe2de-950a-495b-b234-4bab76e0dadc" containerName="istio-proxy" Oct 30 22:26:47.908474 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:47.908457 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp" Oct 30 22:26:47.981130 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:47.981099 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/544a91a1-bd59-4dd2-91a6-33a33bebccb9-metrics-certs\") pod \"maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp\" (UID: \"544a91a1-bd59-4dd2-91a6-33a33bebccb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp" Oct 30 22:26:47.981327 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:47.981164 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/544a91a1-bd59-4dd2-91a6-33a33bebccb9-openshift-service-ca\") pod \"maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp\" (UID: \"544a91a1-bd59-4dd2-91a6-33a33bebccb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp" Oct 30 22:26:47.981327 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:47.981311 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/544a91a1-bd59-4dd2-91a6-33a33bebccb9-maas-api-tls\") pod \"maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp\" (UID: \"544a91a1-bd59-4dd2-91a6-33a33bebccb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp" Oct 30 22:26:47.981498 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:47.981353 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnzqk\" (UniqueName: \"kubernetes.io/projected/544a91a1-bd59-4dd2-91a6-33a33bebccb9-kube-api-access-hnzqk\") pod \"maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp\" (UID: \"544a91a1-bd59-4dd2-91a6-33a33bebccb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp" Oct 30 22:26:48.014652 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:48.014628 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-d67bf9-serving-cert\"" Oct 30 22:26:48.018351 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:48.018329 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp"] Oct 30 22:26:48.082571 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:48.082539 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/544a91a1-bd59-4dd2-91a6-33a33bebccb9-metrics-certs\") pod \"maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp\" (UID: \"544a91a1-bd59-4dd2-91a6-33a33bebccb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp" Oct 30 22:26:48.082757 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:48.082623 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/544a91a1-bd59-4dd2-91a6-33a33bebccb9-openshift-service-ca\") pod \"maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp\" (UID: \"544a91a1-bd59-4dd2-91a6-33a33bebccb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp" Oct 30 22:26:48.082757 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:48.082677 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/544a91a1-bd59-4dd2-91a6-33a33bebccb9-maas-api-tls\") pod \"maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp\" (UID: \"544a91a1-bd59-4dd2-91a6-33a33bebccb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp" Oct 30 22:26:48.082757 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:48.082703 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnzqk\" (UniqueName: \"kubernetes.io/projected/544a91a1-bd59-4dd2-91a6-33a33bebccb9-kube-api-access-hnzqk\") pod \"maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp\" (UID: \"544a91a1-bd59-4dd2-91a6-33a33bebccb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp" Oct 30 22:26:48.082904 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:26:48.082824 5918 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-d67bf9-serving-cert: secret "maas-api-e2e-api-a-d67bf9-serving-cert" not found Oct 30 22:26:48.082904 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:26:48.082896 5918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/544a91a1-bd59-4dd2-91a6-33a33bebccb9-maas-api-tls podName:544a91a1-bd59-4dd2-91a6-33a33bebccb9 nodeName:}" failed. No retries permitted until 2026-10-30 22:26:48.582879973 +0000 UTC m=+2168.853578049 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/544a91a1-bd59-4dd2-91a6-33a33bebccb9-maas-api-tls") pod "maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp" (UID: "544a91a1-bd59-4dd2-91a6-33a33bebccb9") : secret "maas-api-e2e-api-a-d67bf9-serving-cert" not found Oct 30 22:26:48.083383 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:48.083365 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/544a91a1-bd59-4dd2-91a6-33a33bebccb9-openshift-service-ca\") pod \"maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp\" (UID: \"544a91a1-bd59-4dd2-91a6-33a33bebccb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp" Oct 30 22:26:48.085053 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:48.085029 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/544a91a1-bd59-4dd2-91a6-33a33bebccb9-metrics-certs\") pod \"maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp\" (UID: \"544a91a1-bd59-4dd2-91a6-33a33bebccb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp" Oct 30 22:26:48.146257 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:48.146198 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnzqk\" (UniqueName: \"kubernetes.io/projected/544a91a1-bd59-4dd2-91a6-33a33bebccb9-kube-api-access-hnzqk\") pod \"maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp\" (UID: \"544a91a1-bd59-4dd2-91a6-33a33bebccb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp" Oct 30 22:26:48.590181 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:48.590145 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/544a91a1-bd59-4dd2-91a6-33a33bebccb9-maas-api-tls\") pod \"maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp\" (UID: \"544a91a1-bd59-4dd2-91a6-33a33bebccb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp" Oct 30 22:26:48.592677 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:48.592656 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/544a91a1-bd59-4dd2-91a6-33a33bebccb9-maas-api-tls\") pod \"maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp\" (UID: \"544a91a1-bd59-4dd2-91a6-33a33bebccb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp" Oct 30 22:26:48.825005 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:48.824966 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp" Oct 30 22:26:49.203024 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:49.202994 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp"] Oct 30 22:26:49.205073 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:26:49.205045 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod544a91a1_bd59_4dd2_91a6_33a33bebccb9.slice/crio-417bd8f3d51266f15fbbeafa06af5a513689b90fa118581f3ae7075a0d697e19 WatchSource:0}: Error finding container 417bd8f3d51266f15fbbeafa06af5a513689b90fa118581f3ae7075a0d697e19: Status 404 returned error can't find the container with id 417bd8f3d51266f15fbbeafa06af5a513689b90fa118581f3ae7075a0d697e19 Oct 30 22:26:49.332104 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:49.332071 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp" event={"ID":"544a91a1-bd59-4dd2-91a6-33a33bebccb9","Type":"ContainerStarted","Data":"417bd8f3d51266f15fbbeafa06af5a513689b90fa118581f3ae7075a0d697e19"} Oct 30 22:26:50.342208 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:50.342109 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp" event={"ID":"544a91a1-bd59-4dd2-91a6-33a33bebccb9","Type":"ContainerStarted","Data":"9fa36a2a0e417b4f96bf5ea5bae9b4b1341b5c8e8489df028808aff6f658befa"} Oct 30 22:26:50.342208 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:50.342195 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp" Oct 30 22:26:50.475363 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:50.475295 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp" podStartSLOduration=2.640106947 podStartE2EDuration="3.47527696s" totalImagesPullingTime="835.170013ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:50.471684651 +0000 UTC m=+2170.742382747" watchObservedRunningTime="2026-10-30 22:26:50.47527696 +0000 UTC m=+2170.745975046" Oct 30 22:26:51.276392 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:26:51.276355 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:26:55.131747 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:55.131711 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v"] Oct 30 22:26:55.135818 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:55.135798 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" Oct 30 22:26:55.174374 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:55.174343 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0018134-1144-4076-95ca-7383b46a2985-istio-token\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-7h99v\" (UID: \"e0018134-1144-4076-95ca-7383b46a2985\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" Oct 30 22:26:55.174500 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:55.174455 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfvv4\" (UniqueName: \"kubernetes.io/projected/e0018134-1144-4076-95ca-7383b46a2985-kube-api-access-xfvv4\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-7h99v\" (UID: \"e0018134-1144-4076-95ca-7383b46a2985\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" Oct 30 22:26:55.174500 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:55.174480 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0018134-1144-4076-95ca-7383b46a2985-istio-podinfo\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-7h99v\" (UID: \"e0018134-1144-4076-95ca-7383b46a2985\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" Oct 30 22:26:55.174577 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:55.174510 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0018134-1144-4076-95ca-7383b46a2985-istio-data\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-7h99v\" (UID: \"e0018134-1144-4076-95ca-7383b46a2985\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" Oct 30 22:26:55.174642 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:55.174621 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0018134-1144-4076-95ca-7383b46a2985-credential-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-7h99v\" (UID: \"e0018134-1144-4076-95ca-7383b46a2985\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" Oct 30 22:26:55.174683 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:55.174661 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0018134-1144-4076-95ca-7383b46a2985-istiod-ca-cert\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-7h99v\" (UID: \"e0018134-1144-4076-95ca-7383b46a2985\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" Oct 30 22:26:55.174716 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:55.174689 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0018134-1144-4076-95ca-7383b46a2985-workload-certs\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-7h99v\" (UID: \"e0018134-1144-4076-95ca-7383b46a2985\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" Oct 30 22:26:55.174807 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:55.174786 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0018134-1144-4076-95ca-7383b46a2985-workload-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-7h99v\" (UID: \"e0018134-1144-4076-95ca-7383b46a2985\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" Oct 30 22:26:55.174891 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:55.174879 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0018134-1144-4076-95ca-7383b46a2985-istio-envoy\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-7h99v\" (UID: \"e0018134-1144-4076-95ca-7383b46a2985\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" Oct 30 22:26:55.189401 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:55.189381 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-422c1029-openshift-default-dockercfg-f8xcc\"" Oct 30 22:26:55.275594 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:55.275561 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0018134-1144-4076-95ca-7383b46a2985-istio-envoy\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-7h99v\" (UID: \"e0018134-1144-4076-95ca-7383b46a2985\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" Oct 30 22:26:55.275781 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:55.275623 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0018134-1144-4076-95ca-7383b46a2985-istio-token\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-7h99v\" (UID: \"e0018134-1144-4076-95ca-7383b46a2985\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" Oct 30 22:26:55.275781 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:55.275672 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfvv4\" (UniqueName: \"kubernetes.io/projected/e0018134-1144-4076-95ca-7383b46a2985-kube-api-access-xfvv4\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-7h99v\" (UID: \"e0018134-1144-4076-95ca-7383b46a2985\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" Oct 30 22:26:55.275781 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:55.275697 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0018134-1144-4076-95ca-7383b46a2985-istio-podinfo\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-7h99v\" (UID: \"e0018134-1144-4076-95ca-7383b46a2985\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" Oct 30 22:26:55.275781 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:55.275727 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0018134-1144-4076-95ca-7383b46a2985-istio-data\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-7h99v\" (UID: \"e0018134-1144-4076-95ca-7383b46a2985\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" Oct 30 22:26:55.275982 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:55.275889 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0018134-1144-4076-95ca-7383b46a2985-credential-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-7h99v\" (UID: \"e0018134-1144-4076-95ca-7383b46a2985\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" Oct 30 22:26:55.275982 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:55.275956 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0018134-1144-4076-95ca-7383b46a2985-istiod-ca-cert\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-7h99v\" (UID: \"e0018134-1144-4076-95ca-7383b46a2985\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" Oct 30 22:26:55.276091 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:55.275988 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0018134-1144-4076-95ca-7383b46a2985-workload-certs\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-7h99v\" (UID: \"e0018134-1144-4076-95ca-7383b46a2985\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" Oct 30 22:26:55.276091 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:55.276052 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0018134-1144-4076-95ca-7383b46a2985-workload-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-7h99v\" (UID: \"e0018134-1144-4076-95ca-7383b46a2985\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" Oct 30 22:26:55.276198 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:55.276129 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0018134-1144-4076-95ca-7383b46a2985-istio-data\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-7h99v\" (UID: \"e0018134-1144-4076-95ca-7383b46a2985\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" Oct 30 22:26:55.276198 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:55.276192 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0018134-1144-4076-95ca-7383b46a2985-credential-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-7h99v\" (UID: \"e0018134-1144-4076-95ca-7383b46a2985\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" Oct 30 22:26:55.276410 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:55.276388 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0018134-1144-4076-95ca-7383b46a2985-workload-certs\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-7h99v\" (UID: \"e0018134-1144-4076-95ca-7383b46a2985\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" Oct 30 22:26:55.276474 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:55.276451 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0018134-1144-4076-95ca-7383b46a2985-workload-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-7h99v\" (UID: \"e0018134-1144-4076-95ca-7383b46a2985\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" Oct 30 22:26:55.276641 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:55.276625 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0018134-1144-4076-95ca-7383b46a2985-istiod-ca-cert\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-7h99v\" (UID: \"e0018134-1144-4076-95ca-7383b46a2985\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" Oct 30 22:26:55.278372 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:55.278354 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0018134-1144-4076-95ca-7383b46a2985-istio-podinfo\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-7h99v\" (UID: \"e0018134-1144-4076-95ca-7383b46a2985\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" Oct 30 22:26:55.278524 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:55.278509 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0018134-1144-4076-95ca-7383b46a2985-istio-envoy\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-7h99v\" (UID: \"e0018134-1144-4076-95ca-7383b46a2985\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" Oct 30 22:26:55.352152 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:55.352119 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfvv4\" (UniqueName: \"kubernetes.io/projected/e0018134-1144-4076-95ca-7383b46a2985-kube-api-access-xfvv4\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-7h99v\" (UID: \"e0018134-1144-4076-95ca-7383b46a2985\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" Oct 30 22:26:55.352294 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:55.352130 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0018134-1144-4076-95ca-7383b46a2985-istio-token\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-7h99v\" (UID: \"e0018134-1144-4076-95ca-7383b46a2985\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" Oct 30 22:26:55.369163 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:55.369072 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v"] Oct 30 22:26:55.450170 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:55.450144 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" Oct 30 22:26:55.886468 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:55.886445 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v"] Oct 30 22:26:55.888012 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:26:55.887984 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0018134_1144_4076_95ca_7383b46a2985.slice/crio-756670e5e3d7415fa457e08ed14f1bddcbfa2c759d4bc209965975660ad6ead3 WatchSource:0}: Error finding container 756670e5e3d7415fa457e08ed14f1bddcbfa2c759d4bc209965975660ad6ead3: Status 404 returned error can't find the container with id 756670e5e3d7415fa457e08ed14f1bddcbfa2c759d4bc209965975660ad6ead3 Oct 30 22:26:55.890437 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:55.890405 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:26:55.890497 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:55.890469 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:26:55.890530 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:55.890499 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:26:56.354113 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:56.354084 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp" Oct 30 22:26:56.385888 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:56.385838 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" event={"ID":"e0018134-1144-4076-95ca-7383b46a2985","Type":"ContainerStarted","Data":"3dcf52a5b85487a2bc84f4e33aaad4633963baa5afdef5b05528010a06eb2a24"} Oct 30 22:26:56.386096 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:56.385923 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" event={"ID":"e0018134-1144-4076-95ca-7383b46a2985","Type":"ContainerStarted","Data":"756670e5e3d7415fa457e08ed14f1bddcbfa2c759d4bc209965975660ad6ead3"} Oct 30 22:26:56.450762 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:56.450731 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" Oct 30 22:26:56.452461 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:56.452429 5918 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 22:26:56.452606 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:56.452488 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" podUID="e0018134-1144-4076-95ca-7383b46a2985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 22:26:56.617311 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:56.617172 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" podStartSLOduration=1.6171493369999999 podStartE2EDuration="1.617149337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:56.614641999 +0000 UTC m=+2176.885340086" watchObservedRunningTime="2026-10-30 22:26:56.617149337 +0000 UTC m=+2176.887847422" Oct 30 22:26:56.967302 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:56.967268 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz"] Oct 30 22:26:56.974168 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:56.974137 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb"] Oct 30 22:26:57.014945 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:57.014898 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp"] Oct 30 22:26:57.450684 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:57.450658 5918 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 22:26:57.451112 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:57.450726 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" podUID="e0018134-1144-4076-95ca-7383b46a2985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 22:26:58.450983 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:58.450946 5918 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 22:26:58.451414 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:58.451009 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" podUID="e0018134-1144-4076-95ca-7383b46a2985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 22:26:59.450553 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:59.450523 5918 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 22:26:59.450811 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:59.450581 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" podUID="e0018134-1144-4076-95ca-7383b46a2985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 22:26:59.884422 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:59.884336 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh"] Oct 30 22:26:59.911533 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:59.911493 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" Oct 30 22:26:59.955774 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:59.955735 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4488ee36-392f-42c2-8dad-962472895f91-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-dkzwh\" (UID: \"4488ee36-392f-42c2-8dad-962472895f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" Oct 30 22:26:59.955936 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:59.955778 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4488ee36-392f-42c2-8dad-962472895f91-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-dkzwh\" (UID: \"4488ee36-392f-42c2-8dad-962472895f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" Oct 30 22:26:59.955936 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:59.955862 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4488ee36-392f-42c2-8dad-962472895f91-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-dkzwh\" (UID: \"4488ee36-392f-42c2-8dad-962472895f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" Oct 30 22:26:59.956023 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:59.955963 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4488ee36-392f-42c2-8dad-962472895f91-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-dkzwh\" (UID: \"4488ee36-392f-42c2-8dad-962472895f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" Oct 30 22:26:59.956023 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:59.956008 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4488ee36-392f-42c2-8dad-962472895f91-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-dkzwh\" (UID: \"4488ee36-392f-42c2-8dad-962472895f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" Oct 30 22:26:59.956151 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:59.956137 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4488ee36-392f-42c2-8dad-962472895f91-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-dkzwh\" (UID: \"4488ee36-392f-42c2-8dad-962472895f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" Oct 30 22:26:59.956196 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:59.956160 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g78xt\" (UniqueName: \"kubernetes.io/projected/4488ee36-392f-42c2-8dad-962472895f91-kube-api-access-g78xt\") pod \"maas-default-gateway-openshift-default-67bf986f99-dkzwh\" (UID: \"4488ee36-392f-42c2-8dad-962472895f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" Oct 30 22:26:59.956297 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:59.956282 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4488ee36-392f-42c2-8dad-962472895f91-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-dkzwh\" (UID: \"4488ee36-392f-42c2-8dad-962472895f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" Oct 30 22:26:59.956406 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:26:59.956383 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4488ee36-392f-42c2-8dad-962472895f91-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-dkzwh\" (UID: \"4488ee36-392f-42c2-8dad-962472895f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" Oct 30 22:27:00.030983 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:00.029089 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh"] Oct 30 22:27:00.057565 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:00.057533 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4488ee36-392f-42c2-8dad-962472895f91-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-dkzwh\" (UID: \"4488ee36-392f-42c2-8dad-962472895f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" Oct 30 22:27:00.057734 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:00.057586 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4488ee36-392f-42c2-8dad-962472895f91-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-dkzwh\" (UID: \"4488ee36-392f-42c2-8dad-962472895f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" Oct 30 22:27:00.057734 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:00.057622 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4488ee36-392f-42c2-8dad-962472895f91-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-dkzwh\" (UID: \"4488ee36-392f-42c2-8dad-962472895f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" Oct 30 22:27:00.057734 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:00.057646 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4488ee36-392f-42c2-8dad-962472895f91-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-dkzwh\" (UID: \"4488ee36-392f-42c2-8dad-962472895f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" Oct 30 22:27:00.057734 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:00.057692 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4488ee36-392f-42c2-8dad-962472895f91-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-dkzwh\" (UID: \"4488ee36-392f-42c2-8dad-962472895f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" Oct 30 22:27:00.057734 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:00.057726 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4488ee36-392f-42c2-8dad-962472895f91-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-dkzwh\" (UID: \"4488ee36-392f-42c2-8dad-962472895f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" Oct 30 22:27:00.057903 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:00.057756 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4488ee36-392f-42c2-8dad-962472895f91-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-dkzwh\" (UID: \"4488ee36-392f-42c2-8dad-962472895f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" Oct 30 22:27:00.057903 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:00.057815 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4488ee36-392f-42c2-8dad-962472895f91-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-dkzwh\" (UID: \"4488ee36-392f-42c2-8dad-962472895f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" Oct 30 22:27:00.057903 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:00.057833 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g78xt\" (UniqueName: \"kubernetes.io/projected/4488ee36-392f-42c2-8dad-962472895f91-kube-api-access-g78xt\") pod \"maas-default-gateway-openshift-default-67bf986f99-dkzwh\" (UID: \"4488ee36-392f-42c2-8dad-962472895f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" Oct 30 22:27:00.058017 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:00.057992 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4488ee36-392f-42c2-8dad-962472895f91-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-dkzwh\" (UID: \"4488ee36-392f-42c2-8dad-962472895f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" Oct 30 22:27:00.058733 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:00.058699 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4488ee36-392f-42c2-8dad-962472895f91-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-dkzwh\" (UID: \"4488ee36-392f-42c2-8dad-962472895f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" Oct 30 22:27:00.058926 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:00.058786 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4488ee36-392f-42c2-8dad-962472895f91-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-dkzwh\" (UID: \"4488ee36-392f-42c2-8dad-962472895f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" Oct 30 22:27:00.059006 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:00.058986 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4488ee36-392f-42c2-8dad-962472895f91-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-dkzwh\" (UID: \"4488ee36-392f-42c2-8dad-962472895f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" Oct 30 22:27:00.059261 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:00.059205 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4488ee36-392f-42c2-8dad-962472895f91-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-dkzwh\" (UID: \"4488ee36-392f-42c2-8dad-962472895f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" Oct 30 22:27:00.061074 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:00.061055 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4488ee36-392f-42c2-8dad-962472895f91-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-dkzwh\" (UID: \"4488ee36-392f-42c2-8dad-962472895f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" Oct 30 22:27:00.061505 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:00.061484 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4488ee36-392f-42c2-8dad-962472895f91-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-dkzwh\" (UID: \"4488ee36-392f-42c2-8dad-962472895f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" Oct 30 22:27:00.124672 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:00.124636 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g78xt\" (UniqueName: \"kubernetes.io/projected/4488ee36-392f-42c2-8dad-962472895f91-kube-api-access-g78xt\") pod \"maas-default-gateway-openshift-default-67bf986f99-dkzwh\" (UID: \"4488ee36-392f-42c2-8dad-962472895f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" Oct 30 22:27:00.127009 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:00.126983 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4488ee36-392f-42c2-8dad-962472895f91-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-dkzwh\" (UID: \"4488ee36-392f-42c2-8dad-962472895f91\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" Oct 30 22:27:00.227037 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:00.227002 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" Oct 30 22:27:00.450659 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:00.450621 5918 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 22:27:00.450842 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:00.450670 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" podUID="e0018134-1144-4076-95ca-7383b46a2985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 22:27:00.683424 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:00.683396 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh"] Oct 30 22:27:00.684492 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:27:00.684465 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4488ee36_392f_42c2_8dad_962472895f91.slice/crio-ee9c81243c73eb6df93bb6477104f0067e9b4aa2f729b5a494f6e9ca01e6ac76 WatchSource:0}: Error finding container ee9c81243c73eb6df93bb6477104f0067e9b4aa2f729b5a494f6e9ca01e6ac76: Status 404 returned error can't find the container with id ee9c81243c73eb6df93bb6477104f0067e9b4aa2f729b5a494f6e9ca01e6ac76 Oct 30 22:27:00.705243 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:00.693118 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:27:00.705243 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:00.693230 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:27:00.705243 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:00.693284 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:27:01.440288 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:01.440252 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" event={"ID":"4488ee36-392f-42c2-8dad-962472895f91","Type":"ContainerStarted","Data":"729e50437516ff7b9182c968437d1575975ba3fc93df50f70b541848bf9aab22"} Oct 30 22:27:01.440674 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:01.440313 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" event={"ID":"4488ee36-392f-42c2-8dad-962472895f91","Type":"ContainerStarted","Data":"ee9c81243c73eb6df93bb6477104f0067e9b4aa2f729b5a494f6e9ca01e6ac76"} Oct 30 22:27:01.450842 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:01.450818 5918 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 22:27:01.450909 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:01.450864 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" podUID="e0018134-1144-4076-95ca-7383b46a2985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 22:27:01.580753 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:01.580702 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" podStartSLOduration=2.580688753 podStartE2EDuration="2.580688753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:01.578283593 +0000 UTC m=+2181.848981678" watchObservedRunningTime="2026-10-30 22:27:01.580688753 +0000 UTC m=+2181.851386836" Oct 30 22:27:02.227910 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:02.227873 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" Oct 30 22:27:02.229247 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:02.229196 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dkzwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 22:27:02.229449 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:02.229282 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" podUID="4488ee36-392f-42c2-8dad-962472895f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 22:27:02.930119 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:02.930082 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" Oct 30 22:27:02.930693 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:02.930670 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" Oct 30 22:27:02.935018 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:02.934986 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" Oct 30 22:27:03.228289 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:03.228174 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dkzwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 22:27:03.228289 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:03.228264 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" podUID="4488ee36-392f-42c2-8dad-962472895f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 22:27:04.227945 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:04.227903 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dkzwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 22:27:04.228367 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:04.227983 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" podUID="4488ee36-392f-42c2-8dad-962472895f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 22:27:05.054798 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:05.054761 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6920-6dfd21"] Oct 30 22:27:05.058658 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:05.058640 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6920-6dfd21" Oct 30 22:27:05.115038 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:05.115007 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6920-6dfd21"] Oct 30 22:27:05.133195 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:05.133173 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-9tw55\"" Oct 30 22:27:05.146247 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:05.146198 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7twrz\" (UniqueName: \"kubernetes.io/projected/c59f200e-18ca-49aa-a4a3-2e3ca93a7e62-kube-api-access-7twrz\") pod \"test-curl-6920-6dfd21\" (UID: \"c59f200e-18ca-49aa-a4a3-2e3ca93a7e62\") " pod="opendatahub/test-curl-6920-6dfd21" Oct 30 22:27:05.227770 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:05.227733 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dkzwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 22:27:05.227959 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:05.227807 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" podUID="4488ee36-392f-42c2-8dad-962472895f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 22:27:05.247648 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:05.247616 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7twrz\" (UniqueName: \"kubernetes.io/projected/c59f200e-18ca-49aa-a4a3-2e3ca93a7e62-kube-api-access-7twrz\") pod \"test-curl-6920-6dfd21\" (UID: \"c59f200e-18ca-49aa-a4a3-2e3ca93a7e62\") " pod="opendatahub/test-curl-6920-6dfd21" Oct 30 22:27:05.315362 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:05.315280 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7twrz\" (UniqueName: \"kubernetes.io/projected/c59f200e-18ca-49aa-a4a3-2e3ca93a7e62-kube-api-access-7twrz\") pod \"test-curl-6920-6dfd21\" (UID: \"c59f200e-18ca-49aa-a4a3-2e3ca93a7e62\") " pod="opendatahub/test-curl-6920-6dfd21" Oct 30 22:27:05.372050 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:05.372020 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6920-6dfd21" Oct 30 22:27:05.776053 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:05.776030 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6920-6dfd21"] Oct 30 22:27:05.777376 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:27:05.777343 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc59f200e_18ca_49aa_a4a3_2e3ca93a7e62.slice/crio-f82d4f17060562348874586deb2ac56a37107df8d3293831ce25cc606db8c755 WatchSource:0}: Error finding container f82d4f17060562348874586deb2ac56a37107df8d3293831ce25cc606db8c755: Status 404 returned error can't find the container with id f82d4f17060562348874586deb2ac56a37107df8d3293831ce25cc606db8c755 Oct 30 22:27:06.228267 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:06.228190 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dkzwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 22:27:06.228450 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:06.228290 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" podUID="4488ee36-392f-42c2-8dad-962472895f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 22:27:06.479904 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:06.479812 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6920-6dfd21" event={"ID":"c59f200e-18ca-49aa-a4a3-2e3ca93a7e62","Type":"ContainerStarted","Data":"5113a00ddb2f538ada4f609fe46ef0b7aea0cd07187f2cb26908b352d3e97ceb"} Oct 30 22:27:06.479904 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:06.479860 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6920-6dfd21" event={"ID":"c59f200e-18ca-49aa-a4a3-2e3ca93a7e62","Type":"ContainerStarted","Data":"f82d4f17060562348874586deb2ac56a37107df8d3293831ce25cc606db8c755"} Oct 30 22:27:07.228039 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:07.228002 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dkzwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 22:27:07.228252 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:07.228072 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" podUID="4488ee36-392f-42c2-8dad-962472895f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 22:27:08.177295 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:08.177195 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6920-6dfd21" podStartSLOduration=3.177180392 podStartE2EDuration="3.177180392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:06.612400792 +0000 UTC m=+2186.883098875" watchObservedRunningTime="2026-10-30 22:27:08.177180392 +0000 UTC m=+2188.447878472" Oct 30 22:27:08.178546 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:08.178519 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6920-6dfd21"] Oct 30 22:27:08.181998 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:08.181966 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6920-6dfd21"] Oct 30 22:27:08.227886 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:08.227857 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dkzwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 22:27:08.228057 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:08.227905 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" podUID="4488ee36-392f-42c2-8dad-962472895f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 22:27:08.494287 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:08.494177 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6920-6dfd21" podUID="c59f200e-18ca-49aa-a4a3-2e3ca93a7e62" containerName="test-curl-6920-6dfd21" containerID="cri-o://5113a00ddb2f538ada4f609fe46ef0b7aea0cd07187f2cb26908b352d3e97ceb" gracePeriod=2 Oct 30 22:27:08.545983 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:27:08.545944 5918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6920-6dfd21\" is forbidden: User \"system:node:ip-10-0-137-81.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-81.us-west-1.compute.internal' and this object" podUID="c59f200e-18ca-49aa-a4a3-2e3ca93a7e62" pod="opendatahub/test-curl-6920-6dfd21" Oct 30 22:27:09.630416 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:09.630375 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" Oct 30 22:27:09.630934 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:09.630650 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" Oct 30 22:27:09.631464 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:09.631446 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" Oct 30 22:27:09.862409 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:27:09.862368 5918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6920-6dfd21\" is forbidden: User \"system:node:ip-10-0-137-81.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-81.us-west-1.compute.internal' and this object" podUID="c59f200e-18ca-49aa-a4a3-2e3ca93a7e62" pod="opendatahub/test-curl-6920-6dfd21" Oct 30 22:27:10.262675 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:10.262637 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm"] Oct 30 22:27:10.263510 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:10.263481 5918 cpu_manager.go:414] "containerMap: removing container" podUID="c59f200e-18ca-49aa-a4a3-2e3ca93a7e62" containerName="test-curl-6920-6dfd21" Oct 30 22:27:10.263704 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:10.263675 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c59f200e-18ca-49aa-a4a3-2e3ca93a7e62" containerName="test-curl-6920-6dfd21" Oct 30 22:27:10.263947 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:10.263931 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c59f200e-18ca-49aa-a4a3-2e3ca93a7e62" containerName="test-curl-6920-6dfd21" Oct 30 22:27:10.267987 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:10.267962 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" Oct 30 22:27:10.352482 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:10.352445 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm"] Oct 30 22:27:10.416071 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:10.416033 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/100467f8-8ee2-4cda-a9a7-f52b11af838c-istio-podinfo\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-n26pm\" (UID: \"100467f8-8ee2-4cda-a9a7-f52b11af838c\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" Oct 30 22:27:10.416276 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:10.416203 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/100467f8-8ee2-4cda-a9a7-f52b11af838c-workload-certs\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-n26pm\" (UID: \"100467f8-8ee2-4cda-a9a7-f52b11af838c\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" Oct 30 22:27:10.416337 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:10.416284 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/100467f8-8ee2-4cda-a9a7-f52b11af838c-credential-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-n26pm\" (UID: \"100467f8-8ee2-4cda-a9a7-f52b11af838c\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" Oct 30 22:27:10.416337 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:10.416328 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fhpx\" (UniqueName: \"kubernetes.io/projected/100467f8-8ee2-4cda-a9a7-f52b11af838c-kube-api-access-7fhpx\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-n26pm\" (UID: \"100467f8-8ee2-4cda-a9a7-f52b11af838c\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" Oct 30 22:27:10.416418 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:10.416360 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/100467f8-8ee2-4cda-a9a7-f52b11af838c-istiod-ca-cert\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-n26pm\" (UID: \"100467f8-8ee2-4cda-a9a7-f52b11af838c\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" Oct 30 22:27:10.416418 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:10.416385 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/100467f8-8ee2-4cda-a9a7-f52b11af838c-istio-data\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-n26pm\" (UID: \"100467f8-8ee2-4cda-a9a7-f52b11af838c\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" Oct 30 22:27:10.416505 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:10.416451 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/100467f8-8ee2-4cda-a9a7-f52b11af838c-istio-envoy\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-n26pm\" (UID: \"100467f8-8ee2-4cda-a9a7-f52b11af838c\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" Oct 30 22:27:10.416505 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:10.416485 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/100467f8-8ee2-4cda-a9a7-f52b11af838c-istio-token\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-n26pm\" (UID: \"100467f8-8ee2-4cda-a9a7-f52b11af838c\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" Oct 30 22:27:10.416583 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:10.416522 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/100467f8-8ee2-4cda-a9a7-f52b11af838c-workload-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-n26pm\" (UID: \"100467f8-8ee2-4cda-a9a7-f52b11af838c\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" Oct 30 22:27:10.517978 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:10.517949 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/100467f8-8ee2-4cda-a9a7-f52b11af838c-istio-envoy\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-n26pm\" (UID: \"100467f8-8ee2-4cda-a9a7-f52b11af838c\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" Oct 30 22:27:10.518100 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:10.517988 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/100467f8-8ee2-4cda-a9a7-f52b11af838c-istio-token\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-n26pm\" (UID: \"100467f8-8ee2-4cda-a9a7-f52b11af838c\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" Oct 30 22:27:10.518100 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:10.518012 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/100467f8-8ee2-4cda-a9a7-f52b11af838c-workload-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-n26pm\" (UID: \"100467f8-8ee2-4cda-a9a7-f52b11af838c\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" Oct 30 22:27:10.518100 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:10.518097 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/100467f8-8ee2-4cda-a9a7-f52b11af838c-istio-podinfo\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-n26pm\" (UID: \"100467f8-8ee2-4cda-a9a7-f52b11af838c\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" Oct 30 22:27:10.518265 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:10.518154 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/100467f8-8ee2-4cda-a9a7-f52b11af838c-workload-certs\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-n26pm\" (UID: \"100467f8-8ee2-4cda-a9a7-f52b11af838c\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" Oct 30 22:27:10.518265 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:10.518181 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/100467f8-8ee2-4cda-a9a7-f52b11af838c-credential-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-n26pm\" (UID: \"100467f8-8ee2-4cda-a9a7-f52b11af838c\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" Oct 30 22:27:10.518265 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:10.518239 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fhpx\" (UniqueName: \"kubernetes.io/projected/100467f8-8ee2-4cda-a9a7-f52b11af838c-kube-api-access-7fhpx\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-n26pm\" (UID: \"100467f8-8ee2-4cda-a9a7-f52b11af838c\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" Oct 30 22:27:10.518520 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:10.518491 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/100467f8-8ee2-4cda-a9a7-f52b11af838c-workload-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-n26pm\" (UID: \"100467f8-8ee2-4cda-a9a7-f52b11af838c\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" Oct 30 22:27:10.518608 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:10.518533 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/100467f8-8ee2-4cda-a9a7-f52b11af838c-workload-certs\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-n26pm\" (UID: \"100467f8-8ee2-4cda-a9a7-f52b11af838c\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" Oct 30 22:27:10.518608 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:10.518558 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/100467f8-8ee2-4cda-a9a7-f52b11af838c-credential-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-n26pm\" (UID: \"100467f8-8ee2-4cda-a9a7-f52b11af838c\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" Oct 30 22:27:10.518720 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:10.518624 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/100467f8-8ee2-4cda-a9a7-f52b11af838c-istiod-ca-cert\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-n26pm\" (UID: \"100467f8-8ee2-4cda-a9a7-f52b11af838c\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" Oct 30 22:27:10.518720 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:10.518656 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/100467f8-8ee2-4cda-a9a7-f52b11af838c-istio-data\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-n26pm\" (UID: \"100467f8-8ee2-4cda-a9a7-f52b11af838c\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" Oct 30 22:27:10.519103 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:10.519078 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/100467f8-8ee2-4cda-a9a7-f52b11af838c-istio-data\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-n26pm\" (UID: \"100467f8-8ee2-4cda-a9a7-f52b11af838c\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" Oct 30 22:27:10.519575 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:10.519525 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/100467f8-8ee2-4cda-a9a7-f52b11af838c-istiod-ca-cert\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-n26pm\" (UID: \"100467f8-8ee2-4cda-a9a7-f52b11af838c\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" Oct 30 22:27:10.521482 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:10.521463 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/100467f8-8ee2-4cda-a9a7-f52b11af838c-istio-envoy\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-n26pm\" (UID: \"100467f8-8ee2-4cda-a9a7-f52b11af838c\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" Oct 30 22:27:10.521561 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:10.521516 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/100467f8-8ee2-4cda-a9a7-f52b11af838c-istio-podinfo\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-n26pm\" (UID: \"100467f8-8ee2-4cda-a9a7-f52b11af838c\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" Oct 30 22:27:10.596259 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:10.586248 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fhpx\" (UniqueName: \"kubernetes.io/projected/100467f8-8ee2-4cda-a9a7-f52b11af838c-kube-api-access-7fhpx\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-n26pm\" (UID: \"100467f8-8ee2-4cda-a9a7-f52b11af838c\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" Oct 30 22:27:10.596259 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:10.594936 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/100467f8-8ee2-4cda-a9a7-f52b11af838c-istio-token\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-n26pm\" (UID: \"100467f8-8ee2-4cda-a9a7-f52b11af838c\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" Oct 30 22:27:10.768726 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:10.768633 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6920-6dfd21" Oct 30 22:27:10.821010 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:27:10.820970 5918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6920-6dfd21\" is forbidden: User \"system:node:ip-10-0-137-81.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-81.us-west-1.compute.internal' and this object" podUID="c59f200e-18ca-49aa-a4a3-2e3ca93a7e62" pod="opendatahub/test-curl-6920-6dfd21" Oct 30 22:27:10.885665 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:10.885634 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" Oct 30 22:27:10.923407 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:10.923380 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c59f200e-18ca-49aa-a4a3-2e3ca93a7e62-kube-api-access-7twrz\" (UniqueName: \"kubernetes.io/projected/c59f200e-18ca-49aa-a4a3-2e3ca93a7e62-kube-api-access-7twrz\") pod \"c59f200e-18ca-49aa-a4a3-2e3ca93a7e62\" (UID: \"c59f200e-18ca-49aa-a4a3-2e3ca93a7e62\") " Oct 30 22:27:10.925561 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:10.925535 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c59f200e-18ca-49aa-a4a3-2e3ca93a7e62-kube-api-access-7twrz" pod "c59f200e-18ca-49aa-a4a3-2e3ca93a7e62" (UID: "c59f200e-18ca-49aa-a4a3-2e3ca93a7e62"). InnerVolumeSpecName "kube-api-access-7twrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:11.025502 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:11.025420 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7twrz\" (UniqueName: \"kubernetes.io/projected/c59f200e-18ca-49aa-a4a3-2e3ca93a7e62-kube-api-access-7twrz\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:11.319743 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:11.319653 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm"] Oct 30 22:27:11.355175 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:27:11.355136 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100467f8_8ee2_4cda_a9a7_f52b11af838c.slice/crio-4e0d836d5b883eead59f326341cd62286a6c62b463b2243767e975e32f111a7e WatchSource:0}: Error finding container 4e0d836d5b883eead59f326341cd62286a6c62b463b2243767e975e32f111a7e: Status 404 returned error can't find the container with id 4e0d836d5b883eead59f326341cd62286a6c62b463b2243767e975e32f111a7e Oct 30 22:27:11.357794 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:11.357761 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:27:11.357890 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:11.357831 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:27:11.357890 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:11.357857 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:27:11.517635 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:11.517604 5918 generic.go:356] "Generic (PLEG): container finished" podID="c59f200e-18ca-49aa-a4a3-2e3ca93a7e62" containerID="5113a00ddb2f538ada4f609fe46ef0b7aea0cd07187f2cb26908b352d3e97ceb" exitCode=137 Oct 30 22:27:11.517766 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:11.517756 5918 scope.go:122] "RemoveContainer" containerID="5113a00ddb2f538ada4f609fe46ef0b7aea0cd07187f2cb26908b352d3e97ceb" Oct 30 22:27:11.517904 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:11.517888 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6920-6dfd21" Oct 30 22:27:11.522042 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:11.522016 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" event={"ID":"100467f8-8ee2-4cda-a9a7-f52b11af838c","Type":"ContainerStarted","Data":"b02729058eec38f80828a0236a705def98d2dfe3be1d5ab611632ce2d47815fd"} Oct 30 22:27:11.522165 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:11.522060 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" event={"ID":"100467f8-8ee2-4cda-a9a7-f52b11af838c","Type":"ContainerStarted","Data":"4e0d836d5b883eead59f326341cd62286a6c62b463b2243767e975e32f111a7e"} Oct 30 22:27:11.529877 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:11.529858 5918 scope.go:122] "RemoveContainer" containerID="5113a00ddb2f538ada4f609fe46ef0b7aea0cd07187f2cb26908b352d3e97ceb" Oct 30 22:27:11.530156 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:27:11.530137 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5113a00ddb2f538ada4f609fe46ef0b7aea0cd07187f2cb26908b352d3e97ceb\": container with ID starting with 5113a00ddb2f538ada4f609fe46ef0b7aea0cd07187f2cb26908b352d3e97ceb not found: ID does not exist" containerID="5113a00ddb2f538ada4f609fe46ef0b7aea0cd07187f2cb26908b352d3e97ceb" Oct 30 22:27:11.530259 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:11.530165 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5113a00ddb2f538ada4f609fe46ef0b7aea0cd07187f2cb26908b352d3e97ceb"} err="failed to get container status \"5113a00ddb2f538ada4f609fe46ef0b7aea0cd07187f2cb26908b352d3e97ceb\": rpc error: code = NotFound desc = could not find container \"5113a00ddb2f538ada4f609fe46ef0b7aea0cd07187f2cb26908b352d3e97ceb\": container with ID starting with 5113a00ddb2f538ada4f609fe46ef0b7aea0cd07187f2cb26908b352d3e97ceb not found: ID does not exist" Oct 30 22:27:11.569874 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:27:11.569807 5918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6920-6dfd21\" is forbidden: User \"system:node:ip-10-0-137-81.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-81.us-west-1.compute.internal' and this object" podUID="c59f200e-18ca-49aa-a4a3-2e3ca93a7e62" pod="opendatahub/test-curl-6920-6dfd21" Oct 30 22:27:11.694665 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:11.694603 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" podStartSLOduration=1.694577446 podStartE2EDuration="1.694577446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:11.693847564 +0000 UTC m=+2191.964545659" watchObservedRunningTime="2026-10-30 22:27:11.694577446 +0000 UTC m=+2191.965275531" Oct 30 22:27:11.745435 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:27:11.745400 5918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6920-6dfd21\" is forbidden: User \"system:node:ip-10-0-137-81.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-81.us-west-1.compute.internal' and this object" podUID="c59f200e-18ca-49aa-a4a3-2e3ca93a7e62" pod="opendatahub/test-curl-6920-6dfd21" Oct 30 22:27:11.852701 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:11.852617 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c59f200e-18ca-49aa-a4a3-2e3ca93a7e62" path="/var/lib/kubelet/pods/c59f200e-18ca-49aa-a4a3-2e3ca93a7e62/volumes" Oct 30 22:27:11.893030 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:11.892314 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" Oct 30 22:27:12.016995 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.015285 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" podUID="395b4d7d-4aeb-4576-b616-67638bb1a2c2" containerName="main" containerID="cri-o://fe1b8c437772f9d9f45f5d72ead538d279183e43337d3ab38bc2a80f719d123d" gracePeriod=45 Oct 30 22:27:12.019035 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.018992 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" podUID="6536df0a-a317-4b24-aaa9-c4315d2f7dd5" containerName="main" containerID="cri-o://8fa29214bd7bfb44e6c0beeddc5a7d31b2526af18d0966e9703f6d81379a0b29" gracePeriod=45 Oct 30 22:27:12.038755 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.038720 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" podUID="f5ae2434-f161-4a5c-8086-2f0711678bb4" containerName="main" containerID="cri-o://24eff2da57e35ab3006bd0b98e55a9db408b1d66be3971dc22a9c68228f6f4e9" gracePeriod=45 Oct 30 22:27:12.139866 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.139827 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" podUID="f5ae2434-f161-4a5c-8086-2f0711678bb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": dial tcp 10.134.0.47:8000: connect: connection refused" Oct 30 22:27:12.357016 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.356986 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" Oct 30 22:27:12.381672 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.381649 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" Oct 30 22:27:12.385523 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.385499 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" Oct 30 22:27:12.445142 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.445111 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/395b4d7d-4aeb-4576-b616-67638bb1a2c2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/395b4d7d-4aeb-4576-b616-67638bb1a2c2-dshm\") pod \"395b4d7d-4aeb-4576-b616-67638bb1a2c2\" (UID: \"395b4d7d-4aeb-4576-b616-67638bb1a2c2\") " Oct 30 22:27:12.445369 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.445152 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/395b4d7d-4aeb-4576-b616-67638bb1a2c2-tls-certs\" (UniqueName: \"kubernetes.io/secret/395b4d7d-4aeb-4576-b616-67638bb1a2c2-tls-certs\") pod \"395b4d7d-4aeb-4576-b616-67638bb1a2c2\" (UID: \"395b4d7d-4aeb-4576-b616-67638bb1a2c2\") " Oct 30 22:27:12.445369 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.445235 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/395b4d7d-4aeb-4576-b616-67638bb1a2c2-home\" (UniqueName: \"kubernetes.io/empty-dir/395b4d7d-4aeb-4576-b616-67638bb1a2c2-home\") pod \"395b4d7d-4aeb-4576-b616-67638bb1a2c2\" (UID: \"395b4d7d-4aeb-4576-b616-67638bb1a2c2\") " Oct 30 22:27:12.445507 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.445374 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/395b4d7d-4aeb-4576-b616-67638bb1a2c2-kube-api-access-276vk\" (UniqueName: \"kubernetes.io/projected/395b4d7d-4aeb-4576-b616-67638bb1a2c2-kube-api-access-276vk\") pod \"395b4d7d-4aeb-4576-b616-67638bb1a2c2\" (UID: \"395b4d7d-4aeb-4576-b616-67638bb1a2c2\") " Oct 30 22:27:12.445507 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.445416 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/395b4d7d-4aeb-4576-b616-67638bb1a2c2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/395b4d7d-4aeb-4576-b616-67638bb1a2c2-tmp-dir\") pod \"395b4d7d-4aeb-4576-b616-67638bb1a2c2\" (UID: \"395b4d7d-4aeb-4576-b616-67638bb1a2c2\") " Oct 30 22:27:12.445611 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.445500 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/395b4d7d-4aeb-4576-b616-67638bb1a2c2-home" pod "395b4d7d-4aeb-4576-b616-67638bb1a2c2" (UID: "395b4d7d-4aeb-4576-b616-67638bb1a2c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:12.445675 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.445644 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/395b4d7d-4aeb-4576-b616-67638bb1a2c2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/395b4d7d-4aeb-4576-b616-67638bb1a2c2-model-cache\") pod \"395b4d7d-4aeb-4576-b616-67638bb1a2c2\" (UID: \"395b4d7d-4aeb-4576-b616-67638bb1a2c2\") " Oct 30 22:27:12.445780 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.445755 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/395b4d7d-4aeb-4576-b616-67638bb1a2c2-tmp-dir" pod "395b4d7d-4aeb-4576-b616-67638bb1a2c2" (UID: "395b4d7d-4aeb-4576-b616-67638bb1a2c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:12.445949 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.445904 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/395b4d7d-4aeb-4576-b616-67638bb1a2c2-model-cache" pod "395b4d7d-4aeb-4576-b616-67638bb1a2c2" (UID: "395b4d7d-4aeb-4576-b616-67638bb1a2c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:12.446378 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.446355 5918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/395b4d7d-4aeb-4576-b616-67638bb1a2c2-model-cache\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:12.446479 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.446382 5918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/395b4d7d-4aeb-4576-b616-67638bb1a2c2-home\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:12.446479 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.446399 5918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/395b4d7d-4aeb-4576-b616-67638bb1a2c2-tmp-dir\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:12.447607 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.447586 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/395b4d7d-4aeb-4576-b616-67638bb1a2c2-dshm" pod "395b4d7d-4aeb-4576-b616-67638bb1a2c2" (UID: "395b4d7d-4aeb-4576-b616-67638bb1a2c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:12.447708 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.447617 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/395b4d7d-4aeb-4576-b616-67638bb1a2c2-tls-certs" pod "395b4d7d-4aeb-4576-b616-67638bb1a2c2" (UID: "395b4d7d-4aeb-4576-b616-67638bb1a2c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:27:12.448042 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.448027 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/395b4d7d-4aeb-4576-b616-67638bb1a2c2-kube-api-access-276vk" pod "395b4d7d-4aeb-4576-b616-67638bb1a2c2" (UID: "395b4d7d-4aeb-4576-b616-67638bb1a2c2"). InnerVolumeSpecName "kube-api-access-276vk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:12.549671 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.549619 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-dshm\") pod \"6536df0a-a317-4b24-aaa9-c4315d2f7dd5\" (UID: \"6536df0a-a317-4b24-aaa9-c4315d2f7dd5\") " Oct 30 22:27:12.549671 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.549675 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-model-cache\") pod \"6536df0a-a317-4b24-aaa9-c4315d2f7dd5\" (UID: \"6536df0a-a317-4b24-aaa9-c4315d2f7dd5\") " Oct 30 22:27:12.549950 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.549774 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-tls-certs\" (UniqueName: \"kubernetes.io/secret/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-tls-certs\") pod \"6536df0a-a317-4b24-aaa9-c4315d2f7dd5\" (UID: \"6536df0a-a317-4b24-aaa9-c4315d2f7dd5\") " Oct 30 22:27:12.549950 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.549828 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-home\" (UniqueName: \"kubernetes.io/empty-dir/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-home\") pod \"6536df0a-a317-4b24-aaa9-c4315d2f7dd5\" (UID: \"6536df0a-a317-4b24-aaa9-c4315d2f7dd5\") " Oct 30 22:27:12.549950 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.549861 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5ae2434-f161-4a5c-8086-2f0711678bb4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5ae2434-f161-4a5c-8086-2f0711678bb4-dshm\") pod \"f5ae2434-f161-4a5c-8086-2f0711678bb4\" (UID: \"f5ae2434-f161-4a5c-8086-2f0711678bb4\") " Oct 30 22:27:12.549950 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.549898 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5ae2434-f161-4a5c-8086-2f0711678bb4-kube-api-access-dqqmk\" (UniqueName: \"kubernetes.io/projected/f5ae2434-f161-4a5c-8086-2f0711678bb4-kube-api-access-dqqmk\") pod \"f5ae2434-f161-4a5c-8086-2f0711678bb4\" (UID: \"f5ae2434-f161-4a5c-8086-2f0711678bb4\") " Oct 30 22:27:12.549950 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.549930 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5ae2434-f161-4a5c-8086-2f0711678bb4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5ae2434-f161-4a5c-8086-2f0711678bb4-model-cache\") pod \"f5ae2434-f161-4a5c-8086-2f0711678bb4\" (UID: \"f5ae2434-f161-4a5c-8086-2f0711678bb4\") " Oct 30 22:27:12.550232 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.549979 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5ae2434-f161-4a5c-8086-2f0711678bb4-home\" (UniqueName: \"kubernetes.io/empty-dir/f5ae2434-f161-4a5c-8086-2f0711678bb4-home\") pod \"f5ae2434-f161-4a5c-8086-2f0711678bb4\" (UID: \"f5ae2434-f161-4a5c-8086-2f0711678bb4\") " Oct 30 22:27:12.550232 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.550032 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-kube-api-access-dl8rn\" (UniqueName: \"kubernetes.io/projected/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-kube-api-access-dl8rn\") pod \"6536df0a-a317-4b24-aaa9-c4315d2f7dd5\" (UID: \"6536df0a-a317-4b24-aaa9-c4315d2f7dd5\") " Oct 30 22:27:12.550232 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.550093 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5ae2434-f161-4a5c-8086-2f0711678bb4-tls-certs\" (UniqueName: \"kubernetes.io/secret/f5ae2434-f161-4a5c-8086-2f0711678bb4-tls-certs\") pod \"f5ae2434-f161-4a5c-8086-2f0711678bb4\" (UID: \"f5ae2434-f161-4a5c-8086-2f0711678bb4\") " Oct 30 22:27:12.550232 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.550184 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5ae2434-f161-4a5c-8086-2f0711678bb4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5ae2434-f161-4a5c-8086-2f0711678bb4-tmp-dir\") pod \"f5ae2434-f161-4a5c-8086-2f0711678bb4\" (UID: \"f5ae2434-f161-4a5c-8086-2f0711678bb4\") " Oct 30 22:27:12.550452 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.550278 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-tmp-dir\") pod \"6536df0a-a317-4b24-aaa9-c4315d2f7dd5\" (UID: \"6536df0a-a317-4b24-aaa9-c4315d2f7dd5\") " Oct 30 22:27:12.550792 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.550774 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-276vk\" (UniqueName: \"kubernetes.io/projected/395b4d7d-4aeb-4576-b616-67638bb1a2c2-kube-api-access-276vk\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:12.550900 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.550797 5918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/395b4d7d-4aeb-4576-b616-67638bb1a2c2-dshm\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:12.550900 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.550814 5918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/395b4d7d-4aeb-4576-b616-67638bb1a2c2-tls-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:12.556984 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.556515 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5ae2434-f161-4a5c-8086-2f0711678bb4-home" pod "f5ae2434-f161-4a5c-8086-2f0711678bb4" (UID: "f5ae2434-f161-4a5c-8086-2f0711678bb4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:12.557728 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.557443 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5ae2434-f161-4a5c-8086-2f0711678bb4-model-cache" pod "f5ae2434-f161-4a5c-8086-2f0711678bb4" (UID: "f5ae2434-f161-4a5c-8086-2f0711678bb4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:12.558969 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.558369 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5ae2434-f161-4a5c-8086-2f0711678bb4-tmp-dir" pod "f5ae2434-f161-4a5c-8086-2f0711678bb4" (UID: "f5ae2434-f161-4a5c-8086-2f0711678bb4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:12.558969 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.558545 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-model-cache" pod "6536df0a-a317-4b24-aaa9-c4315d2f7dd5" (UID: "6536df0a-a317-4b24-aaa9-c4315d2f7dd5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:12.558969 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.558829 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-home" pod "6536df0a-a317-4b24-aaa9-c4315d2f7dd5" (UID: "6536df0a-a317-4b24-aaa9-c4315d2f7dd5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:12.560394 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.560367 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-tmp-dir" pod "6536df0a-a317-4b24-aaa9-c4315d2f7dd5" (UID: "6536df0a-a317-4b24-aaa9-c4315d2f7dd5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:12.561510 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.561488 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5ae2434-f161-4a5c-8086-2f0711678bb4-dshm" pod "f5ae2434-f161-4a5c-8086-2f0711678bb4" (UID: "f5ae2434-f161-4a5c-8086-2f0711678bb4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:12.564520 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.564490 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5ae2434-f161-4a5c-8086-2f0711678bb4-tls-certs" pod "f5ae2434-f161-4a5c-8086-2f0711678bb4" (UID: "f5ae2434-f161-4a5c-8086-2f0711678bb4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:27:12.568434 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.567566 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5ae2434-f161-4a5c-8086-2f0711678bb4-kube-api-access-dqqmk" pod "f5ae2434-f161-4a5c-8086-2f0711678bb4" (UID: "f5ae2434-f161-4a5c-8086-2f0711678bb4"). InnerVolumeSpecName "kube-api-access-dqqmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:12.568434 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.567607 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-dshm" pod "6536df0a-a317-4b24-aaa9-c4315d2f7dd5" (UID: "6536df0a-a317-4b24-aaa9-c4315d2f7dd5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:12.568434 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.568385 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-kube-api-access-dl8rn" pod "6536df0a-a317-4b24-aaa9-c4315d2f7dd5" (UID: "6536df0a-a317-4b24-aaa9-c4315d2f7dd5"). InnerVolumeSpecName "kube-api-access-dl8rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:12.569692 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.569664 5918 generic.go:356] "Generic (PLEG): container finished" podID="395b4d7d-4aeb-4576-b616-67638bb1a2c2" containerID="fe1b8c437772f9d9f45f5d72ead538d279183e43337d3ab38bc2a80f719d123d" exitCode=0 Oct 30 22:27:12.569804 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.569791 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" event={"ID":"395b4d7d-4aeb-4576-b616-67638bb1a2c2","Type":"ContainerDied","Data":"fe1b8c437772f9d9f45f5d72ead538d279183e43337d3ab38bc2a80f719d123d"} Oct 30 22:27:12.569873 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.569825 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" event={"ID":"395b4d7d-4aeb-4576-b616-67638bb1a2c2","Type":"ContainerDied","Data":"92eafdef17522c187b7992f548775a539c62e64bfae1fdbd6670dd7fa3cb2aba"} Oct 30 22:27:12.569873 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.569848 5918 scope.go:122] "RemoveContainer" containerID="fe1b8c437772f9d9f45f5d72ead538d279183e43337d3ab38bc2a80f719d123d" Oct 30 22:27:12.569983 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.569951 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz" Oct 30 22:27:12.577377 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.577358 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-tls-certs" pod "6536df0a-a317-4b24-aaa9-c4315d2f7dd5" (UID: "6536df0a-a317-4b24-aaa9-c4315d2f7dd5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:27:12.579352 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.578709 5918 generic.go:356] "Generic (PLEG): container finished" podID="f5ae2434-f161-4a5c-8086-2f0711678bb4" containerID="24eff2da57e35ab3006bd0b98e55a9db408b1d66be3971dc22a9c68228f6f4e9" exitCode=0 Oct 30 22:27:12.579352 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.578855 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" event={"ID":"f5ae2434-f161-4a5c-8086-2f0711678bb4","Type":"ContainerDied","Data":"24eff2da57e35ab3006bd0b98e55a9db408b1d66be3971dc22a9c68228f6f4e9"} Oct 30 22:27:12.579352 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.578885 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" event={"ID":"f5ae2434-f161-4a5c-8086-2f0711678bb4","Type":"ContainerDied","Data":"97bd66c9fc59bc7269c4e58207ead9982082ac6ab8f397e56d8a2708e608f249"} Oct 30 22:27:12.579352 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.578948 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp" Oct 30 22:27:12.583267 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.582576 5918 generic.go:356] "Generic (PLEG): container finished" podID="6536df0a-a317-4b24-aaa9-c4315d2f7dd5" containerID="8fa29214bd7bfb44e6c0beeddc5a7d31b2526af18d0966e9703f6d81379a0b29" exitCode=0 Oct 30 22:27:12.583267 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.582721 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" event={"ID":"6536df0a-a317-4b24-aaa9-c4315d2f7dd5","Type":"ContainerDied","Data":"8fa29214bd7bfb44e6c0beeddc5a7d31b2526af18d0966e9703f6d81379a0b29"} Oct 30 22:27:12.583267 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.582749 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" event={"ID":"6536df0a-a317-4b24-aaa9-c4315d2f7dd5","Type":"ContainerDied","Data":"ac6cd2cff8546d6e1900af41898112e5e675af7ba8d8aba015ce09861519dce1"} Oct 30 22:27:12.583267 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.582811 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb" Oct 30 22:27:12.651956 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.651779 5918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5ae2434-f161-4a5c-8086-2f0711678bb4-home\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:12.652104 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.652084 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl8rn\" (UniqueName: \"kubernetes.io/projected/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-kube-api-access-dl8rn\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:12.652186 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.652176 5918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5ae2434-f161-4a5c-8086-2f0711678bb4-tls-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:12.652268 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.652258 5918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5ae2434-f161-4a5c-8086-2f0711678bb4-tmp-dir\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:12.652356 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.652346 5918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-tmp-dir\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:12.652435 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.652425 5918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-dshm\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:12.652511 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.652499 5918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-model-cache\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:12.652574 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.652566 5918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-tls-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:12.652625 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.652619 5918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6536df0a-a317-4b24-aaa9-c4315d2f7dd5-home\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:12.652672 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.652666 5918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5ae2434-f161-4a5c-8086-2f0711678bb4-dshm\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:12.652726 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.652719 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqqmk\" (UniqueName: \"kubernetes.io/projected/f5ae2434-f161-4a5c-8086-2f0711678bb4-kube-api-access-dqqmk\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:12.652777 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.652771 5918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5ae2434-f161-4a5c-8086-2f0711678bb4-model-cache\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:12.657388 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.657368 5918 scope.go:122] "RemoveContainer" containerID="fe1b8c437772f9d9f45f5d72ead538d279183e43337d3ab38bc2a80f719d123d" Oct 30 22:27:12.658128 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:27:12.658083 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe1b8c437772f9d9f45f5d72ead538d279183e43337d3ab38bc2a80f719d123d\": container with ID starting with fe1b8c437772f9d9f45f5d72ead538d279183e43337d3ab38bc2a80f719d123d not found: ID does not exist" containerID="fe1b8c437772f9d9f45f5d72ead538d279183e43337d3ab38bc2a80f719d123d" Oct 30 22:27:12.658522 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.658492 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe1b8c437772f9d9f45f5d72ead538d279183e43337d3ab38bc2a80f719d123d"} err="failed to get container status \"fe1b8c437772f9d9f45f5d72ead538d279183e43337d3ab38bc2a80f719d123d\": rpc error: code = NotFound desc = could not find container \"fe1b8c437772f9d9f45f5d72ead538d279183e43337d3ab38bc2a80f719d123d\": container with ID starting with fe1b8c437772f9d9f45f5d72ead538d279183e43337d3ab38bc2a80f719d123d not found: ID does not exist" Oct 30 22:27:12.658657 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.658646 5918 scope.go:122] "RemoveContainer" containerID="24eff2da57e35ab3006bd0b98e55a9db408b1d66be3971dc22a9c68228f6f4e9" Oct 30 22:27:12.676264 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.676179 5918 scope.go:122] "RemoveContainer" containerID="24eff2da57e35ab3006bd0b98e55a9db408b1d66be3971dc22a9c68228f6f4e9" Oct 30 22:27:12.676781 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:27:12.676759 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24eff2da57e35ab3006bd0b98e55a9db408b1d66be3971dc22a9c68228f6f4e9\": container with ID starting with 24eff2da57e35ab3006bd0b98e55a9db408b1d66be3971dc22a9c68228f6f4e9 not found: ID does not exist" containerID="24eff2da57e35ab3006bd0b98e55a9db408b1d66be3971dc22a9c68228f6f4e9" Oct 30 22:27:12.676905 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.676787 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24eff2da57e35ab3006bd0b98e55a9db408b1d66be3971dc22a9c68228f6f4e9"} err="failed to get container status \"24eff2da57e35ab3006bd0b98e55a9db408b1d66be3971dc22a9c68228f6f4e9\": rpc error: code = NotFound desc = could not find container \"24eff2da57e35ab3006bd0b98e55a9db408b1d66be3971dc22a9c68228f6f4e9\": container with ID starting with 24eff2da57e35ab3006bd0b98e55a9db408b1d66be3971dc22a9c68228f6f4e9 not found: ID does not exist" Oct 30 22:27:12.676905 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.676806 5918 scope.go:122] "RemoveContainer" containerID="8fa29214bd7bfb44e6c0beeddc5a7d31b2526af18d0966e9703f6d81379a0b29" Oct 30 22:27:12.692535 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.688439 5918 scope.go:122] "RemoveContainer" containerID="8fa29214bd7bfb44e6c0beeddc5a7d31b2526af18d0966e9703f6d81379a0b29" Oct 30 22:27:12.692535 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:27:12.688925 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fa29214bd7bfb44e6c0beeddc5a7d31b2526af18d0966e9703f6d81379a0b29\": container with ID starting with 8fa29214bd7bfb44e6c0beeddc5a7d31b2526af18d0966e9703f6d81379a0b29 not found: ID does not exist" containerID="8fa29214bd7bfb44e6c0beeddc5a7d31b2526af18d0966e9703f6d81379a0b29" Oct 30 22:27:12.692535 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.688961 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fa29214bd7bfb44e6c0beeddc5a7d31b2526af18d0966e9703f6d81379a0b29"} err="failed to get container status \"8fa29214bd7bfb44e6c0beeddc5a7d31b2526af18d0966e9703f6d81379a0b29\": rpc error: code = NotFound desc = could not find container \"8fa29214bd7bfb44e6c0beeddc5a7d31b2526af18d0966e9703f6d81379a0b29\": container with ID starting with 8fa29214bd7bfb44e6c0beeddc5a7d31b2526af18d0966e9703f6d81379a0b29 not found: ID does not exist" Oct 30 22:27:12.890344 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.890308 5918 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 22:27:12.890932 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.890899 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" podUID="100467f8-8ee2-4cda-a9a7-f52b11af838c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 22:27:12.927301 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.927032 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb"] Oct 30 22:27:12.931739 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:12.931711 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-6a0a09/distinct-w0-6a0a09-kserve-b97fffc48-wnhcb"] Oct 30 22:27:13.050856 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:13.050817 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp"] Oct 30 22:27:13.063726 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:13.056448 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-6a0a09/simulator-w0-6a0a09-kserve-75bbdcdddd-6tbqp"] Oct 30 22:27:13.176593 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:13.176557 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz"] Oct 30 22:27:13.182696 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:13.182616 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-6a0a09/distinct-2-w0-6a0a09-kserve-744f74cf9-784kz"] Oct 30 22:27:13.861406 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:13.861353 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="395b4d7d-4aeb-4576-b616-67638bb1a2c2" path="/var/lib/kubelet/pods/395b4d7d-4aeb-4576-b616-67638bb1a2c2/volumes" Oct 30 22:27:13.862327 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:13.862303 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6536df0a-a317-4b24-aaa9-c4315d2f7dd5" path="/var/lib/kubelet/pods/6536df0a-a317-4b24-aaa9-c4315d2f7dd5/volumes" Oct 30 22:27:13.863054 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:13.863033 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5ae2434-f161-4a5c-8086-2f0711678bb4" path="/var/lib/kubelet/pods/f5ae2434-f161-4a5c-8086-2f0711678bb4/volumes" Oct 30 22:27:13.890238 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:13.886196 5918 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 22:27:13.890439 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:13.890411 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" podUID="100467f8-8ee2-4cda-a9a7-f52b11af838c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 22:27:14.896449 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:14.896412 5918 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 22:27:14.911375 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:14.911336 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" podUID="100467f8-8ee2-4cda-a9a7-f52b11af838c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 22:27:15.029242 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.026696 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p"] Oct 30 22:27:15.029242 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.027567 5918 cpu_manager.go:414] "containerMap: removing container" podUID="6536df0a-a317-4b24-aaa9-c4315d2f7dd5" containerName="main" Oct 30 22:27:15.029242 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.027589 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6536df0a-a317-4b24-aaa9-c4315d2f7dd5" containerName="main" Oct 30 22:27:15.029242 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.027694 5918 cpu_manager.go:414] "containerMap: removing container" podUID="f5ae2434-f161-4a5c-8086-2f0711678bb4" containerName="main" Oct 30 22:27:15.029242 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.027704 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5ae2434-f161-4a5c-8086-2f0711678bb4" containerName="main" Oct 30 22:27:15.029242 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.027727 5918 cpu_manager.go:414] "containerMap: removing container" podUID="395b4d7d-4aeb-4576-b616-67638bb1a2c2" containerName="main" Oct 30 22:27:15.029242 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.027736 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="395b4d7d-4aeb-4576-b616-67638bb1a2c2" containerName="main" Oct 30 22:27:15.029242 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.027880 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5ae2434-f161-4a5c-8086-2f0711678bb4" containerName="main" Oct 30 22:27:15.029242 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.027895 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="395b4d7d-4aeb-4576-b616-67638bb1a2c2" containerName="main" Oct 30 22:27:15.029242 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.027911 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="6536df0a-a317-4b24-aaa9-c4315d2f7dd5" containerName="main" Oct 30 22:27:15.064538 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.064508 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" Oct 30 22:27:15.180294 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.145344 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p"] Oct 30 22:27:15.206235 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.206175 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe658cf-6504-47e1-8640-cc6404e8a882-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-xmr6p\" (UID: \"1fe658cf-6504-47e1-8640-cc6404e8a882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" Oct 30 22:27:15.212542 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.206541 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe658cf-6504-47e1-8640-cc6404e8a882-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-xmr6p\" (UID: \"1fe658cf-6504-47e1-8640-cc6404e8a882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" Oct 30 22:27:15.220873 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.220837 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pxvt\" (UniqueName: \"kubernetes.io/projected/1fe658cf-6504-47e1-8640-cc6404e8a882-kube-api-access-2pxvt\") pod \"maas-default-gateway-openshift-default-67bf986f99-xmr6p\" (UID: \"1fe658cf-6504-47e1-8640-cc6404e8a882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" Oct 30 22:27:15.228863 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.228828 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fe658cf-6504-47e1-8640-cc6404e8a882-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-xmr6p\" (UID: \"1fe658cf-6504-47e1-8640-cc6404e8a882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" Oct 30 22:27:15.229200 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.229178 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe658cf-6504-47e1-8640-cc6404e8a882-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-xmr6p\" (UID: \"1fe658cf-6504-47e1-8640-cc6404e8a882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" Oct 30 22:27:15.229991 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.229583 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe658cf-6504-47e1-8640-cc6404e8a882-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-xmr6p\" (UID: \"1fe658cf-6504-47e1-8640-cc6404e8a882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" Oct 30 22:27:15.235397 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.235364 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe658cf-6504-47e1-8640-cc6404e8a882-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-xmr6p\" (UID: \"1fe658cf-6504-47e1-8640-cc6404e8a882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" Oct 30 22:27:15.235763 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.235631 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe658cf-6504-47e1-8640-cc6404e8a882-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-xmr6p\" (UID: \"1fe658cf-6504-47e1-8640-cc6404e8a882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" Oct 30 22:27:15.236101 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.236083 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe658cf-6504-47e1-8640-cc6404e8a882-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-xmr6p\" (UID: \"1fe658cf-6504-47e1-8640-cc6404e8a882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" Oct 30 22:27:15.347240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.343943 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe658cf-6504-47e1-8640-cc6404e8a882-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-xmr6p\" (UID: \"1fe658cf-6504-47e1-8640-cc6404e8a882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" Oct 30 22:27:15.347240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.344079 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe658cf-6504-47e1-8640-cc6404e8a882-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-xmr6p\" (UID: \"1fe658cf-6504-47e1-8640-cc6404e8a882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" Oct 30 22:27:15.347240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.344108 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pxvt\" (UniqueName: \"kubernetes.io/projected/1fe658cf-6504-47e1-8640-cc6404e8a882-kube-api-access-2pxvt\") pod \"maas-default-gateway-openshift-default-67bf986f99-xmr6p\" (UID: \"1fe658cf-6504-47e1-8640-cc6404e8a882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" Oct 30 22:27:15.347240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.344140 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fe658cf-6504-47e1-8640-cc6404e8a882-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-xmr6p\" (UID: \"1fe658cf-6504-47e1-8640-cc6404e8a882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" Oct 30 22:27:15.347240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.344175 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe658cf-6504-47e1-8640-cc6404e8a882-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-xmr6p\" (UID: \"1fe658cf-6504-47e1-8640-cc6404e8a882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" Oct 30 22:27:15.347240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.344246 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe658cf-6504-47e1-8640-cc6404e8a882-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-xmr6p\" (UID: \"1fe658cf-6504-47e1-8640-cc6404e8a882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" Oct 30 22:27:15.347240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.344289 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe658cf-6504-47e1-8640-cc6404e8a882-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-xmr6p\" (UID: \"1fe658cf-6504-47e1-8640-cc6404e8a882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" Oct 30 22:27:15.347240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.344317 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe658cf-6504-47e1-8640-cc6404e8a882-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-xmr6p\" (UID: \"1fe658cf-6504-47e1-8640-cc6404e8a882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" Oct 30 22:27:15.347240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.344412 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe658cf-6504-47e1-8640-cc6404e8a882-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-xmr6p\" (UID: \"1fe658cf-6504-47e1-8640-cc6404e8a882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" Oct 30 22:27:15.363241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.357323 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe658cf-6504-47e1-8640-cc6404e8a882-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-xmr6p\" (UID: \"1fe658cf-6504-47e1-8640-cc6404e8a882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" Oct 30 22:27:15.363241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.359410 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe658cf-6504-47e1-8640-cc6404e8a882-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-xmr6p\" (UID: \"1fe658cf-6504-47e1-8640-cc6404e8a882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" Oct 30 22:27:15.375258 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.372604 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe658cf-6504-47e1-8640-cc6404e8a882-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-xmr6p\" (UID: \"1fe658cf-6504-47e1-8640-cc6404e8a882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" Oct 30 22:27:15.375258 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.373063 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe658cf-6504-47e1-8640-cc6404e8a882-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-xmr6p\" (UID: \"1fe658cf-6504-47e1-8640-cc6404e8a882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" Oct 30 22:27:15.375258 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.373827 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe658cf-6504-47e1-8640-cc6404e8a882-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-xmr6p\" (UID: \"1fe658cf-6504-47e1-8640-cc6404e8a882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" Oct 30 22:27:15.375258 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.374112 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe658cf-6504-47e1-8640-cc6404e8a882-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-xmr6p\" (UID: \"1fe658cf-6504-47e1-8640-cc6404e8a882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" Oct 30 22:27:15.375744 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.375721 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe658cf-6504-47e1-8640-cc6404e8a882-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-xmr6p\" (UID: \"1fe658cf-6504-47e1-8640-cc6404e8a882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" Oct 30 22:27:15.434092 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.434017 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fe658cf-6504-47e1-8640-cc6404e8a882-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-xmr6p\" (UID: \"1fe658cf-6504-47e1-8640-cc6404e8a882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" Oct 30 22:27:15.434292 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.434198 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pxvt\" (UniqueName: \"kubernetes.io/projected/1fe658cf-6504-47e1-8640-cc6404e8a882-kube-api-access-2pxvt\") pod \"maas-default-gateway-openshift-default-67bf986f99-xmr6p\" (UID: \"1fe658cf-6504-47e1-8640-cc6404e8a882\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" Oct 30 22:27:15.703511 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.703423 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" Oct 30 22:27:15.911244 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.910321 5918 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 22:27:15.911244 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:15.910383 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" podUID="100467f8-8ee2-4cda-a9a7-f52b11af838c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 22:27:16.144369 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:16.144337 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p"] Oct 30 22:27:16.145763 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:27:16.145733 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe658cf_6504_47e1_8640_cc6404e8a882.slice/crio-6d3ccbadd14515b313b7fb76a826a77d07bac5036e18f9a71dcf6d855c3a0944 WatchSource:0}: Error finding container 6d3ccbadd14515b313b7fb76a826a77d07bac5036e18f9a71dcf6d855c3a0944: Status 404 returned error can't find the container with id 6d3ccbadd14515b313b7fb76a826a77d07bac5036e18f9a71dcf6d855c3a0944 Oct 30 22:27:16.148382 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:16.148349 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:27:16.148441 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:16.148412 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:27:16.148477 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:16.148443 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:27:16.658817 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:16.658786 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" event={"ID":"1fe658cf-6504-47e1-8640-cc6404e8a882","Type":"ContainerStarted","Data":"1488b865fe8307f0a807c108f9a8626cdedf9b8340d2c26e201e2c31ddd14771"} Oct 30 22:27:16.659020 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:16.658843 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" event={"ID":"1fe658cf-6504-47e1-8640-cc6404e8a882","Type":"ContainerStarted","Data":"6d3ccbadd14515b313b7fb76a826a77d07bac5036e18f9a71dcf6d855c3a0944"} Oct 30 22:27:16.711242 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:16.704285 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" Oct 30 22:27:16.834093 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:16.834033 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" podStartSLOduration=2.834017987 podStartE2EDuration="2.834017987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:16.833749852 +0000 UTC m=+2197.104447952" watchObservedRunningTime="2026-10-30 22:27:16.834017987 +0000 UTC m=+2197.104716074" Oct 30 22:27:16.906239 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:16.899541 5918 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 22:27:16.906239 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:16.899597 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" podUID="100467f8-8ee2-4cda-a9a7-f52b11af838c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 22:27:17.717253 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:17.713338 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-xmr6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:27:17.717253 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:17.713404 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" podUID="1fe658cf-6504-47e1-8640-cc6404e8a882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 22:27:18.067186 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:18.067095 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" Oct 30 22:27:18.067732 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:18.067703 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" Oct 30 22:27:18.069965 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:18.069945 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" Oct 30 22:27:18.711358 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:18.706615 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-xmr6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:27:18.711358 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:18.706682 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" podUID="1fe658cf-6504-47e1-8640-cc6404e8a882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 22:27:19.234965 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:19.234928 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-ngjmd"] Oct 30 22:27:19.362662 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:19.362623 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-n54lj"] Oct 30 22:27:19.710242 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:19.704647 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-xmr6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:27:19.710242 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:19.704711 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" podUID="1fe658cf-6504-47e1-8640-cc6404e8a882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 22:27:20.711280 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:20.709591 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-xmr6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:27:20.711280 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:20.709651 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" podUID="1fe658cf-6504-47e1-8640-cc6404e8a882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 22:27:21.281246 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:27:21.281167 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:27:21.709241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:21.707757 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-xmr6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:27:21.709241 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:21.707833 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" podUID="1fe658cf-6504-47e1-8640-cc6404e8a882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 22:27:22.704798 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:22.704762 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-xmr6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:27:22.705191 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:22.704814 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" podUID="1fe658cf-6504-47e1-8640-cc6404e8a882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 22:27:23.704761 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:23.704724 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-xmr6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:27:23.704938 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:23.704803 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" podUID="1fe658cf-6504-47e1-8640-cc6404e8a882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 22:27:24.235551 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:24.235505 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-ngjmd" podUID="a036216b-8457-4f47-8dc6-1399b627ff7f" containerName="payload-processing" containerID="cri-o://246f4c179d8dd99c0e8c96b15363c425f1d512c15d9442650180e8afa92c9cd6" gracePeriod=25 Oct 30 22:27:24.363284 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:24.363234 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-n54lj" podUID="69e2e901-0602-41fe-9321-12f0e2093e75" containerName="payload-pre-processing" containerID="cri-o://b075569687c7aef0c284a1577eacca921db6faac1d60dd4e25576d100677518f" gracePeriod=25 Oct 30 22:27:24.533963 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:24.533631 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-ngjmd" Oct 30 22:27:24.588133 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:24.587555 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a036216b-8457-4f47-8dc6-1399b627ff7f-kube-api-access-b2wnr\" (UniqueName: \"kubernetes.io/projected/a036216b-8457-4f47-8dc6-1399b627ff7f-kube-api-access-b2wnr\") pod \"a036216b-8457-4f47-8dc6-1399b627ff7f\" (UID: \"a036216b-8457-4f47-8dc6-1399b627ff7f\") " Oct 30 22:27:24.588133 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:24.587678 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a036216b-8457-4f47-8dc6-1399b627ff7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a036216b-8457-4f47-8dc6-1399b627ff7f-plugins-config-volume\") pod \"a036216b-8457-4f47-8dc6-1399b627ff7f\" (UID: \"a036216b-8457-4f47-8dc6-1399b627ff7f\") " Oct 30 22:27:24.588896 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:24.588579 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a036216b-8457-4f47-8dc6-1399b627ff7f-plugins-config-volume" pod "a036216b-8457-4f47-8dc6-1399b627ff7f" (UID: "a036216b-8457-4f47-8dc6-1399b627ff7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:24.588896 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:24.588690 5918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a036216b-8457-4f47-8dc6-1399b627ff7f-plugins-config-volume\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:24.601313 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:24.601264 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a036216b-8457-4f47-8dc6-1399b627ff7f-kube-api-access-b2wnr" pod "a036216b-8457-4f47-8dc6-1399b627ff7f" (UID: "a036216b-8457-4f47-8dc6-1399b627ff7f"). InnerVolumeSpecName "kube-api-access-b2wnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:24.664061 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:24.664027 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-n54lj" Oct 30 22:27:24.689716 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:24.689687 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69e2e901-0602-41fe-9321-12f0e2093e75-kube-api-access-8n92m\" (UniqueName: \"kubernetes.io/projected/69e2e901-0602-41fe-9321-12f0e2093e75-kube-api-access-8n92m\") pod \"69e2e901-0602-41fe-9321-12f0e2093e75\" (UID: \"69e2e901-0602-41fe-9321-12f0e2093e75\") " Oct 30 22:27:24.690148 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:24.690109 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69e2e901-0602-41fe-9321-12f0e2093e75-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69e2e901-0602-41fe-9321-12f0e2093e75-plugins-config-volume\") pod \"69e2e901-0602-41fe-9321-12f0e2093e75\" (UID: \"69e2e901-0602-41fe-9321-12f0e2093e75\") " Oct 30 22:27:24.690529 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:24.690489 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69e2e901-0602-41fe-9321-12f0e2093e75-plugins-config-volume" pod "69e2e901-0602-41fe-9321-12f0e2093e75" (UID: "69e2e901-0602-41fe-9321-12f0e2093e75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:24.691431 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:24.691406 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2wnr\" (UniqueName: \"kubernetes.io/projected/a036216b-8457-4f47-8dc6-1399b627ff7f-kube-api-access-b2wnr\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:24.691504 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:24.691436 5918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69e2e901-0602-41fe-9321-12f0e2093e75-plugins-config-volume\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:24.692679 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:24.692657 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69e2e901-0602-41fe-9321-12f0e2093e75-kube-api-access-8n92m" pod "69e2e901-0602-41fe-9321-12f0e2093e75" (UID: "69e2e901-0602-41fe-9321-12f0e2093e75"). InnerVolumeSpecName "kube-api-access-8n92m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:24.744295 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:24.744266 5918 generic.go:356] "Generic (PLEG): container finished" podID="69e2e901-0602-41fe-9321-12f0e2093e75" containerID="b075569687c7aef0c284a1577eacca921db6faac1d60dd4e25576d100677518f" exitCode=0 Oct 30 22:27:24.744724 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:24.744340 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-n54lj" Oct 30 22:27:24.744724 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:24.744355 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-n54lj" event={"ID":"69e2e901-0602-41fe-9321-12f0e2093e75","Type":"ContainerDied","Data":"b075569687c7aef0c284a1577eacca921db6faac1d60dd4e25576d100677518f"} Oct 30 22:27:24.744724 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:24.744423 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-n54lj" event={"ID":"69e2e901-0602-41fe-9321-12f0e2093e75","Type":"ContainerDied","Data":"c8bbb75766980739639bb552dfde5797f7a6a4f21656d2844ca9c2bd78bd86e8"} Oct 30 22:27:24.744724 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:24.744457 5918 scope.go:122] "RemoveContainer" containerID="b075569687c7aef0c284a1577eacca921db6faac1d60dd4e25576d100677518f" Oct 30 22:27:24.746347 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:24.746320 5918 generic.go:356] "Generic (PLEG): container finished" podID="a036216b-8457-4f47-8dc6-1399b627ff7f" containerID="246f4c179d8dd99c0e8c96b15363c425f1d512c15d9442650180e8afa92c9cd6" exitCode=0 Oct 30 22:27:24.746480 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:24.746359 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-ngjmd" event={"ID":"a036216b-8457-4f47-8dc6-1399b627ff7f","Type":"ContainerDied","Data":"246f4c179d8dd99c0e8c96b15363c425f1d512c15d9442650180e8afa92c9cd6"} Oct 30 22:27:24.746480 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:24.746381 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-ngjmd" Oct 30 22:27:24.746480 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:24.746398 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-ngjmd" event={"ID":"a036216b-8457-4f47-8dc6-1399b627ff7f","Type":"ContainerDied","Data":"f9ac13b06a10212e283c94cd88ab1851b1cbb66fa77804b3bc7454a71a30dafe"} Oct 30 22:27:24.756479 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:24.756460 5918 scope.go:122] "RemoveContainer" containerID="b075569687c7aef0c284a1577eacca921db6faac1d60dd4e25576d100677518f" Oct 30 22:27:24.756780 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:27:24.756758 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b075569687c7aef0c284a1577eacca921db6faac1d60dd4e25576d100677518f\": container with ID starting with b075569687c7aef0c284a1577eacca921db6faac1d60dd4e25576d100677518f not found: ID does not exist" containerID="b075569687c7aef0c284a1577eacca921db6faac1d60dd4e25576d100677518f" Oct 30 22:27:24.756953 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:24.756902 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b075569687c7aef0c284a1577eacca921db6faac1d60dd4e25576d100677518f"} err="failed to get container status \"b075569687c7aef0c284a1577eacca921db6faac1d60dd4e25576d100677518f\": rpc error: code = NotFound desc = could not find container \"b075569687c7aef0c284a1577eacca921db6faac1d60dd4e25576d100677518f\": container with ID starting with b075569687c7aef0c284a1577eacca921db6faac1d60dd4e25576d100677518f not found: ID does not exist" Oct 30 22:27:24.756953 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:24.756933 5918 scope.go:122] "RemoveContainer" containerID="246f4c179d8dd99c0e8c96b15363c425f1d512c15d9442650180e8afa92c9cd6" Oct 30 22:27:24.768787 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:24.768762 5918 scope.go:122] "RemoveContainer" containerID="246f4c179d8dd99c0e8c96b15363c425f1d512c15d9442650180e8afa92c9cd6" Oct 30 22:27:24.769067 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:27:24.769046 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"246f4c179d8dd99c0e8c96b15363c425f1d512c15d9442650180e8afa92c9cd6\": container with ID starting with 246f4c179d8dd99c0e8c96b15363c425f1d512c15d9442650180e8afa92c9cd6 not found: ID does not exist" containerID="246f4c179d8dd99c0e8c96b15363c425f1d512c15d9442650180e8afa92c9cd6" Oct 30 22:27:24.769146 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:24.769076 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"246f4c179d8dd99c0e8c96b15363c425f1d512c15d9442650180e8afa92c9cd6"} err="failed to get container status \"246f4c179d8dd99c0e8c96b15363c425f1d512c15d9442650180e8afa92c9cd6\": rpc error: code = NotFound desc = could not find container \"246f4c179d8dd99c0e8c96b15363c425f1d512c15d9442650180e8afa92c9cd6\": container with ID starting with 246f4c179d8dd99c0e8c96b15363c425f1d512c15d9442650180e8afa92c9cd6 not found: ID does not exist" Oct 30 22:27:24.792930 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:24.792899 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n92m\" (UniqueName: \"kubernetes.io/projected/69e2e901-0602-41fe-9321-12f0e2093e75-kube-api-access-8n92m\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:24.898782 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:24.898738 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-ngjmd"] Oct 30 22:27:24.904359 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:24.904332 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-422c1029-769464bdcc-ngjmd"] Oct 30 22:27:25.019485 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.019374 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-n54lj"] Oct 30 22:27:25.025416 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.025384 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-422c1029-5cc6f68c9-n54lj"] Oct 30 22:27:25.120583 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.120552 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" Oct 30 22:27:25.121074 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.121047 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" Oct 30 22:27:25.121824 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.121802 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" Oct 30 22:27:25.411824 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.409378 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf"] Oct 30 22:27:25.411824 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.410237 5918 cpu_manager.go:414] "containerMap: removing container" podUID="a036216b-8457-4f47-8dc6-1399b627ff7f" containerName="payload-processing" Oct 30 22:27:25.411824 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.410257 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a036216b-8457-4f47-8dc6-1399b627ff7f" containerName="payload-processing" Oct 30 22:27:25.411824 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.410307 5918 cpu_manager.go:414] "containerMap: removing container" podUID="69e2e901-0602-41fe-9321-12f0e2093e75" containerName="payload-pre-processing" Oct 30 22:27:25.411824 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.410317 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69e2e901-0602-41fe-9321-12f0e2093e75" containerName="payload-pre-processing" Oct 30 22:27:25.411824 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.410473 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="69e2e901-0602-41fe-9321-12f0e2093e75" containerName="payload-pre-processing" Oct 30 22:27:25.411824 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.410496 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a036216b-8457-4f47-8dc6-1399b627ff7f" containerName="payload-processing" Oct 30 22:27:25.418320 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.417905 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" Oct 30 22:27:25.474733 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.474702 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf"] Oct 30 22:27:25.501468 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.501437 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f4e41b-50c7-441c-8ff8-6c815143b79b-istio-podinfo\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf\" (UID: \"97f4e41b-50c7-441c-8ff8-6c815143b79b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" Oct 30 22:27:25.501633 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.501575 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f4e41b-50c7-441c-8ff8-6c815143b79b-istio-envoy\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf\" (UID: \"97f4e41b-50c7-441c-8ff8-6c815143b79b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" Oct 30 22:27:25.501633 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.501598 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97f4e41b-50c7-441c-8ff8-6c815143b79b-istio-token\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf\" (UID: \"97f4e41b-50c7-441c-8ff8-6c815143b79b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" Oct 30 22:27:25.501742 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.501713 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f4e41b-50c7-441c-8ff8-6c815143b79b-credential-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf\" (UID: \"97f4e41b-50c7-441c-8ff8-6c815143b79b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" Oct 30 22:27:25.501807 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.501789 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f4e41b-50c7-441c-8ff8-6c815143b79b-istio-data\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf\" (UID: \"97f4e41b-50c7-441c-8ff8-6c815143b79b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" Oct 30 22:27:25.501909 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.501893 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f4e41b-50c7-441c-8ff8-6c815143b79b-workload-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf\" (UID: \"97f4e41b-50c7-441c-8ff8-6c815143b79b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" Oct 30 22:27:25.501958 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.501944 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f4e41b-50c7-441c-8ff8-6c815143b79b-istiod-ca-cert\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf\" (UID: \"97f4e41b-50c7-441c-8ff8-6c815143b79b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" Oct 30 22:27:25.501991 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.501975 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnbq6\" (UniqueName: \"kubernetes.io/projected/97f4e41b-50c7-441c-8ff8-6c815143b79b-kube-api-access-wnbq6\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf\" (UID: \"97f4e41b-50c7-441c-8ff8-6c815143b79b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" Oct 30 22:27:25.502094 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.502079 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f4e41b-50c7-441c-8ff8-6c815143b79b-workload-certs\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf\" (UID: \"97f4e41b-50c7-441c-8ff8-6c815143b79b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" Oct 30 22:27:25.603494 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.603459 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f4e41b-50c7-441c-8ff8-6c815143b79b-workload-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf\" (UID: \"97f4e41b-50c7-441c-8ff8-6c815143b79b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" Oct 30 22:27:25.603672 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.603502 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f4e41b-50c7-441c-8ff8-6c815143b79b-istiod-ca-cert\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf\" (UID: \"97f4e41b-50c7-441c-8ff8-6c815143b79b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" Oct 30 22:27:25.603672 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.603525 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnbq6\" (UniqueName: \"kubernetes.io/projected/97f4e41b-50c7-441c-8ff8-6c815143b79b-kube-api-access-wnbq6\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf\" (UID: \"97f4e41b-50c7-441c-8ff8-6c815143b79b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" Oct 30 22:27:25.603672 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.603551 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f4e41b-50c7-441c-8ff8-6c815143b79b-workload-certs\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf\" (UID: \"97f4e41b-50c7-441c-8ff8-6c815143b79b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" Oct 30 22:27:25.603672 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.603597 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f4e41b-50c7-441c-8ff8-6c815143b79b-istio-podinfo\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf\" (UID: \"97f4e41b-50c7-441c-8ff8-6c815143b79b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" Oct 30 22:27:25.603898 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.603706 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f4e41b-50c7-441c-8ff8-6c815143b79b-istio-envoy\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf\" (UID: \"97f4e41b-50c7-441c-8ff8-6c815143b79b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" Oct 30 22:27:25.603898 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.603734 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97f4e41b-50c7-441c-8ff8-6c815143b79b-istio-token\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf\" (UID: \"97f4e41b-50c7-441c-8ff8-6c815143b79b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" Oct 30 22:27:25.603898 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.603820 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f4e41b-50c7-441c-8ff8-6c815143b79b-credential-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf\" (UID: \"97f4e41b-50c7-441c-8ff8-6c815143b79b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" Oct 30 22:27:25.603898 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.603865 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f4e41b-50c7-441c-8ff8-6c815143b79b-istio-data\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf\" (UID: \"97f4e41b-50c7-441c-8ff8-6c815143b79b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" Oct 30 22:27:25.604104 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.603963 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f4e41b-50c7-441c-8ff8-6c815143b79b-workload-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf\" (UID: \"97f4e41b-50c7-441c-8ff8-6c815143b79b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" Oct 30 22:27:25.604188 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.604158 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f4e41b-50c7-441c-8ff8-6c815143b79b-credential-socket\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf\" (UID: \"97f4e41b-50c7-441c-8ff8-6c815143b79b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" Oct 30 22:27:25.604278 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.604258 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f4e41b-50c7-441c-8ff8-6c815143b79b-workload-certs\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf\" (UID: \"97f4e41b-50c7-441c-8ff8-6c815143b79b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" Oct 30 22:27:25.604319 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.604274 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f4e41b-50c7-441c-8ff8-6c815143b79b-istio-data\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf\" (UID: \"97f4e41b-50c7-441c-8ff8-6c815143b79b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" Oct 30 22:27:25.604381 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.604330 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f4e41b-50c7-441c-8ff8-6c815143b79b-istiod-ca-cert\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf\" (UID: \"97f4e41b-50c7-441c-8ff8-6c815143b79b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" Oct 30 22:27:25.606176 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.606152 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f4e41b-50c7-441c-8ff8-6c815143b79b-istio-envoy\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf\" (UID: \"97f4e41b-50c7-441c-8ff8-6c815143b79b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" Oct 30 22:27:25.606433 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.606411 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f4e41b-50c7-441c-8ff8-6c815143b79b-istio-podinfo\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf\" (UID: \"97f4e41b-50c7-441c-8ff8-6c815143b79b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" Oct 30 22:27:25.667922 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.667853 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97f4e41b-50c7-441c-8ff8-6c815143b79b-istio-token\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf\" (UID: \"97f4e41b-50c7-441c-8ff8-6c815143b79b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" Oct 30 22:27:25.668272 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.668209 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnbq6\" (UniqueName: \"kubernetes.io/projected/97f4e41b-50c7-441c-8ff8-6c815143b79b-kube-api-access-wnbq6\") pod \"e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf\" (UID: \"97f4e41b-50c7-441c-8ff8-6c815143b79b\") " pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" Oct 30 22:27:25.734577 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.734545 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" Oct 30 22:27:25.856024 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.855987 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69e2e901-0602-41fe-9321-12f0e2093e75" path="/var/lib/kubelet/pods/69e2e901-0602-41fe-9321-12f0e2093e75/volumes" Oct 30 22:27:25.856677 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:25.856655 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a036216b-8457-4f47-8dc6-1399b627ff7f" path="/var/lib/kubelet/pods/a036216b-8457-4f47-8dc6-1399b627ff7f/volumes" Oct 30 22:27:26.168713 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:26.168686 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf"] Oct 30 22:27:26.170066 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:27:26.170044 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f4e41b_50c7_441c_8ff8_6c815143b79b.slice/crio-5acbcea12adf28865a62c403a12968a32d336a9690143be3edfb27e2e6e51a63 WatchSource:0}: Error finding container 5acbcea12adf28865a62c403a12968a32d336a9690143be3edfb27e2e6e51a63: Status 404 returned error can't find the container with id 5acbcea12adf28865a62c403a12968a32d336a9690143be3edfb27e2e6e51a63 Oct 30 22:27:26.172619 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:26.172575 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:27:26.172738 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:26.172657 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:27:26.172738 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:26.172703 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:27:26.762095 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:26.762053 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" event={"ID":"97f4e41b-50c7-441c-8ff8-6c815143b79b","Type":"ContainerStarted","Data":"25bcbdee8822855109eeeb2cfb7ccc96d78849f535cabc5ab6710be39b35c7a9"} Oct 30 22:27:26.762527 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:26.762134 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" event={"ID":"97f4e41b-50c7-441c-8ff8-6c815143b79b","Type":"ContainerStarted","Data":"5acbcea12adf28865a62c403a12968a32d336a9690143be3edfb27e2e6e51a63"} Oct 30 22:27:26.941177 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:26.941127 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" podStartSLOduration=1.941114478 podStartE2EDuration="1.941114478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:26.938622222 +0000 UTC m=+2207.209320307" watchObservedRunningTime="2026-10-30 22:27:26.941114478 +0000 UTC m=+2207.211812561" Oct 30 22:27:27.260377 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:27.260341 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2"] Oct 30 22:27:27.281726 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:27.281695 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc"] Oct 30 22:27:27.734854 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:27.734812 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" Oct 30 22:27:27.736204 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:27.736171 5918 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 22:27:27.736384 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:27.736261 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" podUID="97f4e41b-50c7-441c-8ff8-6c815143b79b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 22:27:28.735687 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:28.735647 5918 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 22:27:28.736191 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:28.735705 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" podUID="97f4e41b-50c7-441c-8ff8-6c815143b79b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 22:27:29.735201 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:29.735163 5918 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 22:27:29.735420 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:29.735257 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" podUID="97f4e41b-50c7-441c-8ff8-6c815143b79b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 22:27:30.735819 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:30.735777 5918 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 22:27:30.736347 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:30.735842 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" podUID="97f4e41b-50c7-441c-8ff8-6c815143b79b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 22:27:31.735878 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:31.735840 5918 patch_prober.go:29] interesting pod/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 22:27:31.736327 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:31.735903 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" podUID="97f4e41b-50c7-441c-8ff8-6c815143b79b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 22:27:32.740418 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:32.740385 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" Oct 30 22:27:32.740939 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:32.740876 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" Oct 30 22:27:32.741597 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:32.741574 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" Oct 30 22:27:39.014413 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:39.014374 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4"] Oct 30 22:27:39.019536 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:39.019500 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4" podUID="f39ab9c9-c404-49e7-8273-4e6e50656378" containerName="maas-api" containerID="cri-o://9f0f9590386a53172cde3857c16f2c6250716bd5d91305686d398dc0655c6b65" gracePeriod=30 Oct 30 22:27:39.340149 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:39.340123 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4" Oct 30 22:27:39.418608 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:39.418552 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f39ab9c9-c404-49e7-8273-4e6e50656378-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f39ab9c9-c404-49e7-8273-4e6e50656378-maas-api-tls\") pod \"f39ab9c9-c404-49e7-8273-4e6e50656378\" (UID: \"f39ab9c9-c404-49e7-8273-4e6e50656378\") " Oct 30 22:27:39.418823 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:39.418698 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f39ab9c9-c404-49e7-8273-4e6e50656378-kube-api-access-t8xfs\" (UniqueName: \"kubernetes.io/projected/f39ab9c9-c404-49e7-8273-4e6e50656378-kube-api-access-t8xfs\") pod \"f39ab9c9-c404-49e7-8273-4e6e50656378\" (UID: \"f39ab9c9-c404-49e7-8273-4e6e50656378\") " Oct 30 22:27:39.418823 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:39.418805 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f39ab9c9-c404-49e7-8273-4e6e50656378-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f39ab9c9-c404-49e7-8273-4e6e50656378-metrics-certs\") pod \"f39ab9c9-c404-49e7-8273-4e6e50656378\" (UID: \"f39ab9c9-c404-49e7-8273-4e6e50656378\") " Oct 30 22:27:39.418940 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:39.418869 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f39ab9c9-c404-49e7-8273-4e6e50656378-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f39ab9c9-c404-49e7-8273-4e6e50656378-openshift-service-ca\") pod \"f39ab9c9-c404-49e7-8273-4e6e50656378\" (UID: \"f39ab9c9-c404-49e7-8273-4e6e50656378\") " Oct 30 22:27:39.419919 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:39.419890 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f39ab9c9-c404-49e7-8273-4e6e50656378-openshift-service-ca" pod "f39ab9c9-c404-49e7-8273-4e6e50656378" (UID: "f39ab9c9-c404-49e7-8273-4e6e50656378"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:39.423595 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:39.423564 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f39ab9c9-c404-49e7-8273-4e6e50656378-maas-api-tls" pod "f39ab9c9-c404-49e7-8273-4e6e50656378" (UID: "f39ab9c9-c404-49e7-8273-4e6e50656378"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:27:39.432943 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:39.432878 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f39ab9c9-c404-49e7-8273-4e6e50656378-metrics-certs" pod "f39ab9c9-c404-49e7-8273-4e6e50656378" (UID: "f39ab9c9-c404-49e7-8273-4e6e50656378"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:27:39.440779 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:39.440741 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f39ab9c9-c404-49e7-8273-4e6e50656378-kube-api-access-t8xfs" pod "f39ab9c9-c404-49e7-8273-4e6e50656378" (UID: "f39ab9c9-c404-49e7-8273-4e6e50656378"). InnerVolumeSpecName "kube-api-access-t8xfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:39.520403 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:39.520372 5918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f39ab9c9-c404-49e7-8273-4e6e50656378-maas-api-tls\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:39.520403 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:39.520401 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8xfs\" (UniqueName: \"kubernetes.io/projected/f39ab9c9-c404-49e7-8273-4e6e50656378-kube-api-access-t8xfs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:39.520611 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:39.520416 5918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f39ab9c9-c404-49e7-8273-4e6e50656378-metrics-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:39.520611 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:39.520429 5918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f39ab9c9-c404-49e7-8273-4e6e50656378-openshift-service-ca\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:39.868547 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:39.868510 5918 generic.go:356] "Generic (PLEG): container finished" podID="f39ab9c9-c404-49e7-8273-4e6e50656378" containerID="9f0f9590386a53172cde3857c16f2c6250716bd5d91305686d398dc0655c6b65" exitCode=0 Oct 30 22:27:39.868685 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:39.868556 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4" Oct 30 22:27:39.868685 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:39.868590 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4" event={"ID":"f39ab9c9-c404-49e7-8273-4e6e50656378","Type":"ContainerDied","Data":"9f0f9590386a53172cde3857c16f2c6250716bd5d91305686d398dc0655c6b65"} Oct 30 22:27:39.868685 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:39.868658 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4" event={"ID":"f39ab9c9-c404-49e7-8273-4e6e50656378","Type":"ContainerDied","Data":"0a44a7e6ef47e85595c82584fb921d99d7d70adb7b35c48d2af5208d56646bb1"} Oct 30 22:27:39.868807 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:39.868694 5918 scope.go:122] "RemoveContainer" containerID="9f0f9590386a53172cde3857c16f2c6250716bd5d91305686d398dc0655c6b65" Oct 30 22:27:39.880707 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:39.880645 5918 scope.go:122] "RemoveContainer" containerID="9f0f9590386a53172cde3857c16f2c6250716bd5d91305686d398dc0655c6b65" Oct 30 22:27:39.880977 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:27:39.880954 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f0f9590386a53172cde3857c16f2c6250716bd5d91305686d398dc0655c6b65\": container with ID starting with 9f0f9590386a53172cde3857c16f2c6250716bd5d91305686d398dc0655c6b65 not found: ID does not exist" containerID="9f0f9590386a53172cde3857c16f2c6250716bd5d91305686d398dc0655c6b65" Oct 30 22:27:39.881048 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:39.880988 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f0f9590386a53172cde3857c16f2c6250716bd5d91305686d398dc0655c6b65"} err="failed to get container status \"9f0f9590386a53172cde3857c16f2c6250716bd5d91305686d398dc0655c6b65\": rpc error: code = NotFound desc = could not find container \"9f0f9590386a53172cde3857c16f2c6250716bd5d91305686d398dc0655c6b65\": container with ID starting with 9f0f9590386a53172cde3857c16f2c6250716bd5d91305686d398dc0655c6b65 not found: ID does not exist" Oct 30 22:27:40.046317 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:40.046264 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4"] Oct 30 22:27:40.094122 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:40.094079 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-422c1029-6d5f756bb7-zmpd4"] Oct 30 22:27:41.851924 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:41.851885 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f39ab9c9-c404-49e7-8273-4e6e50656378" path="/var/lib/kubelet/pods/f39ab9c9-c404-49e7-8273-4e6e50656378/volumes" Oct 30 22:27:42.282613 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.282540 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" podUID="05ab563b-c4c2-4fa7-85e4-423141587664" containerName="main" containerID="cri-o://014f7ad8d5c88a866b43cf490faf12cd4c9fdcfc1e82acff8afa65b3c1a4d96d" gracePeriod=45 Oct 30 22:27:42.327263 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.327207 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" podUID="21ef7a75-4005-4d56-9b35-783dc4d9cd2a" containerName="main" containerID="cri-o://9dc189e679b85d3f1848c3085a7f7fe37aa4d302f3af979411743e95a16ac390" gracePeriod=45 Oct 30 22:27:42.399542 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.399509 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" podUID="21ef7a75-4005-4d56-9b35-783dc4d9cd2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": dial tcp 10.134.0.48:8000: connect: connection refused" Oct 30 22:27:42.613549 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.613524 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" Oct 30 22:27:42.616711 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.616692 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" Oct 30 22:27:42.664223 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.664195 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ab563b-c4c2-4fa7-85e4-423141587664-dshm\" (UniqueName: \"kubernetes.io/empty-dir/05ab563b-c4c2-4fa7-85e4-423141587664-dshm\") pod \"05ab563b-c4c2-4fa7-85e4-423141587664\" (UID: \"05ab563b-c4c2-4fa7-85e4-423141587664\") " Oct 30 22:27:42.664380 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.664284 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05ab563b-c4c2-4fa7-85e4-423141587664-kube-api-access-tc2c6\" (UniqueName: \"kubernetes.io/projected/05ab563b-c4c2-4fa7-85e4-423141587664-kube-api-access-tc2c6\") pod \"05ab563b-c4c2-4fa7-85e4-423141587664\" (UID: \"05ab563b-c4c2-4fa7-85e4-423141587664\") " Oct 30 22:27:42.664380 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.664331 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-tls-certs\" (UniqueName: \"kubernetes.io/secret/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-tls-certs\") pod \"21ef7a75-4005-4d56-9b35-783dc4d9cd2a\" (UID: \"21ef7a75-4005-4d56-9b35-783dc4d9cd2a\") " Oct 30 22:27:42.664380 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.664356 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05ab563b-c4c2-4fa7-85e4-423141587664-tls-certs\" (UniqueName: \"kubernetes.io/secret/05ab563b-c4c2-4fa7-85e4-423141587664-tls-certs\") pod \"05ab563b-c4c2-4fa7-85e4-423141587664\" (UID: \"05ab563b-c4c2-4fa7-85e4-423141587664\") " Oct 30 22:27:42.664380 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.664376 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-tmp-dir\") pod \"21ef7a75-4005-4d56-9b35-783dc4d9cd2a\" (UID: \"21ef7a75-4005-4d56-9b35-783dc4d9cd2a\") " Oct 30 22:27:42.664613 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.664423 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ab563b-c4c2-4fa7-85e4-423141587664-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05ab563b-c4c2-4fa7-85e4-423141587664-tmp-dir\") pod \"05ab563b-c4c2-4fa7-85e4-423141587664\" (UID: \"05ab563b-c4c2-4fa7-85e4-423141587664\") " Oct 30 22:27:42.664613 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.664482 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-dshm\") pod \"21ef7a75-4005-4d56-9b35-783dc4d9cd2a\" (UID: \"21ef7a75-4005-4d56-9b35-783dc4d9cd2a\") " Oct 30 22:27:42.664613 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.664555 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-model-cache\") pod \"21ef7a75-4005-4d56-9b35-783dc4d9cd2a\" (UID: \"21ef7a75-4005-4d56-9b35-783dc4d9cd2a\") " Oct 30 22:27:42.664613 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.664605 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ab563b-c4c2-4fa7-85e4-423141587664-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05ab563b-c4c2-4fa7-85e4-423141587664-model-cache\") pod \"05ab563b-c4c2-4fa7-85e4-423141587664\" (UID: \"05ab563b-c4c2-4fa7-85e4-423141587664\") " Oct 30 22:27:42.664807 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.664639 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-kube-api-access-jjfnr\" (UniqueName: \"kubernetes.io/projected/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-kube-api-access-jjfnr\") pod \"21ef7a75-4005-4d56-9b35-783dc4d9cd2a\" (UID: \"21ef7a75-4005-4d56-9b35-783dc4d9cd2a\") " Oct 30 22:27:42.664807 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.664672 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ab563b-c4c2-4fa7-85e4-423141587664-home\" (UniqueName: \"kubernetes.io/empty-dir/05ab563b-c4c2-4fa7-85e4-423141587664-home\") pod \"05ab563b-c4c2-4fa7-85e4-423141587664\" (UID: \"05ab563b-c4c2-4fa7-85e4-423141587664\") " Oct 30 22:27:42.664807 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.664684 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-tmp-dir" pod "21ef7a75-4005-4d56-9b35-783dc4d9cd2a" (UID: "21ef7a75-4005-4d56-9b35-783dc4d9cd2a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:42.664807 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.664761 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-home\" (UniqueName: \"kubernetes.io/empty-dir/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-home\") pod \"21ef7a75-4005-4d56-9b35-783dc4d9cd2a\" (UID: \"21ef7a75-4005-4d56-9b35-783dc4d9cd2a\") " Oct 30 22:27:42.665187 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.664879 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ab563b-c4c2-4fa7-85e4-423141587664-model-cache" pod "05ab563b-c4c2-4fa7-85e4-423141587664" (UID: "05ab563b-c4c2-4fa7-85e4-423141587664"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:42.665187 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.664874 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-model-cache" pod "21ef7a75-4005-4d56-9b35-783dc4d9cd2a" (UID: "21ef7a75-4005-4d56-9b35-783dc4d9cd2a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:42.665350 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.665315 5918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-tmp-dir\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:42.665350 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.665337 5918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-model-cache\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:42.665474 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.665351 5918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05ab563b-c4c2-4fa7-85e4-423141587664-model-cache\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:42.665474 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.665350 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ab563b-c4c2-4fa7-85e4-423141587664-tmp-dir" pod "05ab563b-c4c2-4fa7-85e4-423141587664" (UID: "05ab563b-c4c2-4fa7-85e4-423141587664"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:42.666496 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.665686 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ab563b-c4c2-4fa7-85e4-423141587664-home" pod "05ab563b-c4c2-4fa7-85e4-423141587664" (UID: "05ab563b-c4c2-4fa7-85e4-423141587664"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:42.666496 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.665709 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-home" pod "21ef7a75-4005-4d56-9b35-783dc4d9cd2a" (UID: "21ef7a75-4005-4d56-9b35-783dc4d9cd2a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:42.667329 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.667165 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ab563b-c4c2-4fa7-85e4-423141587664-dshm" pod "05ab563b-c4c2-4fa7-85e4-423141587664" (UID: "05ab563b-c4c2-4fa7-85e4-423141587664"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:42.667329 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.667287 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-kube-api-access-jjfnr" pod "21ef7a75-4005-4d56-9b35-783dc4d9cd2a" (UID: "21ef7a75-4005-4d56-9b35-783dc4d9cd2a"). InnerVolumeSpecName "kube-api-access-jjfnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:42.667504 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.667346 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-tls-certs" pod "21ef7a75-4005-4d56-9b35-783dc4d9cd2a" (UID: "21ef7a75-4005-4d56-9b35-783dc4d9cd2a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:27:42.667504 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.667353 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05ab563b-c4c2-4fa7-85e4-423141587664-kube-api-access-tc2c6" pod "05ab563b-c4c2-4fa7-85e4-423141587664" (UID: "05ab563b-c4c2-4fa7-85e4-423141587664"). InnerVolumeSpecName "kube-api-access-tc2c6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:42.667504 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.667366 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05ab563b-c4c2-4fa7-85e4-423141587664-tls-certs" pod "05ab563b-c4c2-4fa7-85e4-423141587664" (UID: "05ab563b-c4c2-4fa7-85e4-423141587664"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:27:42.667671 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.667658 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-dshm" pod "21ef7a75-4005-4d56-9b35-783dc4d9cd2a" (UID: "21ef7a75-4005-4d56-9b35-783dc4d9cd2a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:27:42.766397 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.766357 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjfnr\" (UniqueName: \"kubernetes.io/projected/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-kube-api-access-jjfnr\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:42.766397 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.766386 5918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05ab563b-c4c2-4fa7-85e4-423141587664-home\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:42.766397 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.766395 5918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-home\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:42.766397 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.766403 5918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05ab563b-c4c2-4fa7-85e4-423141587664-dshm\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:42.766750 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.766413 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc2c6\" (UniqueName: \"kubernetes.io/projected/05ab563b-c4c2-4fa7-85e4-423141587664-kube-api-access-tc2c6\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:42.766750 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.766421 5918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-tls-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:42.766750 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.766430 5918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05ab563b-c4c2-4fa7-85e4-423141587664-tls-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:42.766750 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.766437 5918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05ab563b-c4c2-4fa7-85e4-423141587664-tmp-dir\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:42.766750 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.766444 5918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21ef7a75-4005-4d56-9b35-783dc4d9cd2a-dshm\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:42.892739 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.892645 5918 generic.go:356] "Generic (PLEG): container finished" podID="05ab563b-c4c2-4fa7-85e4-423141587664" containerID="014f7ad8d5c88a866b43cf490faf12cd4c9fdcfc1e82acff8afa65b3c1a4d96d" exitCode=0 Oct 30 22:27:42.892739 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.892696 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" Oct 30 22:27:42.893331 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.892815 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" event={"ID":"05ab563b-c4c2-4fa7-85e4-423141587664","Type":"ContainerDied","Data":"014f7ad8d5c88a866b43cf490faf12cd4c9fdcfc1e82acff8afa65b3c1a4d96d"} Oct 30 22:27:42.893331 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.892855 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2" event={"ID":"05ab563b-c4c2-4fa7-85e4-423141587664","Type":"ContainerDied","Data":"f80482c7e8ebd6dba8823f6014803286bd4cb2db08eb044c2c4be1378d51f4a3"} Oct 30 22:27:42.893331 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.892878 5918 scope.go:122] "RemoveContainer" containerID="014f7ad8d5c88a866b43cf490faf12cd4c9fdcfc1e82acff8afa65b3c1a4d96d" Oct 30 22:27:42.894898 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.894840 5918 generic.go:356] "Generic (PLEG): container finished" podID="21ef7a75-4005-4d56-9b35-783dc4d9cd2a" containerID="9dc189e679b85d3f1848c3085a7f7fe37aa4d302f3af979411743e95a16ac390" exitCode=0 Oct 30 22:27:42.894898 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.894899 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" Oct 30 22:27:42.895149 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.894957 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" event={"ID":"21ef7a75-4005-4d56-9b35-783dc4d9cd2a","Type":"ContainerDied","Data":"9dc189e679b85d3f1848c3085a7f7fe37aa4d302f3af979411743e95a16ac390"} Oct 30 22:27:42.895149 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.894985 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc" event={"ID":"21ef7a75-4005-4d56-9b35-783dc4d9cd2a","Type":"ContainerDied","Data":"d704f3b0464688bdd244e7bab42514c8313a0953dc3140d89267c7ac7fceb73a"} Oct 30 22:27:42.906524 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.906281 5918 scope.go:122] "RemoveContainer" containerID="014f7ad8d5c88a866b43cf490faf12cd4c9fdcfc1e82acff8afa65b3c1a4d96d" Oct 30 22:27:42.906683 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:27:42.906579 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"014f7ad8d5c88a866b43cf490faf12cd4c9fdcfc1e82acff8afa65b3c1a4d96d\": container with ID starting with 014f7ad8d5c88a866b43cf490faf12cd4c9fdcfc1e82acff8afa65b3c1a4d96d not found: ID does not exist" containerID="014f7ad8d5c88a866b43cf490faf12cd4c9fdcfc1e82acff8afa65b3c1a4d96d" Oct 30 22:27:42.906683 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.906625 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"014f7ad8d5c88a866b43cf490faf12cd4c9fdcfc1e82acff8afa65b3c1a4d96d"} err="failed to get container status \"014f7ad8d5c88a866b43cf490faf12cd4c9fdcfc1e82acff8afa65b3c1a4d96d\": rpc error: code = NotFound desc = could not find container \"014f7ad8d5c88a866b43cf490faf12cd4c9fdcfc1e82acff8afa65b3c1a4d96d\": container with ID starting with 014f7ad8d5c88a866b43cf490faf12cd4c9fdcfc1e82acff8afa65b3c1a4d96d not found: ID does not exist" Oct 30 22:27:42.906683 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.906647 5918 scope.go:122] "RemoveContainer" containerID="9dc189e679b85d3f1848c3085a7f7fe37aa4d302f3af979411743e95a16ac390" Oct 30 22:27:42.917738 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.917721 5918 scope.go:122] "RemoveContainer" containerID="9dc189e679b85d3f1848c3085a7f7fe37aa4d302f3af979411743e95a16ac390" Oct 30 22:27:42.918085 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:27:42.918054 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dc189e679b85d3f1848c3085a7f7fe37aa4d302f3af979411743e95a16ac390\": container with ID starting with 9dc189e679b85d3f1848c3085a7f7fe37aa4d302f3af979411743e95a16ac390 not found: ID does not exist" containerID="9dc189e679b85d3f1848c3085a7f7fe37aa4d302f3af979411743e95a16ac390" Oct 30 22:27:42.918161 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:42.918095 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dc189e679b85d3f1848c3085a7f7fe37aa4d302f3af979411743e95a16ac390"} err="failed to get container status \"9dc189e679b85d3f1848c3085a7f7fe37aa4d302f3af979411743e95a16ac390\": rpc error: code = NotFound desc = could not find container \"9dc189e679b85d3f1848c3085a7f7fe37aa4d302f3af979411743e95a16ac390\": container with ID starting with 9dc189e679b85d3f1848c3085a7f7fe37aa4d302f3af979411743e95a16ac390 not found: ID does not exist" Oct 30 22:27:43.096774 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:43.096739 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc"] Oct 30 22:27:43.101293 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:43.101266 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-9cff60/simulator-w1-9cff60-kserve-7c7f9c446b-n79jc"] Oct 30 22:27:43.215040 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:43.215003 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2"] Oct 30 22:27:43.221559 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:43.221534 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-9cff60/distinct-w1-9cff60-kserve-759f8f574b-hcgn2"] Oct 30 22:27:43.851851 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:43.851822 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05ab563b-c4c2-4fa7-85e4-423141587664" path="/var/lib/kubelet/pods/05ab563b-c4c2-4fa7-85e4-423141587664/volumes" Oct 30 22:27:43.852363 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:43.852344 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21ef7a75-4005-4d56-9b35-783dc4d9cd2a" path="/var/lib/kubelet/pods/21ef7a75-4005-4d56-9b35-783dc4d9cd2a/volumes" Oct 30 22:27:44.476542 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.476503 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb"] Oct 30 22:27:44.477047 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.477033 5918 cpu_manager.go:414] "containerMap: removing container" podUID="f39ab9c9-c404-49e7-8273-4e6e50656378" containerName="maas-api" Oct 30 22:27:44.477086 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.477048 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f39ab9c9-c404-49e7-8273-4e6e50656378" containerName="maas-api" Oct 30 22:27:44.477086 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.477080 5918 cpu_manager.go:414] "containerMap: removing container" podUID="21ef7a75-4005-4d56-9b35-783dc4d9cd2a" containerName="main" Oct 30 22:27:44.477086 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.477085 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21ef7a75-4005-4d56-9b35-783dc4d9cd2a" containerName="main" Oct 30 22:27:44.477193 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.477104 5918 cpu_manager.go:414] "containerMap: removing container" podUID="05ab563b-c4c2-4fa7-85e4-423141587664" containerName="main" Oct 30 22:27:44.477193 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.477110 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05ab563b-c4c2-4fa7-85e4-423141587664" containerName="main" Oct 30 22:27:44.477193 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.477177 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="21ef7a75-4005-4d56-9b35-783dc4d9cd2a" containerName="main" Oct 30 22:27:44.477193 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.477186 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="05ab563b-c4c2-4fa7-85e4-423141587664" containerName="main" Oct 30 22:27:44.477342 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.477200 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f39ab9c9-c404-49e7-8273-4e6e50656378" containerName="maas-api" Oct 30 22:27:44.481837 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.481818 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:27:44.544432 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.540973 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d59c58-openshift-default-dockercfg-tlqqx\"" Oct 30 22:27:44.588086 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.588054 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvqtn\" (UniqueName: \"kubernetes.io/projected/e5345433-337f-4434-a218-a9880269ddc5-kube-api-access-tvqtn\") pod \"e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb\" (UID: \"e5345433-337f-4434-a218-a9880269ddc5\") " pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:27:44.588314 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.588115 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5345433-337f-4434-a218-a9880269ddc5-credential-socket\") pod \"e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb\" (UID: \"e5345433-337f-4434-a218-a9880269ddc5\") " pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:27:44.588314 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.588283 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5345433-337f-4434-a218-a9880269ddc5-istio-envoy\") pod \"e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb\" (UID: \"e5345433-337f-4434-a218-a9880269ddc5\") " pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:27:44.588455 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.588389 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5345433-337f-4434-a218-a9880269ddc5-workload-socket\") pod \"e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb\" (UID: \"e5345433-337f-4434-a218-a9880269ddc5\") " pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:27:44.588518 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.588496 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5345433-337f-4434-a218-a9880269ddc5-istio-data\") pod \"e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb\" (UID: \"e5345433-337f-4434-a218-a9880269ddc5\") " pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:27:44.588554 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.588536 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5345433-337f-4434-a218-a9880269ddc5-istiod-ca-cert\") pod \"e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb\" (UID: \"e5345433-337f-4434-a218-a9880269ddc5\") " pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:27:44.588635 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.588617 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5345433-337f-4434-a218-a9880269ddc5-istio-podinfo\") pod \"e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb\" (UID: \"e5345433-337f-4434-a218-a9880269ddc5\") " pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:27:44.588725 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.588711 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5345433-337f-4434-a218-a9880269ddc5-workload-certs\") pod \"e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb\" (UID: \"e5345433-337f-4434-a218-a9880269ddc5\") " pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:27:44.588807 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.588792 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5345433-337f-4434-a218-a9880269ddc5-istio-token\") pod \"e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb\" (UID: \"e5345433-337f-4434-a218-a9880269ddc5\") " pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:27:44.595486 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.595456 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb"] Oct 30 22:27:44.690538 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.690506 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5345433-337f-4434-a218-a9880269ddc5-credential-socket\") pod \"e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb\" (UID: \"e5345433-337f-4434-a218-a9880269ddc5\") " pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:27:44.690736 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.690555 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5345433-337f-4434-a218-a9880269ddc5-istio-envoy\") pod \"e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb\" (UID: \"e5345433-337f-4434-a218-a9880269ddc5\") " pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:27:44.690736 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.690603 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5345433-337f-4434-a218-a9880269ddc5-workload-socket\") pod \"e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb\" (UID: \"e5345433-337f-4434-a218-a9880269ddc5\") " pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:27:44.690736 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.690643 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5345433-337f-4434-a218-a9880269ddc5-istio-data\") pod \"e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb\" (UID: \"e5345433-337f-4434-a218-a9880269ddc5\") " pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:27:44.690736 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.690662 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5345433-337f-4434-a218-a9880269ddc5-istiod-ca-cert\") pod \"e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb\" (UID: \"e5345433-337f-4434-a218-a9880269ddc5\") " pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:27:44.690995 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.690800 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5345433-337f-4434-a218-a9880269ddc5-istio-podinfo\") pod \"e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb\" (UID: \"e5345433-337f-4434-a218-a9880269ddc5\") " pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:27:44.690995 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.690873 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5345433-337f-4434-a218-a9880269ddc5-workload-certs\") pod \"e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb\" (UID: \"e5345433-337f-4434-a218-a9880269ddc5\") " pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:27:44.690995 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.690967 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5345433-337f-4434-a218-a9880269ddc5-istio-token\") pod \"e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb\" (UID: \"e5345433-337f-4434-a218-a9880269ddc5\") " pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:27:44.690995 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.690979 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5345433-337f-4434-a218-a9880269ddc5-istio-data\") pod \"e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb\" (UID: \"e5345433-337f-4434-a218-a9880269ddc5\") " pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:27:44.691234 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.690967 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5345433-337f-4434-a218-a9880269ddc5-credential-socket\") pod \"e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb\" (UID: \"e5345433-337f-4434-a218-a9880269ddc5\") " pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:27:44.691234 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.691055 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvqtn\" (UniqueName: \"kubernetes.io/projected/e5345433-337f-4434-a218-a9880269ddc5-kube-api-access-tvqtn\") pod \"e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb\" (UID: \"e5345433-337f-4434-a218-a9880269ddc5\") " pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:27:44.691350 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.691278 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5345433-337f-4434-a218-a9880269ddc5-workload-certs\") pod \"e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb\" (UID: \"e5345433-337f-4434-a218-a9880269ddc5\") " pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:27:44.691411 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.691389 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5345433-337f-4434-a218-a9880269ddc5-istiod-ca-cert\") pod \"e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb\" (UID: \"e5345433-337f-4434-a218-a9880269ddc5\") " pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:27:44.691526 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.691505 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5345433-337f-4434-a218-a9880269ddc5-workload-socket\") pod \"e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb\" (UID: \"e5345433-337f-4434-a218-a9880269ddc5\") " pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:27:44.693554 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.693527 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5345433-337f-4434-a218-a9880269ddc5-istio-podinfo\") pod \"e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb\" (UID: \"e5345433-337f-4434-a218-a9880269ddc5\") " pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:27:44.694532 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.694510 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5345433-337f-4434-a218-a9880269ddc5-istio-envoy\") pod \"e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb\" (UID: \"e5345433-337f-4434-a218-a9880269ddc5\") " pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:27:44.754280 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.754182 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5345433-337f-4434-a218-a9880269ddc5-istio-token\") pod \"e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb\" (UID: \"e5345433-337f-4434-a218-a9880269ddc5\") " pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:27:44.754456 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.754422 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvqtn\" (UniqueName: \"kubernetes.io/projected/e5345433-337f-4434-a218-a9880269ddc5-kube-api-access-tvqtn\") pod \"e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb\" (UID: \"e5345433-337f-4434-a218-a9880269ddc5\") " pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:27:44.823275 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:44.823240 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:27:45.284957 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:45.284920 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb"] Oct 30 22:27:45.289596 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:27:45.289568 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5345433_337f_4434_a218_a9880269ddc5.slice/crio-325cefe10d26831cb5e5349211d8dba81866fef41629f884d3d00a1f6cb2968c WatchSource:0}: Error finding container 325cefe10d26831cb5e5349211d8dba81866fef41629f884d3d00a1f6cb2968c: Status 404 returned error can't find the container with id 325cefe10d26831cb5e5349211d8dba81866fef41629f884d3d00a1f6cb2968c Oct 30 22:27:45.292268 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:45.292196 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:27:45.292384 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:45.292309 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:27:45.292384 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:45.292344 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:27:45.921500 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:45.921462 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" event={"ID":"e5345433-337f-4434-a218-a9880269ddc5","Type":"ContainerStarted","Data":"55569ea765efb899cd1a6c5657da7820a75548cd6069b499d0ce7f9047e11102"} Oct 30 22:27:45.921948 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:45.921562 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" event={"ID":"e5345433-337f-4434-a218-a9880269ddc5","Type":"ContainerStarted","Data":"325cefe10d26831cb5e5349211d8dba81866fef41629f884d3d00a1f6cb2968c"} Oct 30 22:27:46.055632 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:46.055575 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" podStartSLOduration=2.055562496 podStartE2EDuration="2.055562496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:46.052781479 +0000 UTC m=+2226.323479563" watchObservedRunningTime="2026-10-30 22:27:46.055562496 +0000 UTC m=+2226.326260579" Oct 30 22:27:46.824314 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:46.824273 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:27:46.829795 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:46.829769 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:27:46.927753 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:46.927720 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:27:46.928721 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:46.928697 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:27:51.275668 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:27:51.275630 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:27:52.754239 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:52.753416 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d59c58-5b46f96fb9-9pd5g"] Oct 30 22:27:52.762240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:52.759602 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d59c58-5b46f96fb9-9pd5g" Oct 30 22:27:52.815955 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:52.815915 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d59c58-5b46f96fb9-9pd5g"] Oct 30 22:27:52.863282 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:52.863251 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d59c58\"" Oct 30 22:27:52.863456 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:52.863304 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d59c58-dockercfg-fz5z4\"" Oct 30 22:27:52.919501 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:52.919467 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/319acc25-060d-4921-bcaa-a62f41412146-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d59c58-5b46f96fb9-9pd5g\" (UID: \"319acc25-060d-4921-bcaa-a62f41412146\") " pod="openshift-ingress/payload-processing-e2e-api-b-d59c58-5b46f96fb9-9pd5g" Oct 30 22:27:52.919844 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:52.919812 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9c7r\" (UniqueName: \"kubernetes.io/projected/319acc25-060d-4921-bcaa-a62f41412146-kube-api-access-c9c7r\") pod \"payload-processing-e2e-api-b-d59c58-5b46f96fb9-9pd5g\" (UID: \"319acc25-060d-4921-bcaa-a62f41412146\") " pod="openshift-ingress/payload-processing-e2e-api-b-d59c58-5b46f96fb9-9pd5g" Oct 30 22:27:52.927676 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:52.927648 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d59c58-7678d58759-jsctb"] Oct 30 22:27:52.931578 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:52.931560 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d59c58-7678d58759-jsctb" Oct 30 22:27:52.995502 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:52.995471 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d59c58-7678d58759-jsctb"] Oct 30 22:27:53.021176 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:53.021094 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvf6t\" (UniqueName: \"kubernetes.io/projected/9b4c802f-e487-4d43-a8aa-ef87ddc27fcf-kube-api-access-rvf6t\") pod \"payload-pre-processing-e2e-api-b-d59c58-7678d58759-jsctb\" (UID: \"9b4c802f-e487-4d43-a8aa-ef87ddc27fcf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d59c58-7678d58759-jsctb" Oct 30 22:27:53.021394 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:53.021228 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b4c802f-e487-4d43-a8aa-ef87ddc27fcf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d59c58-7678d58759-jsctb\" (UID: \"9b4c802f-e487-4d43-a8aa-ef87ddc27fcf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d59c58-7678d58759-jsctb" Oct 30 22:27:53.021394 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:53.021334 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/319acc25-060d-4921-bcaa-a62f41412146-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d59c58-5b46f96fb9-9pd5g\" (UID: \"319acc25-060d-4921-bcaa-a62f41412146\") " pod="openshift-ingress/payload-processing-e2e-api-b-d59c58-5b46f96fb9-9pd5g" Oct 30 22:27:53.021537 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:53.021438 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9c7r\" (UniqueName: \"kubernetes.io/projected/319acc25-060d-4921-bcaa-a62f41412146-kube-api-access-c9c7r\") pod \"payload-processing-e2e-api-b-d59c58-5b46f96fb9-9pd5g\" (UID: \"319acc25-060d-4921-bcaa-a62f41412146\") " pod="openshift-ingress/payload-processing-e2e-api-b-d59c58-5b46f96fb9-9pd5g" Oct 30 22:27:53.022279 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:53.022256 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/319acc25-060d-4921-bcaa-a62f41412146-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d59c58-5b46f96fb9-9pd5g\" (UID: \"319acc25-060d-4921-bcaa-a62f41412146\") " pod="openshift-ingress/payload-processing-e2e-api-b-d59c58-5b46f96fb9-9pd5g" Oct 30 22:27:53.085286 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:53.085249 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9c7r\" (UniqueName: \"kubernetes.io/projected/319acc25-060d-4921-bcaa-a62f41412146-kube-api-access-c9c7r\") pod \"payload-processing-e2e-api-b-d59c58-5b46f96fb9-9pd5g\" (UID: \"319acc25-060d-4921-bcaa-a62f41412146\") " pod="openshift-ingress/payload-processing-e2e-api-b-d59c58-5b46f96fb9-9pd5g" Oct 30 22:27:53.101908 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:53.101880 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d59c58-5b46f96fb9-9pd5g" Oct 30 22:27:53.122768 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:53.122735 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvf6t\" (UniqueName: \"kubernetes.io/projected/9b4c802f-e487-4d43-a8aa-ef87ddc27fcf-kube-api-access-rvf6t\") pod \"payload-pre-processing-e2e-api-b-d59c58-7678d58759-jsctb\" (UID: \"9b4c802f-e487-4d43-a8aa-ef87ddc27fcf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d59c58-7678d58759-jsctb" Oct 30 22:27:53.122890 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:53.122811 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b4c802f-e487-4d43-a8aa-ef87ddc27fcf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d59c58-7678d58759-jsctb\" (UID: \"9b4c802f-e487-4d43-a8aa-ef87ddc27fcf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d59c58-7678d58759-jsctb" Oct 30 22:27:53.123689 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:53.123662 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b4c802f-e487-4d43-a8aa-ef87ddc27fcf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d59c58-7678d58759-jsctb\" (UID: \"9b4c802f-e487-4d43-a8aa-ef87ddc27fcf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d59c58-7678d58759-jsctb" Oct 30 22:27:53.186833 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:53.186802 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvf6t\" (UniqueName: \"kubernetes.io/projected/9b4c802f-e487-4d43-a8aa-ef87ddc27fcf-kube-api-access-rvf6t\") pod \"payload-pre-processing-e2e-api-b-d59c58-7678d58759-jsctb\" (UID: \"9b4c802f-e487-4d43-a8aa-ef87ddc27fcf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d59c58-7678d58759-jsctb" Oct 30 22:27:53.245250 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:53.245066 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d59c58-7678d58759-jsctb" Oct 30 22:27:53.481055 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:53.481024 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d59c58-5b46f96fb9-9pd5g"] Oct 30 22:27:53.482374 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:27:53.482344 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod319acc25_060d_4921_bcaa_a62f41412146.slice/crio-df4c3ee2b899c3afaac16762daa580f79cefdb5328c0a277dd8e38c0e2232c1d WatchSource:0}: Error finding container df4c3ee2b899c3afaac16762daa580f79cefdb5328c0a277dd8e38c0e2232c1d: Status 404 returned error can't find the container with id df4c3ee2b899c3afaac16762daa580f79cefdb5328c0a277dd8e38c0e2232c1d Oct 30 22:27:53.622493 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:53.622469 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d59c58-7678d58759-jsctb"] Oct 30 22:27:53.623950 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:27:53.623923 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b4c802f_e487_4d43_a8aa_ef87ddc27fcf.slice/crio-791a697682c22bb58e8e7884a4320cb1e77d80247f19da427b8e76bae90b7cdd WatchSource:0}: Error finding container 791a697682c22bb58e8e7884a4320cb1e77d80247f19da427b8e76bae90b7cdd: Status 404 returned error can't find the container with id 791a697682c22bb58e8e7884a4320cb1e77d80247f19da427b8e76bae90b7cdd Oct 30 22:27:53.983151 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:53.983095 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d59c58-5b46f96fb9-9pd5g" event={"ID":"319acc25-060d-4921-bcaa-a62f41412146","Type":"ContainerStarted","Data":"3671d6cf8a96977d07a220bc505b0212a684820637995cac5ec703f4238a4e3d"} Oct 30 22:27:53.983698 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:53.983298 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-d59c58-5b46f96fb9-9pd5g" Oct 30 22:27:53.983698 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:53.983349 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d59c58-5b46f96fb9-9pd5g" event={"ID":"319acc25-060d-4921-bcaa-a62f41412146","Type":"ContainerStarted","Data":"df4c3ee2b899c3afaac16762daa580f79cefdb5328c0a277dd8e38c0e2232c1d"} Oct 30 22:27:53.985025 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:53.984999 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d59c58-7678d58759-jsctb" event={"ID":"9b4c802f-e487-4d43-a8aa-ef87ddc27fcf","Type":"ContainerStarted","Data":"a1ce79fb24761b7e704af0651abdf99501a5f93c5f81dfde8e68a3e75f327228"} Oct 30 22:27:53.985131 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:53.985046 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d59c58-7678d58759-jsctb" event={"ID":"9b4c802f-e487-4d43-a8aa-ef87ddc27fcf","Type":"ContainerStarted","Data":"791a697682c22bb58e8e7884a4320cb1e77d80247f19da427b8e76bae90b7cdd"} Oct 30 22:27:53.985131 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:53.985087 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d59c58-7678d58759-jsctb" Oct 30 22:27:54.104712 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:54.104664 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-d59c58-5b46f96fb9-9pd5g" podStartSLOduration=2.104652285 podStartE2EDuration="2.104652285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:54.102767278 +0000 UTC m=+2234.373465361" watchObservedRunningTime="2026-10-30 22:27:54.104652285 +0000 UTC m=+2234.375350370" Oct 30 22:27:54.223209 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:27:54.223137 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d59c58-7678d58759-jsctb" podStartSLOduration=2.2231199090000002 podStartE2EDuration="2.223119909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:54.222262938 +0000 UTC m=+2234.492961024" watchObservedRunningTime="2026-10-30 22:27:54.223119909 +0000 UTC m=+2234.493817994" Oct 30 22:28:04.993881 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:04.993846 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d59c58-7678d58759-jsctb" Oct 30 22:28:04.994422 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:04.994091 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-d59c58-5b46f96fb9-9pd5g" Oct 30 22:28:08.201358 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:08.201327 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs"] Oct 30 22:28:08.201768 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:08.201546 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs" podUID="d9a3c6a5-841a-49c8-afb7-7627ab28963c" containerName="maas-api" containerID="cri-o://b77251d5d864e70ea5539f8de73502a1db04cca2a1f6f63d41e8558781ea21dd" gracePeriod=30 Oct 30 22:28:08.507256 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:08.507231 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs" Oct 30 22:28:08.619239 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:08.617773 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9a3c6a5-841a-49c8-afb7-7627ab28963c-kube-api-access-jspdd\" (UniqueName: \"kubernetes.io/projected/d9a3c6a5-841a-49c8-afb7-7627ab28963c-kube-api-access-jspdd\") pod \"d9a3c6a5-841a-49c8-afb7-7627ab28963c\" (UID: \"d9a3c6a5-841a-49c8-afb7-7627ab28963c\") " Oct 30 22:28:08.619239 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:08.617823 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9a3c6a5-841a-49c8-afb7-7627ab28963c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9a3c6a5-841a-49c8-afb7-7627ab28963c-metrics-certs\") pod \"d9a3c6a5-841a-49c8-afb7-7627ab28963c\" (UID: \"d9a3c6a5-841a-49c8-afb7-7627ab28963c\") " Oct 30 22:28:08.619239 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:08.617908 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9a3c6a5-841a-49c8-afb7-7627ab28963c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9a3c6a5-841a-49c8-afb7-7627ab28963c-openshift-service-ca\") pod \"d9a3c6a5-841a-49c8-afb7-7627ab28963c\" (UID: \"d9a3c6a5-841a-49c8-afb7-7627ab28963c\") " Oct 30 22:28:08.619239 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:08.617985 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9a3c6a5-841a-49c8-afb7-7627ab28963c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9a3c6a5-841a-49c8-afb7-7627ab28963c-maas-api-tls\") pod \"d9a3c6a5-841a-49c8-afb7-7627ab28963c\" (UID: \"d9a3c6a5-841a-49c8-afb7-7627ab28963c\") " Oct 30 22:28:08.627238 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:08.621107 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9a3c6a5-841a-49c8-afb7-7627ab28963c-openshift-service-ca" pod "d9a3c6a5-841a-49c8-afb7-7627ab28963c" (UID: "d9a3c6a5-841a-49c8-afb7-7627ab28963c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:08.633235 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:08.628378 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9a3c6a5-841a-49c8-afb7-7627ab28963c-metrics-certs" pod "d9a3c6a5-841a-49c8-afb7-7627ab28963c" (UID: "d9a3c6a5-841a-49c8-afb7-7627ab28963c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:28:08.633235 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:08.629295 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9a3c6a5-841a-49c8-afb7-7627ab28963c-maas-api-tls" pod "d9a3c6a5-841a-49c8-afb7-7627ab28963c" (UID: "d9a3c6a5-841a-49c8-afb7-7627ab28963c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:28:08.646248 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:08.645593 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9a3c6a5-841a-49c8-afb7-7627ab28963c-kube-api-access-jspdd" pod "d9a3c6a5-841a-49c8-afb7-7627ab28963c" (UID: "d9a3c6a5-841a-49c8-afb7-7627ab28963c"). InnerVolumeSpecName "kube-api-access-jspdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:08.719544 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:08.719504 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jspdd\" (UniqueName: \"kubernetes.io/projected/d9a3c6a5-841a-49c8-afb7-7627ab28963c-kube-api-access-jspdd\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:08.719544 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:08.719533 5918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9a3c6a5-841a-49c8-afb7-7627ab28963c-metrics-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:08.719544 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:08.719551 5918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9a3c6a5-841a-49c8-afb7-7627ab28963c-openshift-service-ca\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:08.719846 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:08.719563 5918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9a3c6a5-841a-49c8-afb7-7627ab28963c-maas-api-tls\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:09.079780 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:09.079742 5918 generic.go:356] "Generic (PLEG): container finished" podID="d9a3c6a5-841a-49c8-afb7-7627ab28963c" containerID="b77251d5d864e70ea5539f8de73502a1db04cca2a1f6f63d41e8558781ea21dd" exitCode=0 Oct 30 22:28:09.079958 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:09.079787 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs" Oct 30 22:28:09.079958 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:09.079825 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs" event={"ID":"d9a3c6a5-841a-49c8-afb7-7627ab28963c","Type":"ContainerDied","Data":"b77251d5d864e70ea5539f8de73502a1db04cca2a1f6f63d41e8558781ea21dd"} Oct 30 22:28:09.079958 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:09.079879 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs" event={"ID":"d9a3c6a5-841a-49c8-afb7-7627ab28963c","Type":"ContainerDied","Data":"e76c532c2f2157322fc362f5f2fc6e1fb06d0b1fb20f65036ba0c79332cd5fcd"} Oct 30 22:28:09.079958 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:09.079905 5918 scope.go:122] "RemoveContainer" containerID="b77251d5d864e70ea5539f8de73502a1db04cca2a1f6f63d41e8558781ea21dd" Oct 30 22:28:09.090351 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:09.090330 5918 scope.go:122] "RemoveContainer" containerID="b77251d5d864e70ea5539f8de73502a1db04cca2a1f6f63d41e8558781ea21dd" Oct 30 22:28:09.090636 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:28:09.090603 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b77251d5d864e70ea5539f8de73502a1db04cca2a1f6f63d41e8558781ea21dd\": container with ID starting with b77251d5d864e70ea5539f8de73502a1db04cca2a1f6f63d41e8558781ea21dd not found: ID does not exist" containerID="b77251d5d864e70ea5539f8de73502a1db04cca2a1f6f63d41e8558781ea21dd" Oct 30 22:28:09.090712 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:09.090648 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b77251d5d864e70ea5539f8de73502a1db04cca2a1f6f63d41e8558781ea21dd"} err="failed to get container status \"b77251d5d864e70ea5539f8de73502a1db04cca2a1f6f63d41e8558781ea21dd\": rpc error: code = NotFound desc = could not find container \"b77251d5d864e70ea5539f8de73502a1db04cca2a1f6f63d41e8558781ea21dd\": container with ID starting with b77251d5d864e70ea5539f8de73502a1db04cca2a1f6f63d41e8558781ea21dd not found: ID does not exist" Oct 30 22:28:09.249158 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:09.249117 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs"] Oct 30 22:28:09.299434 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:09.299399 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6a0a09-c79584dbd-xl6hs"] Oct 30 22:28:09.860609 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:09.860571 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9a3c6a5-841a-49c8-afb7-7627ab28963c" path="/var/lib/kubelet/pods/d9a3c6a5-841a-49c8-afb7-7627ab28963c/volumes" Oct 30 22:28:18.529491 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:18.529456 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf"] Oct 30 22:28:18.530035 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:18.529666 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" podUID="a77bc09f-2593-4ed7-89fe-21fffab26adf" containerName="istio-proxy" containerID="cri-o://eb18f53ae7c5aea2e647c4eb64c2a86ef4f6494f916928d1ef45635137bcdfd2" gracePeriod=30 Oct 30 22:28:20.152188 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:20.152089 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln"] Oct 30 22:28:20.152689 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:20.152313 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln" podUID="47da8d4b-2ae1-459b-8d7b-522745a5fd56" containerName="maas-api" containerID="cri-o://2b5954cbfad37b2dab0001550fe20c77108fe62ed85acbd08c631b7a81b8eb7f" gracePeriod=30 Oct 30 22:28:20.504673 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:20.504637 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln" Oct 30 22:28:20.683831 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:20.683805 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47da8d4b-2ae1-459b-8d7b-522745a5fd56-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47da8d4b-2ae1-459b-8d7b-522745a5fd56-openshift-service-ca\") pod \"47da8d4b-2ae1-459b-8d7b-522745a5fd56\" (UID: \"47da8d4b-2ae1-459b-8d7b-522745a5fd56\") " Oct 30 22:28:20.684034 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:20.683874 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47da8d4b-2ae1-459b-8d7b-522745a5fd56-metrics-certs\" (UniqueName: \"kubernetes.io/secret/47da8d4b-2ae1-459b-8d7b-522745a5fd56-metrics-certs\") pod \"47da8d4b-2ae1-459b-8d7b-522745a5fd56\" (UID: \"47da8d4b-2ae1-459b-8d7b-522745a5fd56\") " Oct 30 22:28:20.684034 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:20.683973 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47da8d4b-2ae1-459b-8d7b-522745a5fd56-kube-api-access-wq2gs\" (UniqueName: \"kubernetes.io/projected/47da8d4b-2ae1-459b-8d7b-522745a5fd56-kube-api-access-wq2gs\") pod \"47da8d4b-2ae1-459b-8d7b-522745a5fd56\" (UID: \"47da8d4b-2ae1-459b-8d7b-522745a5fd56\") " Oct 30 22:28:20.684154 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:20.684103 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47da8d4b-2ae1-459b-8d7b-522745a5fd56-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47da8d4b-2ae1-459b-8d7b-522745a5fd56-maas-api-tls\") pod \"47da8d4b-2ae1-459b-8d7b-522745a5fd56\" (UID: \"47da8d4b-2ae1-459b-8d7b-522745a5fd56\") " Oct 30 22:28:20.684238 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:20.684176 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47da8d4b-2ae1-459b-8d7b-522745a5fd56-openshift-service-ca" pod "47da8d4b-2ae1-459b-8d7b-522745a5fd56" (UID: "47da8d4b-2ae1-459b-8d7b-522745a5fd56"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:20.684570 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:20.684537 5918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47da8d4b-2ae1-459b-8d7b-522745a5fd56-openshift-service-ca\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:20.686272 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:20.686248 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47da8d4b-2ae1-459b-8d7b-522745a5fd56-kube-api-access-wq2gs" pod "47da8d4b-2ae1-459b-8d7b-522745a5fd56" (UID: "47da8d4b-2ae1-459b-8d7b-522745a5fd56"). InnerVolumeSpecName "kube-api-access-wq2gs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:20.686376 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:20.686269 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47da8d4b-2ae1-459b-8d7b-522745a5fd56-metrics-certs" pod "47da8d4b-2ae1-459b-8d7b-522745a5fd56" (UID: "47da8d4b-2ae1-459b-8d7b-522745a5fd56"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:28:20.686376 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:20.686291 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47da8d4b-2ae1-459b-8d7b-522745a5fd56-maas-api-tls" pod "47da8d4b-2ae1-459b-8d7b-522745a5fd56" (UID: "47da8d4b-2ae1-459b-8d7b-522745a5fd56"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:28:20.786153 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:20.786119 5918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47da8d4b-2ae1-459b-8d7b-522745a5fd56-metrics-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:20.786153 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:20.786151 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq2gs\" (UniqueName: \"kubernetes.io/projected/47da8d4b-2ae1-459b-8d7b-522745a5fd56-kube-api-access-wq2gs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:20.786153 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:20.786162 5918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47da8d4b-2ae1-459b-8d7b-522745a5fd56-maas-api-tls\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:21.151918 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:21.151827 5918 generic.go:356] "Generic (PLEG): container finished" podID="47da8d4b-2ae1-459b-8d7b-522745a5fd56" containerID="2b5954cbfad37b2dab0001550fe20c77108fe62ed85acbd08c631b7a81b8eb7f" exitCode=0 Oct 30 22:28:21.151918 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:21.151888 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln" Oct 30 22:28:21.151918 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:21.151905 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln" event={"ID":"47da8d4b-2ae1-459b-8d7b-522745a5fd56","Type":"ContainerDied","Data":"2b5954cbfad37b2dab0001550fe20c77108fe62ed85acbd08c631b7a81b8eb7f"} Oct 30 22:28:21.152162 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:21.151961 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln" event={"ID":"47da8d4b-2ae1-459b-8d7b-522745a5fd56","Type":"ContainerDied","Data":"b0c95bb4b102502284e0890767e54a152d24fe620d282713795f637a3609e0b2"} Oct 30 22:28:21.152162 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:21.151987 5918 scope.go:122] "RemoveContainer" containerID="2b5954cbfad37b2dab0001550fe20c77108fe62ed85acbd08c631b7a81b8eb7f" Oct 30 22:28:21.162751 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:21.162731 5918 scope.go:122] "RemoveContainer" containerID="2b5954cbfad37b2dab0001550fe20c77108fe62ed85acbd08c631b7a81b8eb7f" Oct 30 22:28:21.163045 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:28:21.163023 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b5954cbfad37b2dab0001550fe20c77108fe62ed85acbd08c631b7a81b8eb7f\": container with ID starting with 2b5954cbfad37b2dab0001550fe20c77108fe62ed85acbd08c631b7a81b8eb7f not found: ID does not exist" containerID="2b5954cbfad37b2dab0001550fe20c77108fe62ed85acbd08c631b7a81b8eb7f" Oct 30 22:28:21.163109 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:21.163054 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b5954cbfad37b2dab0001550fe20c77108fe62ed85acbd08c631b7a81b8eb7f"} err="failed to get container status \"2b5954cbfad37b2dab0001550fe20c77108fe62ed85acbd08c631b7a81b8eb7f\": rpc error: code = NotFound desc = could not find container \"2b5954cbfad37b2dab0001550fe20c77108fe62ed85acbd08c631b7a81b8eb7f\": container with ID starting with 2b5954cbfad37b2dab0001550fe20c77108fe62ed85acbd08c631b7a81b8eb7f not found: ID does not exist" Oct 30 22:28:21.274879 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:28:21.274845 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:28:21.323554 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:21.323525 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln"] Oct 30 22:28:21.401238 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:21.400887 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-9cff60-6db7f6654-5tkln"] Oct 30 22:28:21.859238 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:21.859032 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47da8d4b-2ae1-459b-8d7b-522745a5fd56" path="/var/lib/kubelet/pods/47da8d4b-2ae1-459b-8d7b-522745a5fd56/volumes" Oct 30 22:28:22.483086 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:22.483046 5918 patch_prober.go:29] interesting pod/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:28:22.483589 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:22.483132 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" podUID="a77bc09f-2593-4ed7-89fe-21fffab26adf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:28:23.885042 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:23.885018 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" Oct 30 22:28:23.963046 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:23.963009 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a77bc09f-2593-4ed7-89fe-21fffab26adf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a77bc09f-2593-4ed7-89fe-21fffab26adf-istio-podinfo\") pod \"a77bc09f-2593-4ed7-89fe-21fffab26adf\" (UID: \"a77bc09f-2593-4ed7-89fe-21fffab26adf\") " Oct 30 22:28:23.963254 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:23.963052 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a77bc09f-2593-4ed7-89fe-21fffab26adf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77bc09f-2593-4ed7-89fe-21fffab26adf-credential-socket\") pod \"a77bc09f-2593-4ed7-89fe-21fffab26adf\" (UID: \"a77bc09f-2593-4ed7-89fe-21fffab26adf\") " Oct 30 22:28:23.963254 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:23.963121 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a77bc09f-2593-4ed7-89fe-21fffab26adf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a77bc09f-2593-4ed7-89fe-21fffab26adf-workload-certs\") pod \"a77bc09f-2593-4ed7-89fe-21fffab26adf\" (UID: \"a77bc09f-2593-4ed7-89fe-21fffab26adf\") " Oct 30 22:28:23.963254 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:23.963203 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a77bc09f-2593-4ed7-89fe-21fffab26adf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a77bc09f-2593-4ed7-89fe-21fffab26adf-istio-envoy\") pod \"a77bc09f-2593-4ed7-89fe-21fffab26adf\" (UID: \"a77bc09f-2593-4ed7-89fe-21fffab26adf\") " Oct 30 22:28:23.963409 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:23.963259 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a77bc09f-2593-4ed7-89fe-21fffab26adf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77bc09f-2593-4ed7-89fe-21fffab26adf-workload-socket\") pod \"a77bc09f-2593-4ed7-89fe-21fffab26adf\" (UID: \"a77bc09f-2593-4ed7-89fe-21fffab26adf\") " Oct 30 22:28:23.963409 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:23.963319 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a77bc09f-2593-4ed7-89fe-21fffab26adf-istio-token\" (UniqueName: \"kubernetes.io/projected/a77bc09f-2593-4ed7-89fe-21fffab26adf-istio-token\") pod \"a77bc09f-2593-4ed7-89fe-21fffab26adf\" (UID: \"a77bc09f-2593-4ed7-89fe-21fffab26adf\") " Oct 30 22:28:23.963409 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:23.963358 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a77bc09f-2593-4ed7-89fe-21fffab26adf-kube-api-access-2qpdr\" (UniqueName: \"kubernetes.io/projected/a77bc09f-2593-4ed7-89fe-21fffab26adf-kube-api-access-2qpdr\") pod \"a77bc09f-2593-4ed7-89fe-21fffab26adf\" (UID: \"a77bc09f-2593-4ed7-89fe-21fffab26adf\") " Oct 30 22:28:23.963571 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:23.963416 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a77bc09f-2593-4ed7-89fe-21fffab26adf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a77bc09f-2593-4ed7-89fe-21fffab26adf-istio-data\") pod \"a77bc09f-2593-4ed7-89fe-21fffab26adf\" (UID: \"a77bc09f-2593-4ed7-89fe-21fffab26adf\") " Oct 30 22:28:23.963571 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:23.963480 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a77bc09f-2593-4ed7-89fe-21fffab26adf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a77bc09f-2593-4ed7-89fe-21fffab26adf-istiod-ca-cert\") pod \"a77bc09f-2593-4ed7-89fe-21fffab26adf\" (UID: \"a77bc09f-2593-4ed7-89fe-21fffab26adf\") " Oct 30 22:28:23.963571 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:23.963480 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77bc09f-2593-4ed7-89fe-21fffab26adf-credential-socket" pod "a77bc09f-2593-4ed7-89fe-21fffab26adf" (UID: "a77bc09f-2593-4ed7-89fe-21fffab26adf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:23.963571 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:23.963491 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77bc09f-2593-4ed7-89fe-21fffab26adf-workload-socket" pod "a77bc09f-2593-4ed7-89fe-21fffab26adf" (UID: "a77bc09f-2593-4ed7-89fe-21fffab26adf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:23.963571 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:23.963480 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77bc09f-2593-4ed7-89fe-21fffab26adf-workload-certs" pod "a77bc09f-2593-4ed7-89fe-21fffab26adf" (UID: "a77bc09f-2593-4ed7-89fe-21fffab26adf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:23.963827 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:23.963686 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77bc09f-2593-4ed7-89fe-21fffab26adf-istio-data" pod "a77bc09f-2593-4ed7-89fe-21fffab26adf" (UID: "a77bc09f-2593-4ed7-89fe-21fffab26adf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:23.963925 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:23.963902 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a77bc09f-2593-4ed7-89fe-21fffab26adf-istiod-ca-cert" pod "a77bc09f-2593-4ed7-89fe-21fffab26adf" (UID: "a77bc09f-2593-4ed7-89fe-21fffab26adf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:23.964243 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:23.964152 5918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a77bc09f-2593-4ed7-89fe-21fffab26adf-istio-data\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:23.964373 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:23.964244 5918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a77bc09f-2593-4ed7-89fe-21fffab26adf-istiod-ca-cert\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:23.964373 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:23.964263 5918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77bc09f-2593-4ed7-89fe-21fffab26adf-credential-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:23.964373 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:23.964277 5918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a77bc09f-2593-4ed7-89fe-21fffab26adf-workload-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:23.964373 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:23.964290 5918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77bc09f-2593-4ed7-89fe-21fffab26adf-workload-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:23.965730 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:23.965709 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a77bc09f-2593-4ed7-89fe-21fffab26adf-istio-podinfo" pod "a77bc09f-2593-4ed7-89fe-21fffab26adf" (UID: "a77bc09f-2593-4ed7-89fe-21fffab26adf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:28:23.965730 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:23.965717 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77bc09f-2593-4ed7-89fe-21fffab26adf-istio-envoy" pod "a77bc09f-2593-4ed7-89fe-21fffab26adf" (UID: "a77bc09f-2593-4ed7-89fe-21fffab26adf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:23.965861 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:23.965752 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a77bc09f-2593-4ed7-89fe-21fffab26adf-istio-token" pod "a77bc09f-2593-4ed7-89fe-21fffab26adf" (UID: "a77bc09f-2593-4ed7-89fe-21fffab26adf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:23.965904 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:23.965893 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a77bc09f-2593-4ed7-89fe-21fffab26adf-kube-api-access-2qpdr" pod "a77bc09f-2593-4ed7-89fe-21fffab26adf" (UID: "a77bc09f-2593-4ed7-89fe-21fffab26adf"). InnerVolumeSpecName "kube-api-access-2qpdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:24.065978 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:24.065938 5918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a77bc09f-2593-4ed7-89fe-21fffab26adf-istio-token\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:24.065978 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:24.065968 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qpdr\" (UniqueName: \"kubernetes.io/projected/a77bc09f-2593-4ed7-89fe-21fffab26adf-kube-api-access-2qpdr\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:24.065978 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:24.065978 5918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a77bc09f-2593-4ed7-89fe-21fffab26adf-istio-podinfo\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:24.065978 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:24.065988 5918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a77bc09f-2593-4ed7-89fe-21fffab26adf-istio-envoy\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:24.176775 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:24.176747 5918 generic.go:356] "Generic (PLEG): container finished" podID="a77bc09f-2593-4ed7-89fe-21fffab26adf" containerID="eb18f53ae7c5aea2e647c4eb64c2a86ef4f6494f916928d1ef45635137bcdfd2" exitCode=0 Oct 30 22:28:24.176933 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:24.176798 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" Oct 30 22:28:24.176933 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:24.176801 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" event={"ID":"a77bc09f-2593-4ed7-89fe-21fffab26adf","Type":"ContainerDied","Data":"eb18f53ae7c5aea2e647c4eb64c2a86ef4f6494f916928d1ef45635137bcdfd2"} Oct 30 22:28:24.176933 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:24.176918 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf" event={"ID":"a77bc09f-2593-4ed7-89fe-21fffab26adf","Type":"ContainerDied","Data":"30c7cf577b6692095cc9d10824ba4a49c7d29b3d7992a13b4257da0a72c753fe"} Oct 30 22:28:24.177049 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:24.176944 5918 scope.go:122] "RemoveContainer" containerID="eb18f53ae7c5aea2e647c4eb64c2a86ef4f6494f916928d1ef45635137bcdfd2" Oct 30 22:28:24.205426 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:24.204620 5918 scope.go:122] "RemoveContainer" containerID="eb18f53ae7c5aea2e647c4eb64c2a86ef4f6494f916928d1ef45635137bcdfd2" Oct 30 22:28:24.205426 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:28:24.205067 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb18f53ae7c5aea2e647c4eb64c2a86ef4f6494f916928d1ef45635137bcdfd2\": container with ID starting with eb18f53ae7c5aea2e647c4eb64c2a86ef4f6494f916928d1ef45635137bcdfd2 not found: ID does not exist" containerID="eb18f53ae7c5aea2e647c4eb64c2a86ef4f6494f916928d1ef45635137bcdfd2" Oct 30 22:28:24.205426 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:24.205104 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb18f53ae7c5aea2e647c4eb64c2a86ef4f6494f916928d1ef45635137bcdfd2"} err="failed to get container status \"eb18f53ae7c5aea2e647c4eb64c2a86ef4f6494f916928d1ef45635137bcdfd2\": rpc error: code = NotFound desc = could not find container \"eb18f53ae7c5aea2e647c4eb64c2a86ef4f6494f916928d1ef45635137bcdfd2\": container with ID starting with eb18f53ae7c5aea2e647c4eb64c2a86ef4f6494f916928d1ef45635137bcdfd2 not found: ID does not exist" Oct 30 22:28:24.370176 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:24.370136 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf"] Oct 30 22:28:24.377157 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:24.377128 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6a0a09-openshift-default-5f7c779d7-dl4zf"] Oct 30 22:28:24.662819 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:24.662788 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87"] Oct 30 22:28:24.670252 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:24.670177 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z"] Oct 30 22:28:25.856456 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:25.856424 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a77bc09f-2593-4ed7-89fe-21fffab26adf" path="/var/lib/kubelet/pods/a77bc09f-2593-4ed7-89fe-21fffab26adf/volumes" Oct 30 22:28:27.834608 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:27.834566 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q"] Oct 30 22:28:27.835278 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:27.835259 5918 cpu_manager.go:414] "containerMap: removing container" podUID="d9a3c6a5-841a-49c8-afb7-7627ab28963c" containerName="maas-api" Oct 30 22:28:27.835278 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:27.835278 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9a3c6a5-841a-49c8-afb7-7627ab28963c" containerName="maas-api" Oct 30 22:28:27.835470 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:27.835302 5918 cpu_manager.go:414] "containerMap: removing container" podUID="a77bc09f-2593-4ed7-89fe-21fffab26adf" containerName="istio-proxy" Oct 30 22:28:27.835470 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:27.835308 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a77bc09f-2593-4ed7-89fe-21fffab26adf" containerName="istio-proxy" Oct 30 22:28:27.835470 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:27.835322 5918 cpu_manager.go:414] "containerMap: removing container" podUID="47da8d4b-2ae1-459b-8d7b-522745a5fd56" containerName="maas-api" Oct 30 22:28:27.835470 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:27.835327 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47da8d4b-2ae1-459b-8d7b-522745a5fd56" containerName="maas-api" Oct 30 22:28:27.835470 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:27.835458 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9a3c6a5-841a-49c8-afb7-7627ab28963c" containerName="maas-api" Oct 30 22:28:27.835470 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:27.835470 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a77bc09f-2593-4ed7-89fe-21fffab26adf" containerName="istio-proxy" Oct 30 22:28:27.835688 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:27.835482 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="47da8d4b-2ae1-459b-8d7b-522745a5fd56" containerName="maas-api" Oct 30 22:28:27.840589 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:27.840570 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q" Oct 30 22:28:27.898550 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:27.898523 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q"] Oct 30 22:28:27.911992 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:27.911679 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de7b01aa-6f28-4f10-b7ce-b615537fb99f-openshift-service-ca\") pod \"maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q\" (UID: \"de7b01aa-6f28-4f10-b7ce-b615537fb99f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q" Oct 30 22:28:27.911992 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:27.911734 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de7b01aa-6f28-4f10-b7ce-b615537fb99f-metrics-certs\") pod \"maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q\" (UID: \"de7b01aa-6f28-4f10-b7ce-b615537fb99f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q" Oct 30 22:28:27.912208 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:27.912033 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjrxj\" (UniqueName: \"kubernetes.io/projected/de7b01aa-6f28-4f10-b7ce-b615537fb99f-kube-api-access-zjrxj\") pod \"maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q\" (UID: \"de7b01aa-6f28-4f10-b7ce-b615537fb99f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q" Oct 30 22:28:27.912208 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:27.912161 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de7b01aa-6f28-4f10-b7ce-b615537fb99f-maas-api-tls\") pod \"maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q\" (UID: \"de7b01aa-6f28-4f10-b7ce-b615537fb99f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q" Oct 30 22:28:27.919091 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:27.919071 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d59c58-serving-cert\"" Oct 30 22:28:28.013265 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:28.013195 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjrxj\" (UniqueName: \"kubernetes.io/projected/de7b01aa-6f28-4f10-b7ce-b615537fb99f-kube-api-access-zjrxj\") pod \"maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q\" (UID: \"de7b01aa-6f28-4f10-b7ce-b615537fb99f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q" Oct 30 22:28:28.013455 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:28.013288 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de7b01aa-6f28-4f10-b7ce-b615537fb99f-maas-api-tls\") pod \"maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q\" (UID: \"de7b01aa-6f28-4f10-b7ce-b615537fb99f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q" Oct 30 22:28:28.013455 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:28:28.013418 5918 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-d59c58-serving-cert: secret "maas-api-e2e-api-b-d59c58-serving-cert" not found Oct 30 22:28:28.013596 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:28.013451 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de7b01aa-6f28-4f10-b7ce-b615537fb99f-openshift-service-ca\") pod \"maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q\" (UID: \"de7b01aa-6f28-4f10-b7ce-b615537fb99f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q" Oct 30 22:28:28.013596 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:28:28.013485 5918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de7b01aa-6f28-4f10-b7ce-b615537fb99f-maas-api-tls podName:de7b01aa-6f28-4f10-b7ce-b615537fb99f nodeName:}" failed. No retries permitted until 2026-10-30 22:28:28.51346423 +0000 UTC m=+2268.784162318 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/de7b01aa-6f28-4f10-b7ce-b615537fb99f-maas-api-tls") pod "maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q" (UID: "de7b01aa-6f28-4f10-b7ce-b615537fb99f") : secret "maas-api-e2e-api-b-d59c58-serving-cert" not found Oct 30 22:28:28.013596 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:28.013507 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de7b01aa-6f28-4f10-b7ce-b615537fb99f-metrics-certs\") pod \"maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q\" (UID: \"de7b01aa-6f28-4f10-b7ce-b615537fb99f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q" Oct 30 22:28:28.014148 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:28.014126 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de7b01aa-6f28-4f10-b7ce-b615537fb99f-openshift-service-ca\") pod \"maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q\" (UID: \"de7b01aa-6f28-4f10-b7ce-b615537fb99f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q" Oct 30 22:28:28.016554 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:28.016536 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de7b01aa-6f28-4f10-b7ce-b615537fb99f-metrics-certs\") pod \"maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q\" (UID: \"de7b01aa-6f28-4f10-b7ce-b615537fb99f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q" Oct 30 22:28:28.077527 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:28.077490 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjrxj\" (UniqueName: \"kubernetes.io/projected/de7b01aa-6f28-4f10-b7ce-b615537fb99f-kube-api-access-zjrxj\") pod \"maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q\" (UID: \"de7b01aa-6f28-4f10-b7ce-b615537fb99f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q" Oct 30 22:28:28.520285 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:28.520253 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de7b01aa-6f28-4f10-b7ce-b615537fb99f-maas-api-tls\") pod \"maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q\" (UID: \"de7b01aa-6f28-4f10-b7ce-b615537fb99f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q" Oct 30 22:28:28.522870 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:28.522842 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de7b01aa-6f28-4f10-b7ce-b615537fb99f-maas-api-tls\") pod \"maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q\" (UID: \"de7b01aa-6f28-4f10-b7ce-b615537fb99f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q" Oct 30 22:28:28.756035 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:28.755991 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q" Oct 30 22:28:29.151371 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:29.151341 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q"] Oct 30 22:28:29.152707 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:28:29.152678 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde7b01aa_6f28_4f10_b7ce_b615537fb99f.slice/crio-04a1abd867370cac0f885ee4a296d9511a70f8e8c375fd6d4e1f6113be9d5d95 WatchSource:0}: Error finding container 04a1abd867370cac0f885ee4a296d9511a70f8e8c375fd6d4e1f6113be9d5d95: Status 404 returned error can't find the container with id 04a1abd867370cac0f885ee4a296d9511a70f8e8c375fd6d4e1f6113be9d5d95 Oct 30 22:28:29.154269 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:29.154250 5918 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:28:29.209317 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:29.209277 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q" event={"ID":"de7b01aa-6f28-4f10-b7ce-b615537fb99f","Type":"ContainerStarted","Data":"04a1abd867370cac0f885ee4a296d9511a70f8e8c375fd6d4e1f6113be9d5d95"} Oct 30 22:28:30.216853 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:30.216752 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q" event={"ID":"de7b01aa-6f28-4f10-b7ce-b615537fb99f","Type":"ContainerStarted","Data":"dca405e4d6eaa159fcd10cebae2823d3a64d045b2012aff63a79a5c38b85f83e"} Oct 30 22:28:30.216853 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:30.216841 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q" Oct 30 22:28:30.341227 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:30.341157 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q" podStartSLOduration=2.548672995 podStartE2EDuration="3.341138485s" totalImagesPullingTime="792.46549ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:30.337262413 +0000 UTC m=+2270.607960495" watchObservedRunningTime="2026-10-30 22:28:30.341138485 +0000 UTC m=+2270.611836570" Oct 30 22:28:32.197778 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:32.197741 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2"] Oct 30 22:28:32.198278 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:32.197997 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" podUID="f6f37484-37b2-4fab-8b6b-cdc1c171ff93" containerName="istio-proxy" containerID="cri-o://cadd378cbd16596b10517a87e400a134074a3a4b3959ccd5c5b27a27bbe420e5" gracePeriod=30 Oct 30 22:28:36.228396 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:36.228367 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q" Oct 30 22:28:37.473886 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:37.473828 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 22:28:37.528616 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:37.528583 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-istio-envoy\") pod \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\" (UID: \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\") " Oct 30 22:28:37.528794 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:37.528632 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-credential-socket\") pod \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\" (UID: \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\") " Oct 30 22:28:37.528794 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:37.528673 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-istio-podinfo\") pod \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\" (UID: \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\") " Oct 30 22:28:37.528794 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:37.528715 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-istiod-ca-cert\") pod \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\" (UID: \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\") " Oct 30 22:28:37.528794 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:37.528744 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-istio-token\" (UniqueName: \"kubernetes.io/projected/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-istio-token\") pod \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\" (UID: \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\") " Oct 30 22:28:37.528794 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:37.528796 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-kube-api-access-8bg6f\" (UniqueName: \"kubernetes.io/projected/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-kube-api-access-8bg6f\") pod \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\" (UID: \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\") " Oct 30 22:28:37.529083 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:37.528858 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-workload-certs\") pod \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\" (UID: \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\") " Oct 30 22:28:37.529083 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:37.528904 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-istio-data\") pod \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\" (UID: \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\") " Oct 30 22:28:37.529083 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:37.528992 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-credential-socket" pod "f6f37484-37b2-4fab-8b6b-cdc1c171ff93" (UID: "f6f37484-37b2-4fab-8b6b-cdc1c171ff93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:37.529083 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:37.529036 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-workload-socket\") pod \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\" (UID: \"f6f37484-37b2-4fab-8b6b-cdc1c171ff93\") " Oct 30 22:28:37.529318 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:37.529245 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-istio-data" pod "f6f37484-37b2-4fab-8b6b-cdc1c171ff93" (UID: "f6f37484-37b2-4fab-8b6b-cdc1c171ff93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:37.529318 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:37.529277 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-istiod-ca-cert" pod "f6f37484-37b2-4fab-8b6b-cdc1c171ff93" (UID: "f6f37484-37b2-4fab-8b6b-cdc1c171ff93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:37.529470 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:37.529445 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-workload-socket" pod "f6f37484-37b2-4fab-8b6b-cdc1c171ff93" (UID: "f6f37484-37b2-4fab-8b6b-cdc1c171ff93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:37.529470 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:37.529435 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-workload-certs" pod "f6f37484-37b2-4fab-8b6b-cdc1c171ff93" (UID: "f6f37484-37b2-4fab-8b6b-cdc1c171ff93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:37.529827 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:37.529792 5918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-istio-data\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:37.529827 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:37.529819 5918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-workload-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:37.529965 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:37.529834 5918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-credential-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:37.529965 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:37.529850 5918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-istiod-ca-cert\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:37.529965 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:37.529864 5918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-workload-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:37.531107 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:37.531087 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-istio-token" pod "f6f37484-37b2-4fab-8b6b-cdc1c171ff93" (UID: "f6f37484-37b2-4fab-8b6b-cdc1c171ff93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:37.531299 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:37.531279 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-kube-api-access-8bg6f" pod "f6f37484-37b2-4fab-8b6b-cdc1c171ff93" (UID: "f6f37484-37b2-4fab-8b6b-cdc1c171ff93"). InnerVolumeSpecName "kube-api-access-8bg6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:37.531479 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:37.531456 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-istio-envoy" pod "f6f37484-37b2-4fab-8b6b-cdc1c171ff93" (UID: "f6f37484-37b2-4fab-8b6b-cdc1c171ff93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:37.531571 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:37.531472 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-istio-podinfo" pod "f6f37484-37b2-4fab-8b6b-cdc1c171ff93" (UID: "f6f37484-37b2-4fab-8b6b-cdc1c171ff93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:28:37.630917 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:37.630883 5918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-istio-envoy\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:37.630917 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:37.630919 5918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-istio-podinfo\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:37.631120 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:37.630934 5918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-istio-token\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:37.631120 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:37.630950 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bg6f\" (UniqueName: \"kubernetes.io/projected/f6f37484-37b2-4fab-8b6b-cdc1c171ff93-kube-api-access-8bg6f\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:38.275922 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:38.275890 5918 generic.go:356] "Generic (PLEG): container finished" podID="f6f37484-37b2-4fab-8b6b-cdc1c171ff93" containerID="cadd378cbd16596b10517a87e400a134074a3a4b3959ccd5c5b27a27bbe420e5" exitCode=0 Oct 30 22:28:38.276120 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:38.275942 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" Oct 30 22:28:38.276120 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:38.275965 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" event={"ID":"f6f37484-37b2-4fab-8b6b-cdc1c171ff93","Type":"ContainerDied","Data":"cadd378cbd16596b10517a87e400a134074a3a4b3959ccd5c5b27a27bbe420e5"} Oct 30 22:28:38.276120 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:38.276026 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2" event={"ID":"f6f37484-37b2-4fab-8b6b-cdc1c171ff93","Type":"ContainerDied","Data":"a135c2cd5dc55856c09c588c4e38c224992f548503d768c10fdb31c49b36be56"} Oct 30 22:28:38.276120 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:38.276056 5918 scope.go:122] "RemoveContainer" containerID="cadd378cbd16596b10517a87e400a134074a3a4b3959ccd5c5b27a27bbe420e5" Oct 30 22:28:38.285718 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:38.285694 5918 scope.go:122] "RemoveContainer" containerID="b3d14947d3b89637bb8d139bdff16d91ff5cd35a461de9324822e4b4fbbfc0d7" Oct 30 22:28:38.294406 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:38.294388 5918 scope.go:122] "RemoveContainer" containerID="cadd378cbd16596b10517a87e400a134074a3a4b3959ccd5c5b27a27bbe420e5" Oct 30 22:28:38.294689 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:28:38.294666 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cadd378cbd16596b10517a87e400a134074a3a4b3959ccd5c5b27a27bbe420e5\": container with ID starting with cadd378cbd16596b10517a87e400a134074a3a4b3959ccd5c5b27a27bbe420e5 not found: ID does not exist" containerID="cadd378cbd16596b10517a87e400a134074a3a4b3959ccd5c5b27a27bbe420e5" Oct 30 22:28:38.294741 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:38.294698 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cadd378cbd16596b10517a87e400a134074a3a4b3959ccd5c5b27a27bbe420e5"} err="failed to get container status \"cadd378cbd16596b10517a87e400a134074a3a4b3959ccd5c5b27a27bbe420e5\": rpc error: code = NotFound desc = could not find container \"cadd378cbd16596b10517a87e400a134074a3a4b3959ccd5c5b27a27bbe420e5\": container with ID starting with cadd378cbd16596b10517a87e400a134074a3a4b3959ccd5c5b27a27bbe420e5 not found: ID does not exist" Oct 30 22:28:38.294741 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:38.294715 5918 scope.go:122] "RemoveContainer" containerID="b3d14947d3b89637bb8d139bdff16d91ff5cd35a461de9324822e4b4fbbfc0d7" Oct 30 22:28:38.294974 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:28:38.294957 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3d14947d3b89637bb8d139bdff16d91ff5cd35a461de9324822e4b4fbbfc0d7\": container with ID starting with b3d14947d3b89637bb8d139bdff16d91ff5cd35a461de9324822e4b4fbbfc0d7 not found: ID does not exist" containerID="b3d14947d3b89637bb8d139bdff16d91ff5cd35a461de9324822e4b4fbbfc0d7" Oct 30 22:28:38.295015 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:38.294980 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3d14947d3b89637bb8d139bdff16d91ff5cd35a461de9324822e4b4fbbfc0d7"} err="failed to get container status \"b3d14947d3b89637bb8d139bdff16d91ff5cd35a461de9324822e4b4fbbfc0d7\": rpc error: code = NotFound desc = could not find container \"b3d14947d3b89637bb8d139bdff16d91ff5cd35a461de9324822e4b4fbbfc0d7\": container with ID starting with b3d14947d3b89637bb8d139bdff16d91ff5cd35a461de9324822e4b4fbbfc0d7 not found: ID does not exist" Oct 30 22:28:38.495964 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:38.495932 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2"] Oct 30 22:28:38.546375 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:38.546295 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9cff60-openshift-default-d6667c5f9-29zd2"] Oct 30 22:28:39.684882 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:39.684818 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" podUID="ab957b47-90d6-497a-8334-1d5d4541dc7a" containerName="main" containerID="cri-o://b5c70fc58409e56bd674ac90b399e2730cb92cf95a979154f6bb834c5aa78d26" gracePeriod=45 Oct 30 22:28:39.693761 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:39.693730 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" podUID="dc20dd97-eb75-42c5-8e50-6b0b5cb54b45" containerName="main" containerID="cri-o://e35ba9adf08cfe1479cba187071134c39cdcbec689be01b01a492a5a363b367e" gracePeriod=45 Oct 30 22:28:39.852645 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:39.852613 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6f37484-37b2-4fab-8b6b-cdc1c171ff93" path="/var/lib/kubelet/pods/f6f37484-37b2-4fab-8b6b-cdc1c171ff93/volumes" Oct 30 22:28:39.968019 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:39.967991 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" Oct 30 22:28:39.970153 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:39.970133 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" Oct 30 22:28:40.057085 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.057049 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab957b47-90d6-497a-8334-1d5d4541dc7a-home\" (UniqueName: \"kubernetes.io/empty-dir/ab957b47-90d6-497a-8334-1d5d4541dc7a-home\") pod \"ab957b47-90d6-497a-8334-1d5d4541dc7a\" (UID: \"ab957b47-90d6-497a-8334-1d5d4541dc7a\") " Oct 30 22:28:40.057317 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.057096 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab957b47-90d6-497a-8334-1d5d4541dc7a-kube-api-access-lt8m6\" (UniqueName: \"kubernetes.io/projected/ab957b47-90d6-497a-8334-1d5d4541dc7a-kube-api-access-lt8m6\") pod \"ab957b47-90d6-497a-8334-1d5d4541dc7a\" (UID: \"ab957b47-90d6-497a-8334-1d5d4541dc7a\") " Oct 30 22:28:40.057317 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.057137 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-model-cache\") pod \"dc20dd97-eb75-42c5-8e50-6b0b5cb54b45\" (UID: \"dc20dd97-eb75-42c5-8e50-6b0b5cb54b45\") " Oct 30 22:28:40.057317 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.057191 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-home\" (UniqueName: \"kubernetes.io/empty-dir/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-home\") pod \"dc20dd97-eb75-42c5-8e50-6b0b5cb54b45\" (UID: \"dc20dd97-eb75-42c5-8e50-6b0b5cb54b45\") " Oct 30 22:28:40.057317 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.057283 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab957b47-90d6-497a-8334-1d5d4541dc7a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab957b47-90d6-497a-8334-1d5d4541dc7a-model-cache\") pod \"ab957b47-90d6-497a-8334-1d5d4541dc7a\" (UID: \"ab957b47-90d6-497a-8334-1d5d4541dc7a\") " Oct 30 22:28:40.057536 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.057368 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab957b47-90d6-497a-8334-1d5d4541dc7a-home" pod "ab957b47-90d6-497a-8334-1d5d4541dc7a" (UID: "ab957b47-90d6-497a-8334-1d5d4541dc7a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:40.057536 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.057395 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-tls-certs\" (UniqueName: \"kubernetes.io/secret/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-tls-certs\") pod \"dc20dd97-eb75-42c5-8e50-6b0b5cb54b45\" (UID: \"dc20dd97-eb75-42c5-8e50-6b0b5cb54b45\") " Oct 30 22:28:40.057536 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.057426 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-model-cache" pod "dc20dd97-eb75-42c5-8e50-6b0b5cb54b45" (UID: "dc20dd97-eb75-42c5-8e50-6b0b5cb54b45"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:40.057536 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.057450 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-tmp-dir\") pod \"dc20dd97-eb75-42c5-8e50-6b0b5cb54b45\" (UID: \"dc20dd97-eb75-42c5-8e50-6b0b5cb54b45\") " Oct 30 22:28:40.057738 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.057550 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab957b47-90d6-497a-8334-1d5d4541dc7a-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab957b47-90d6-497a-8334-1d5d4541dc7a-tls-certs\") pod \"ab957b47-90d6-497a-8334-1d5d4541dc7a\" (UID: \"ab957b47-90d6-497a-8334-1d5d4541dc7a\") " Oct 30 22:28:40.057738 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.057589 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab957b47-90d6-497a-8334-1d5d4541dc7a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab957b47-90d6-497a-8334-1d5d4541dc7a-tmp-dir\") pod \"ab957b47-90d6-497a-8334-1d5d4541dc7a\" (UID: \"ab957b47-90d6-497a-8334-1d5d4541dc7a\") " Oct 30 22:28:40.057738 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.057606 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab957b47-90d6-497a-8334-1d5d4541dc7a-model-cache" pod "ab957b47-90d6-497a-8334-1d5d4541dc7a" (UID: "ab957b47-90d6-497a-8334-1d5d4541dc7a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:40.057738 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.057612 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-dshm\") pod \"dc20dd97-eb75-42c5-8e50-6b0b5cb54b45\" (UID: \"dc20dd97-eb75-42c5-8e50-6b0b5cb54b45\") " Oct 30 22:28:40.057738 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.057661 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab957b47-90d6-497a-8334-1d5d4541dc7a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab957b47-90d6-497a-8334-1d5d4541dc7a-dshm\") pod \"ab957b47-90d6-497a-8334-1d5d4541dc7a\" (UID: \"ab957b47-90d6-497a-8334-1d5d4541dc7a\") " Oct 30 22:28:40.057988 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.057735 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-kube-api-access-jvkrd\" (UniqueName: \"kubernetes.io/projected/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-kube-api-access-jvkrd\") pod \"dc20dd97-eb75-42c5-8e50-6b0b5cb54b45\" (UID: \"dc20dd97-eb75-42c5-8e50-6b0b5cb54b45\") " Oct 30 22:28:40.057988 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.057804 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab957b47-90d6-497a-8334-1d5d4541dc7a-tmp-dir" pod "ab957b47-90d6-497a-8334-1d5d4541dc7a" (UID: "ab957b47-90d6-497a-8334-1d5d4541dc7a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:40.058306 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.058254 5918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab957b47-90d6-497a-8334-1d5d4541dc7a-home\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:40.058306 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.058276 5918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-model-cache\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:40.058306 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.058288 5918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab957b47-90d6-497a-8334-1d5d4541dc7a-model-cache\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:40.058306 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.058300 5918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab957b47-90d6-497a-8334-1d5d4541dc7a-tmp-dir\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:40.059748 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.059724 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab957b47-90d6-497a-8334-1d5d4541dc7a-kube-api-access-lt8m6" pod "ab957b47-90d6-497a-8334-1d5d4541dc7a" (UID: "ab957b47-90d6-497a-8334-1d5d4541dc7a"). InnerVolumeSpecName "kube-api-access-lt8m6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:40.059995 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.059977 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab957b47-90d6-497a-8334-1d5d4541dc7a-dshm" pod "ab957b47-90d6-497a-8334-1d5d4541dc7a" (UID: "ab957b47-90d6-497a-8334-1d5d4541dc7a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:40.060325 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.060307 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab957b47-90d6-497a-8334-1d5d4541dc7a-tls-certs" pod "ab957b47-90d6-497a-8334-1d5d4541dc7a" (UID: "ab957b47-90d6-497a-8334-1d5d4541dc7a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:28:40.061272 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.061242 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-home" pod "dc20dd97-eb75-42c5-8e50-6b0b5cb54b45" (UID: "dc20dd97-eb75-42c5-8e50-6b0b5cb54b45"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:40.061424 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.061406 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-tmp-dir" pod "dc20dd97-eb75-42c5-8e50-6b0b5cb54b45" (UID: "dc20dd97-eb75-42c5-8e50-6b0b5cb54b45"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:40.063161 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.063136 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-tls-certs" pod "dc20dd97-eb75-42c5-8e50-6b0b5cb54b45" (UID: "dc20dd97-eb75-42c5-8e50-6b0b5cb54b45"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:28:40.063299 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.063182 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-dshm" pod "dc20dd97-eb75-42c5-8e50-6b0b5cb54b45" (UID: "dc20dd97-eb75-42c5-8e50-6b0b5cb54b45"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:40.063350 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.063317 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-kube-api-access-jvkrd" pod "dc20dd97-eb75-42c5-8e50-6b0b5cb54b45" (UID: "dc20dd97-eb75-42c5-8e50-6b0b5cb54b45"). InnerVolumeSpecName "kube-api-access-jvkrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:40.159995 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.159955 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt8m6\" (UniqueName: \"kubernetes.io/projected/ab957b47-90d6-497a-8334-1d5d4541dc7a-kube-api-access-lt8m6\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:40.159995 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.159989 5918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-home\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:40.160188 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.160004 5918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-tls-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:40.160188 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.160017 5918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-tmp-dir\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:40.160188 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.160033 5918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab957b47-90d6-497a-8334-1d5d4541dc7a-tls-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:40.160188 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.160045 5918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-dshm\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:40.160188 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.160057 5918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab957b47-90d6-497a-8334-1d5d4541dc7a-dshm\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:40.160188 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.160070 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvkrd\" (UniqueName: \"kubernetes.io/projected/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45-kube-api-access-jvkrd\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:40.292639 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.292555 5918 generic.go:356] "Generic (PLEG): container finished" podID="dc20dd97-eb75-42c5-8e50-6b0b5cb54b45" containerID="e35ba9adf08cfe1479cba187071134c39cdcbec689be01b01a492a5a363b367e" exitCode=0 Oct 30 22:28:40.292817 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.292673 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" Oct 30 22:28:40.292817 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.292678 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" event={"ID":"dc20dd97-eb75-42c5-8e50-6b0b5cb54b45","Type":"ContainerDied","Data":"e35ba9adf08cfe1479cba187071134c39cdcbec689be01b01a492a5a363b367e"} Oct 30 22:28:40.292817 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.292803 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z" event={"ID":"dc20dd97-eb75-42c5-8e50-6b0b5cb54b45","Type":"ContainerDied","Data":"21fffba930719395b19afb49792b4c169d99fb63b76ceec954625edcb36e3f85"} Oct 30 22:28:40.293003 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.292821 5918 scope.go:122] "RemoveContainer" containerID="e35ba9adf08cfe1479cba187071134c39cdcbec689be01b01a492a5a363b367e" Oct 30 22:28:40.294181 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.294149 5918 generic.go:356] "Generic (PLEG): container finished" podID="ab957b47-90d6-497a-8334-1d5d4541dc7a" containerID="b5c70fc58409e56bd674ac90b399e2730cb92cf95a979154f6bb834c5aa78d26" exitCode=0 Oct 30 22:28:40.294318 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.294189 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" Oct 30 22:28:40.294318 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.294201 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" event={"ID":"ab957b47-90d6-497a-8334-1d5d4541dc7a","Type":"ContainerDied","Data":"b5c70fc58409e56bd674ac90b399e2730cb92cf95a979154f6bb834c5aa78d26"} Oct 30 22:28:40.294318 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.294276 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87" event={"ID":"ab957b47-90d6-497a-8334-1d5d4541dc7a","Type":"ContainerDied","Data":"5fa669a7da27ae5fc98587b81a457e179e098ec5e968a828c1f4f49aced53d67"} Oct 30 22:28:40.304319 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.304301 5918 scope.go:122] "RemoveContainer" containerID="e35ba9adf08cfe1479cba187071134c39cdcbec689be01b01a492a5a363b367e" Oct 30 22:28:40.304670 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:28:40.304597 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e35ba9adf08cfe1479cba187071134c39cdcbec689be01b01a492a5a363b367e\": container with ID starting with e35ba9adf08cfe1479cba187071134c39cdcbec689be01b01a492a5a363b367e not found: ID does not exist" containerID="e35ba9adf08cfe1479cba187071134c39cdcbec689be01b01a492a5a363b367e" Oct 30 22:28:40.304670 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.304623 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e35ba9adf08cfe1479cba187071134c39cdcbec689be01b01a492a5a363b367e"} err="failed to get container status \"e35ba9adf08cfe1479cba187071134c39cdcbec689be01b01a492a5a363b367e\": rpc error: code = NotFound desc = could not find container \"e35ba9adf08cfe1479cba187071134c39cdcbec689be01b01a492a5a363b367e\": container with ID starting with e35ba9adf08cfe1479cba187071134c39cdcbec689be01b01a492a5a363b367e not found: ID does not exist" Oct 30 22:28:40.304670 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.304641 5918 scope.go:122] "RemoveContainer" containerID="b5c70fc58409e56bd674ac90b399e2730cb92cf95a979154f6bb834c5aa78d26" Oct 30 22:28:40.313176 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.313159 5918 scope.go:122] "RemoveContainer" containerID="b5c70fc58409e56bd674ac90b399e2730cb92cf95a979154f6bb834c5aa78d26" Oct 30 22:28:40.313474 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:28:40.313455 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5c70fc58409e56bd674ac90b399e2730cb92cf95a979154f6bb834c5aa78d26\": container with ID starting with b5c70fc58409e56bd674ac90b399e2730cb92cf95a979154f6bb834c5aa78d26 not found: ID does not exist" containerID="b5c70fc58409e56bd674ac90b399e2730cb92cf95a979154f6bb834c5aa78d26" Oct 30 22:28:40.313564 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.313477 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5c70fc58409e56bd674ac90b399e2730cb92cf95a979154f6bb834c5aa78d26"} err="failed to get container status \"b5c70fc58409e56bd674ac90b399e2730cb92cf95a979154f6bb834c5aa78d26\": rpc error: code = NotFound desc = could not find container \"b5c70fc58409e56bd674ac90b399e2730cb92cf95a979154f6bb834c5aa78d26\": container with ID starting with b5c70fc58409e56bd674ac90b399e2730cb92cf95a979154f6bb834c5aa78d26 not found: ID does not exist" Oct 30 22:28:40.463133 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.463100 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87"] Oct 30 22:28:40.470801 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.470769 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-d8837a/distinct-2-w2-d8837a-kserve-597c4754df-jxg87"] Oct 30 22:28:40.583537 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.583452 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z"] Oct 30 22:28:40.591004 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:40.590975 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-d8837a/distinct-w2-d8837a-kserve-7f67bfb654-ph88z"] Oct 30 22:28:41.852237 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:41.852186 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab957b47-90d6-497a-8334-1d5d4541dc7a" path="/var/lib/kubelet/pods/ab957b47-90d6-497a-8334-1d5d4541dc7a/volumes" Oct 30 22:28:41.852635 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:41.852620 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc20dd97-eb75-42c5-8e50-6b0b5cb54b45" path="/var/lib/kubelet/pods/dc20dd97-eb75-42c5-8e50-6b0b5cb54b45/volumes" Oct 30 22:28:42.052020 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:42.051987 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf"] Oct 30 22:28:42.052273 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:42.052209 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" podUID="97f4e41b-50c7-441c-8ff8-6c815143b79b" containerName="istio-proxy" containerID="cri-o://25bcbdee8822855109eeeb2cfb7ccc96d78849f535cabc5ab6710be39b35c7a9" gracePeriod=30 Oct 30 22:28:42.071260 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:42.071234 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm"] Oct 30 22:28:42.071433 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:42.071414 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" podUID="100467f8-8ee2-4cda-a9a7-f52b11af838c" containerName="istio-proxy" containerID="cri-o://b02729058eec38f80828a0236a705def98d2dfe3be1d5ab611632ce2d47815fd" gracePeriod=30 Oct 30 22:28:42.081687 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:42.081659 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v"] Oct 30 22:28:42.081890 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:42.081866 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" podUID="e0018134-1144-4076-95ca-7383b46a2985" containerName="istio-proxy" containerID="cri-o://3dcf52a5b85487a2bc84f4e33aaad4633963baa5afdef5b05528010a06eb2a24" gracePeriod=30 Oct 30 22:28:47.342394 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.342159 5918 generic.go:356] "Generic (PLEG): container finished" podID="100467f8-8ee2-4cda-a9a7-f52b11af838c" containerID="b02729058eec38f80828a0236a705def98d2dfe3be1d5ab611632ce2d47815fd" exitCode=0 Oct 30 22:28:47.342394 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.342321 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" event={"ID":"100467f8-8ee2-4cda-a9a7-f52b11af838c","Type":"ContainerDied","Data":"b02729058eec38f80828a0236a705def98d2dfe3be1d5ab611632ce2d47815fd"} Oct 30 22:28:47.344122 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.344096 5918 generic.go:356] "Generic (PLEG): container finished" podID="e0018134-1144-4076-95ca-7383b46a2985" containerID="3dcf52a5b85487a2bc84f4e33aaad4633963baa5afdef5b05528010a06eb2a24" exitCode=0 Oct 30 22:28:47.344282 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.344174 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" event={"ID":"e0018134-1144-4076-95ca-7383b46a2985","Type":"ContainerDied","Data":"3dcf52a5b85487a2bc84f4e33aaad4633963baa5afdef5b05528010a06eb2a24"} Oct 30 22:28:47.345838 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.345810 5918 generic.go:356] "Generic (PLEG): container finished" podID="97f4e41b-50c7-441c-8ff8-6c815143b79b" containerID="25bcbdee8822855109eeeb2cfb7ccc96d78849f535cabc5ab6710be39b35c7a9" exitCode=0 Oct 30 22:28:47.345956 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.345940 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" event={"ID":"97f4e41b-50c7-441c-8ff8-6c815143b79b","Type":"ContainerDied","Data":"25bcbdee8822855109eeeb2cfb7ccc96d78849f535cabc5ab6710be39b35c7a9"} Oct 30 22:28:47.346003 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.345986 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" event={"ID":"97f4e41b-50c7-441c-8ff8-6c815143b79b","Type":"ContainerDied","Data":"5acbcea12adf28865a62c403a12968a32d336a9690143be3edfb27e2e6e51a63"} Oct 30 22:28:47.346038 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.346011 5918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5acbcea12adf28865a62c403a12968a32d336a9690143be3edfb27e2e6e51a63" Oct 30 22:28:47.350322 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.350302 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" Oct 30 22:28:47.382289 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.382267 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" Oct 30 22:28:47.393864 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.393843 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" Oct 30 22:28:47.442991 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.442958 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97f4e41b-50c7-441c-8ff8-6c815143b79b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f4e41b-50c7-441c-8ff8-6c815143b79b-workload-socket\") pod \"97f4e41b-50c7-441c-8ff8-6c815143b79b\" (UID: \"97f4e41b-50c7-441c-8ff8-6c815143b79b\") " Oct 30 22:28:47.443126 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.443002 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97f4e41b-50c7-441c-8ff8-6c815143b79b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f4e41b-50c7-441c-8ff8-6c815143b79b-istio-envoy\") pod \"97f4e41b-50c7-441c-8ff8-6c815143b79b\" (UID: \"97f4e41b-50c7-441c-8ff8-6c815143b79b\") " Oct 30 22:28:47.443126 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.443054 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97f4e41b-50c7-441c-8ff8-6c815143b79b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f4e41b-50c7-441c-8ff8-6c815143b79b-workload-certs\") pod \"97f4e41b-50c7-441c-8ff8-6c815143b79b\" (UID: \"97f4e41b-50c7-441c-8ff8-6c815143b79b\") " Oct 30 22:28:47.443198 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.443169 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97f4e41b-50c7-441c-8ff8-6c815143b79b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f4e41b-50c7-441c-8ff8-6c815143b79b-istio-data\") pod \"97f4e41b-50c7-441c-8ff8-6c815143b79b\" (UID: \"97f4e41b-50c7-441c-8ff8-6c815143b79b\") " Oct 30 22:28:47.443266 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.443200 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97f4e41b-50c7-441c-8ff8-6c815143b79b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f4e41b-50c7-441c-8ff8-6c815143b79b-credential-socket\") pod \"97f4e41b-50c7-441c-8ff8-6c815143b79b\" (UID: \"97f4e41b-50c7-441c-8ff8-6c815143b79b\") " Oct 30 22:28:47.443317 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.443256 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f4e41b-50c7-441c-8ff8-6c815143b79b-workload-socket" pod "97f4e41b-50c7-441c-8ff8-6c815143b79b" (UID: "97f4e41b-50c7-441c-8ff8-6c815143b79b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:47.443317 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.443281 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97f4e41b-50c7-441c-8ff8-6c815143b79b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f4e41b-50c7-441c-8ff8-6c815143b79b-istio-podinfo\") pod \"97f4e41b-50c7-441c-8ff8-6c815143b79b\" (UID: \"97f4e41b-50c7-441c-8ff8-6c815143b79b\") " Oct 30 22:28:47.443411 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.443335 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97f4e41b-50c7-441c-8ff8-6c815143b79b-kube-api-access-wnbq6\" (UniqueName: \"kubernetes.io/projected/97f4e41b-50c7-441c-8ff8-6c815143b79b-kube-api-access-wnbq6\") pod \"97f4e41b-50c7-441c-8ff8-6c815143b79b\" (UID: \"97f4e41b-50c7-441c-8ff8-6c815143b79b\") " Oct 30 22:28:47.443411 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.443376 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97f4e41b-50c7-441c-8ff8-6c815143b79b-istio-token\" (UniqueName: \"kubernetes.io/projected/97f4e41b-50c7-441c-8ff8-6c815143b79b-istio-token\") pod \"97f4e41b-50c7-441c-8ff8-6c815143b79b\" (UID: \"97f4e41b-50c7-441c-8ff8-6c815143b79b\") " Oct 30 22:28:47.443411 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.443402 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97f4e41b-50c7-441c-8ff8-6c815143b79b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f4e41b-50c7-441c-8ff8-6c815143b79b-istiod-ca-cert\") pod \"97f4e41b-50c7-441c-8ff8-6c815143b79b\" (UID: \"97f4e41b-50c7-441c-8ff8-6c815143b79b\") " Oct 30 22:28:47.443571 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.443452 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f4e41b-50c7-441c-8ff8-6c815143b79b-workload-certs" pod "97f4e41b-50c7-441c-8ff8-6c815143b79b" (UID: "97f4e41b-50c7-441c-8ff8-6c815143b79b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:47.443571 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.443524 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f4e41b-50c7-441c-8ff8-6c815143b79b-credential-socket" pod "97f4e41b-50c7-441c-8ff8-6c815143b79b" (UID: "97f4e41b-50c7-441c-8ff8-6c815143b79b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:47.443571 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.443537 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f4e41b-50c7-441c-8ff8-6c815143b79b-istio-data" pod "97f4e41b-50c7-441c-8ff8-6c815143b79b" (UID: "97f4e41b-50c7-441c-8ff8-6c815143b79b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:47.443948 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.443918 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97f4e41b-50c7-441c-8ff8-6c815143b79b-istiod-ca-cert" pod "97f4e41b-50c7-441c-8ff8-6c815143b79b" (UID: "97f4e41b-50c7-441c-8ff8-6c815143b79b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:47.444204 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.444012 5918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f4e41b-50c7-441c-8ff8-6c815143b79b-istio-data\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:47.444204 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.444032 5918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f4e41b-50c7-441c-8ff8-6c815143b79b-credential-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:47.444204 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.444046 5918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f4e41b-50c7-441c-8ff8-6c815143b79b-workload-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:47.444204 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.444054 5918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f4e41b-50c7-441c-8ff8-6c815143b79b-workload-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:47.446056 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.446033 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f4e41b-50c7-441c-8ff8-6c815143b79b-kube-api-access-wnbq6" pod "97f4e41b-50c7-441c-8ff8-6c815143b79b" (UID: "97f4e41b-50c7-441c-8ff8-6c815143b79b"). InnerVolumeSpecName "kube-api-access-wnbq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:47.446056 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.446043 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f4e41b-50c7-441c-8ff8-6c815143b79b-istio-token" pod "97f4e41b-50c7-441c-8ff8-6c815143b79b" (UID: "97f4e41b-50c7-441c-8ff8-6c815143b79b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:47.446200 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.446059 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97f4e41b-50c7-441c-8ff8-6c815143b79b-istio-podinfo" pod "97f4e41b-50c7-441c-8ff8-6c815143b79b" (UID: "97f4e41b-50c7-441c-8ff8-6c815143b79b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:28:47.446200 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.446064 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f4e41b-50c7-441c-8ff8-6c815143b79b-istio-envoy" pod "97f4e41b-50c7-441c-8ff8-6c815143b79b" (UID: "97f4e41b-50c7-441c-8ff8-6c815143b79b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:47.544984 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.544714 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/100467f8-8ee2-4cda-a9a7-f52b11af838c-kube-api-access-7fhpx\" (UniqueName: \"kubernetes.io/projected/100467f8-8ee2-4cda-a9a7-f52b11af838c-kube-api-access-7fhpx\") pod \"100467f8-8ee2-4cda-a9a7-f52b11af838c\" (UID: \"100467f8-8ee2-4cda-a9a7-f52b11af838c\") " Oct 30 22:28:47.544984 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.544774 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0018134-1144-4076-95ca-7383b46a2985-istio-token\" (UniqueName: \"kubernetes.io/projected/e0018134-1144-4076-95ca-7383b46a2985-istio-token\") pod \"e0018134-1144-4076-95ca-7383b46a2985\" (UID: \"e0018134-1144-4076-95ca-7383b46a2985\") " Oct 30 22:28:47.544984 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.544820 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/100467f8-8ee2-4cda-a9a7-f52b11af838c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/100467f8-8ee2-4cda-a9a7-f52b11af838c-istio-data\") pod \"100467f8-8ee2-4cda-a9a7-f52b11af838c\" (UID: \"100467f8-8ee2-4cda-a9a7-f52b11af838c\") " Oct 30 22:28:47.544984 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.544871 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0018134-1144-4076-95ca-7383b46a2985-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0018134-1144-4076-95ca-7383b46a2985-istio-data\") pod \"e0018134-1144-4076-95ca-7383b46a2985\" (UID: \"e0018134-1144-4076-95ca-7383b46a2985\") " Oct 30 22:28:47.544984 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.544980 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0018134-1144-4076-95ca-7383b46a2985-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0018134-1144-4076-95ca-7383b46a2985-workload-certs\") pod \"e0018134-1144-4076-95ca-7383b46a2985\" (UID: \"e0018134-1144-4076-95ca-7383b46a2985\") " Oct 30 22:28:47.546004 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.545031 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/100467f8-8ee2-4cda-a9a7-f52b11af838c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/100467f8-8ee2-4cda-a9a7-f52b11af838c-workload-socket\") pod \"100467f8-8ee2-4cda-a9a7-f52b11af838c\" (UID: \"100467f8-8ee2-4cda-a9a7-f52b11af838c\") " Oct 30 22:28:47.546004 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.545083 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/100467f8-8ee2-4cda-a9a7-f52b11af838c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/100467f8-8ee2-4cda-a9a7-f52b11af838c-credential-socket\") pod \"100467f8-8ee2-4cda-a9a7-f52b11af838c\" (UID: \"100467f8-8ee2-4cda-a9a7-f52b11af838c\") " Oct 30 22:28:47.546004 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.545130 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0018134-1144-4076-95ca-7383b46a2985-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0018134-1144-4076-95ca-7383b46a2985-istio-podinfo\") pod \"e0018134-1144-4076-95ca-7383b46a2985\" (UID: \"e0018134-1144-4076-95ca-7383b46a2985\") " Oct 30 22:28:47.546004 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.545174 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/100467f8-8ee2-4cda-a9a7-f52b11af838c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/100467f8-8ee2-4cda-a9a7-f52b11af838c-istiod-ca-cert\") pod \"100467f8-8ee2-4cda-a9a7-f52b11af838c\" (UID: \"100467f8-8ee2-4cda-a9a7-f52b11af838c\") " Oct 30 22:28:47.546004 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.545208 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0018134-1144-4076-95ca-7383b46a2985-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0018134-1144-4076-95ca-7383b46a2985-istio-envoy\") pod \"e0018134-1144-4076-95ca-7383b46a2985\" (UID: \"e0018134-1144-4076-95ca-7383b46a2985\") " Oct 30 22:28:47.546004 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.545271 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/100467f8-8ee2-4cda-a9a7-f52b11af838c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/100467f8-8ee2-4cda-a9a7-f52b11af838c-workload-certs\") pod \"100467f8-8ee2-4cda-a9a7-f52b11af838c\" (UID: \"100467f8-8ee2-4cda-a9a7-f52b11af838c\") " Oct 30 22:28:47.546004 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.545350 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/100467f8-8ee2-4cda-a9a7-f52b11af838c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/100467f8-8ee2-4cda-a9a7-f52b11af838c-istio-podinfo\") pod \"100467f8-8ee2-4cda-a9a7-f52b11af838c\" (UID: \"100467f8-8ee2-4cda-a9a7-f52b11af838c\") " Oct 30 22:28:47.546004 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.545398 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0018134-1144-4076-95ca-7383b46a2985-kube-api-access-xfvv4\" (UniqueName: \"kubernetes.io/projected/e0018134-1144-4076-95ca-7383b46a2985-kube-api-access-xfvv4\") pod \"e0018134-1144-4076-95ca-7383b46a2985\" (UID: \"e0018134-1144-4076-95ca-7383b46a2985\") " Oct 30 22:28:47.546004 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.545476 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0018134-1144-4076-95ca-7383b46a2985-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0018134-1144-4076-95ca-7383b46a2985-istiod-ca-cert\") pod \"e0018134-1144-4076-95ca-7383b46a2985\" (UID: \"e0018134-1144-4076-95ca-7383b46a2985\") " Oct 30 22:28:47.546004 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.545575 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0018134-1144-4076-95ca-7383b46a2985-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0018134-1144-4076-95ca-7383b46a2985-credential-socket\") pod \"e0018134-1144-4076-95ca-7383b46a2985\" (UID: \"e0018134-1144-4076-95ca-7383b46a2985\") " Oct 30 22:28:47.546004 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.545620 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0018134-1144-4076-95ca-7383b46a2985-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0018134-1144-4076-95ca-7383b46a2985-workload-socket\") pod \"e0018134-1144-4076-95ca-7383b46a2985\" (UID: \"e0018134-1144-4076-95ca-7383b46a2985\") " Oct 30 22:28:47.546004 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.545670 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/100467f8-8ee2-4cda-a9a7-f52b11af838c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/100467f8-8ee2-4cda-a9a7-f52b11af838c-istio-envoy\") pod \"100467f8-8ee2-4cda-a9a7-f52b11af838c\" (UID: \"100467f8-8ee2-4cda-a9a7-f52b11af838c\") " Oct 30 22:28:47.546004 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.545709 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/100467f8-8ee2-4cda-a9a7-f52b11af838c-istio-token\" (UniqueName: \"kubernetes.io/projected/100467f8-8ee2-4cda-a9a7-f52b11af838c-istio-token\") pod \"100467f8-8ee2-4cda-a9a7-f52b11af838c\" (UID: \"100467f8-8ee2-4cda-a9a7-f52b11af838c\") " Oct 30 22:28:47.547095 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.546265 5918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f4e41b-50c7-441c-8ff8-6c815143b79b-istio-podinfo\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:47.547095 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.546287 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnbq6\" (UniqueName: \"kubernetes.io/projected/97f4e41b-50c7-441c-8ff8-6c815143b79b-kube-api-access-wnbq6\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:47.547095 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.546319 5918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97f4e41b-50c7-441c-8ff8-6c815143b79b-istio-token\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:47.547095 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.546334 5918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f4e41b-50c7-441c-8ff8-6c815143b79b-istiod-ca-cert\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:47.547095 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.546348 5918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f4e41b-50c7-441c-8ff8-6c815143b79b-istio-envoy\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:47.547095 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.546770 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0018134-1144-4076-95ca-7383b46a2985-istio-data" pod "e0018134-1144-4076-95ca-7383b46a2985" (UID: "e0018134-1144-4076-95ca-7383b46a2985"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:47.547095 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.546785 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/100467f8-8ee2-4cda-a9a7-f52b11af838c-istio-data" pod "100467f8-8ee2-4cda-a9a7-f52b11af838c" (UID: "100467f8-8ee2-4cda-a9a7-f52b11af838c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:47.547095 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.547006 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/100467f8-8ee2-4cda-a9a7-f52b11af838c-istiod-ca-cert" pod "100467f8-8ee2-4cda-a9a7-f52b11af838c" (UID: "100467f8-8ee2-4cda-a9a7-f52b11af838c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:47.547095 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.547024 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/100467f8-8ee2-4cda-a9a7-f52b11af838c-workload-socket" pod "100467f8-8ee2-4cda-a9a7-f52b11af838c" (UID: "100467f8-8ee2-4cda-a9a7-f52b11af838c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:47.547095 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.547075 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/100467f8-8ee2-4cda-a9a7-f52b11af838c-credential-socket" pod "100467f8-8ee2-4cda-a9a7-f52b11af838c" (UID: "100467f8-8ee2-4cda-a9a7-f52b11af838c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:47.547677 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.547269 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/100467f8-8ee2-4cda-a9a7-f52b11af838c-workload-certs" pod "100467f8-8ee2-4cda-a9a7-f52b11af838c" (UID: "100467f8-8ee2-4cda-a9a7-f52b11af838c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:47.547677 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.547421 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0018134-1144-4076-95ca-7383b46a2985-credential-socket" pod "e0018134-1144-4076-95ca-7383b46a2985" (UID: "e0018134-1144-4076-95ca-7383b46a2985"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:47.547677 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.547561 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0018134-1144-4076-95ca-7383b46a2985-workload-socket" pod "e0018134-1144-4076-95ca-7383b46a2985" (UID: "e0018134-1144-4076-95ca-7383b46a2985"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:47.547929 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.547901 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0018134-1144-4076-95ca-7383b46a2985-istiod-ca-cert" pod "e0018134-1144-4076-95ca-7383b46a2985" (UID: "e0018134-1144-4076-95ca-7383b46a2985"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:47.548447 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.548427 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0018134-1144-4076-95ca-7383b46a2985-workload-certs" pod "e0018134-1144-4076-95ca-7383b46a2985" (UID: "e0018134-1144-4076-95ca-7383b46a2985"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:47.550535 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.550511 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/100467f8-8ee2-4cda-a9a7-f52b11af838c-istio-envoy" pod "100467f8-8ee2-4cda-a9a7-f52b11af838c" (UID: "100467f8-8ee2-4cda-a9a7-f52b11af838c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:47.552566 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.552540 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0018134-1144-4076-95ca-7383b46a2985-istio-envoy" pod "e0018134-1144-4076-95ca-7383b46a2985" (UID: "e0018134-1144-4076-95ca-7383b46a2985"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:47.552692 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.552573 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/100467f8-8ee2-4cda-a9a7-f52b11af838c-istio-token" pod "100467f8-8ee2-4cda-a9a7-f52b11af838c" (UID: "100467f8-8ee2-4cda-a9a7-f52b11af838c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:47.552692 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.552665 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/100467f8-8ee2-4cda-a9a7-f52b11af838c-istio-podinfo" pod "100467f8-8ee2-4cda-a9a7-f52b11af838c" (UID: "100467f8-8ee2-4cda-a9a7-f52b11af838c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:28:47.552857 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.552702 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0018134-1144-4076-95ca-7383b46a2985-istio-podinfo" pod "e0018134-1144-4076-95ca-7383b46a2985" (UID: "e0018134-1144-4076-95ca-7383b46a2985"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:28:47.553269 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.553245 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0018134-1144-4076-95ca-7383b46a2985-istio-token" pod "e0018134-1144-4076-95ca-7383b46a2985" (UID: "e0018134-1144-4076-95ca-7383b46a2985"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:47.553382 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.553280 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/100467f8-8ee2-4cda-a9a7-f52b11af838c-kube-api-access-7fhpx" pod "100467f8-8ee2-4cda-a9a7-f52b11af838c" (UID: "100467f8-8ee2-4cda-a9a7-f52b11af838c"). InnerVolumeSpecName "kube-api-access-7fhpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:47.554130 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.554111 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0018134-1144-4076-95ca-7383b46a2985-kube-api-access-xfvv4" pod "e0018134-1144-4076-95ca-7383b46a2985" (UID: "e0018134-1144-4076-95ca-7383b46a2985"). InnerVolumeSpecName "kube-api-access-xfvv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:47.647061 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.646979 5918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0018134-1144-4076-95ca-7383b46a2985-workload-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:47.647061 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.647014 5918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/100467f8-8ee2-4cda-a9a7-f52b11af838c-istio-envoy\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:47.647061 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.647026 5918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/100467f8-8ee2-4cda-a9a7-f52b11af838c-istio-token\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:47.647061 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.647036 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fhpx\" (UniqueName: \"kubernetes.io/projected/100467f8-8ee2-4cda-a9a7-f52b11af838c-kube-api-access-7fhpx\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:47.647061 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.647049 5918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0018134-1144-4076-95ca-7383b46a2985-istio-token\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:47.647061 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.647058 5918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/100467f8-8ee2-4cda-a9a7-f52b11af838c-istio-data\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:47.647392 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.647072 5918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0018134-1144-4076-95ca-7383b46a2985-istio-data\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:47.647392 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.647080 5918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0018134-1144-4076-95ca-7383b46a2985-workload-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:47.647392 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.647088 5918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/100467f8-8ee2-4cda-a9a7-f52b11af838c-workload-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:47.647392 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.647096 5918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/100467f8-8ee2-4cda-a9a7-f52b11af838c-credential-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:47.647392 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.647105 5918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0018134-1144-4076-95ca-7383b46a2985-istio-podinfo\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:47.647392 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.647113 5918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/100467f8-8ee2-4cda-a9a7-f52b11af838c-istiod-ca-cert\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:47.647392 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.647121 5918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0018134-1144-4076-95ca-7383b46a2985-istio-envoy\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:47.647392 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.647129 5918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/100467f8-8ee2-4cda-a9a7-f52b11af838c-workload-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:47.647392 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.647139 5918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/100467f8-8ee2-4cda-a9a7-f52b11af838c-istio-podinfo\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:47.647392 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.647152 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfvv4\" (UniqueName: \"kubernetes.io/projected/e0018134-1144-4076-95ca-7383b46a2985-kube-api-access-xfvv4\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:47.647392 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.647160 5918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0018134-1144-4076-95ca-7383b46a2985-istiod-ca-cert\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:47.647392 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:47.647167 5918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0018134-1144-4076-95ca-7383b46a2985-credential-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:48.353728 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:48.353642 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" Oct 30 22:28:48.353728 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:48.353662 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm" event={"ID":"100467f8-8ee2-4cda-a9a7-f52b11af838c","Type":"ContainerDied","Data":"4e0d836d5b883eead59f326341cd62286a6c62b463b2243767e975e32f111a7e"} Oct 30 22:28:48.354275 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:48.353736 5918 scope.go:122] "RemoveContainer" containerID="b02729058eec38f80828a0236a705def98d2dfe3be1d5ab611632ce2d47815fd" Oct 30 22:28:48.355172 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:48.355151 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" event={"ID":"e0018134-1144-4076-95ca-7383b46a2985","Type":"ContainerDied","Data":"756670e5e3d7415fa457e08ed14f1bddcbfa2c759d4bc209965975660ad6ead3"} Oct 30 22:28:48.355295 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:48.355188 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf" Oct 30 22:28:48.355295 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:48.355193 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v" Oct 30 22:28:48.363825 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:48.363803 5918 scope.go:122] "RemoveContainer" containerID="3dcf52a5b85487a2bc84f4e33aaad4633963baa5afdef5b05528010a06eb2a24" Oct 30 22:28:48.574355 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:48.574324 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v"] Oct 30 22:28:48.625050 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:48.624977 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-7h99v"] Oct 30 22:28:48.696518 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:48.696488 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm"] Oct 30 22:28:48.705751 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:48.705723 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-n26pm"] Oct 30 22:28:48.819945 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:48.819909 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf"] Oct 30 22:28:48.828701 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:48.828675 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-422c1029-openshift-default-5786fb5789-5d2xf"] Oct 30 22:28:49.852325 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:49.852291 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="100467f8-8ee2-4cda-a9a7-f52b11af838c" path="/var/lib/kubelet/pods/100467f8-8ee2-4cda-a9a7-f52b11af838c/volumes" Oct 30 22:28:49.852795 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:49.852766 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97f4e41b-50c7-441c-8ff8-6c815143b79b" path="/var/lib/kubelet/pods/97f4e41b-50c7-441c-8ff8-6c815143b79b/volumes" Oct 30 22:28:49.853161 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:49.853148 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0018134-1144-4076-95ca-7383b46a2985" path="/var/lib/kubelet/pods/e0018134-1144-4076-95ca-7383b46a2985/volumes" Oct 30 22:28:51.274892 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:28:51.274857 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:28:56.167731 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:56.167335 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-xmr6p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:28:56.167731 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:28:56.167421 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xmr6p" podUID="1fe658cf-6504-47e1-8640-cc6404e8a882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:29:02.969435 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:02.969397 5918 patch_prober.go:29] interesting pod/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:29:02.969841 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:02.969471 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" podUID="e5345433-337f-4434-a218-a9880269ddc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:29:03.500369 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:03.500336 5918 generic.go:356] "Generic (PLEG): container finished" podID="e5345433-337f-4434-a218-a9880269ddc5" containerID="55569ea765efb899cd1a6c5657da7820a75548cd6069b499d0ce7f9047e11102" exitCode=0 Oct 30 22:29:03.500592 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:03.500408 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" event={"ID":"e5345433-337f-4434-a218-a9880269ddc5","Type":"ContainerDied","Data":"55569ea765efb899cd1a6c5657da7820a75548cd6069b499d0ce7f9047e11102"} Oct 30 22:29:03.500823 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:03.500806 5918 scope.go:122] "RemoveContainer" containerID="55569ea765efb899cd1a6c5657da7820a75548cd6069b499d0ce7f9047e11102" Oct 30 22:29:03.502169 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:03.502133 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:29:03.502275 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:03.502206 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:29:03.502275 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:03.502263 5918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:29:04.524961 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:04.524916 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" event={"ID":"e5345433-337f-4434-a218-a9880269ddc5","Type":"ContainerStarted","Data":"63f79333c56ff4649518117ae1e6cae4caf1680548007953679d46a552b3eed0"} Oct 30 22:29:04.823810 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:04.823719 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:29:04.824027 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:04.823998 5918 patch_prober.go:29] interesting pod/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Oct 30 22:29:04.824083 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:04.824064 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" podUID="e5345433-337f-4434-a218-a9880269ddc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Oct 30 22:29:05.824165 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:05.824133 5918 patch_prober.go:29] interesting pod/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Oct 30 22:29:05.824566 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:05.824187 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" podUID="e5345433-337f-4434-a218-a9880269ddc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Oct 30 22:29:06.823769 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:06.823734 5918 patch_prober.go:29] interesting pod/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Oct 30 22:29:06.823958 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:06.823789 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" podUID="e5345433-337f-4434-a218-a9880269ddc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Oct 30 22:29:07.823726 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:07.823686 5918 patch_prober.go:29] interesting pod/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Oct 30 22:29:07.824140 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:07.823761 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" podUID="e5345433-337f-4434-a218-a9880269ddc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Oct 30 22:29:08.823770 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:08.823721 5918 patch_prober.go:29] interesting pod/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Oct 30 22:29:08.824354 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:08.823810 5918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" podUID="e5345433-337f-4434-a218-a9880269ddc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Oct 30 22:29:09.827082 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:09.827053 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:29:09.827524 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:09.827490 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:29:09.828151 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:09.828134 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:29:21.275164 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:29:21.275127 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:29:35.119431 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:35.119391 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh"] Oct 30 22:29:35.119922 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:35.119637 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" podUID="a7dd97a6-507e-4441-b9d2-381bee4d6dd2" containerName="istio-proxy" containerID="cri-o://b0a42735700db7d8cec94372b4225d403b4ca362494598a87ecaec393ef504d4" gracePeriod=30 Oct 30 22:29:36.536568 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:36.536526 5918 patch_prober.go:29] interesting pod/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:29:36.537120 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:36.536607 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" podUID="a7dd97a6-507e-4441-b9d2-381bee4d6dd2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:29:40.386082 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:40.385980 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" Oct 30 22:29:40.423286 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:40.423252 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-istio-podinfo\") pod \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\" (UID: \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\") " Oct 30 22:29:40.423422 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:40.423309 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-credential-socket\") pod \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\" (UID: \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\") " Oct 30 22:29:40.423422 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:40.423342 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-istio-token\" (UniqueName: \"kubernetes.io/projected/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-istio-token\") pod \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\" (UID: \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\") " Oct 30 22:29:40.423422 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:40.423413 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-workload-socket\") pod \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\" (UID: \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\") " Oct 30 22:29:40.423587 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:40.423491 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-istio-data\") pod \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\" (UID: \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\") " Oct 30 22:29:40.423587 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:40.423537 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-workload-certs\") pod \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\" (UID: \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\") " Oct 30 22:29:40.423587 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:40.423557 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-credential-socket" pod "a7dd97a6-507e-4441-b9d2-381bee4d6dd2" (UID: "a7dd97a6-507e-4441-b9d2-381bee4d6dd2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:40.423736 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:40.423603 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-kube-api-access-75vgf\" (UniqueName: \"kubernetes.io/projected/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-kube-api-access-75vgf\") pod \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\" (UID: \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\") " Oct 30 22:29:40.423736 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:40.423679 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-istio-envoy\") pod \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\" (UID: \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\") " Oct 30 22:29:40.423736 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:40.423716 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-istiod-ca-cert\") pod \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\" (UID: \"a7dd97a6-507e-4441-b9d2-381bee4d6dd2\") " Oct 30 22:29:40.423885 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:40.423732 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-istio-data" pod "a7dd97a6-507e-4441-b9d2-381bee4d6dd2" (UID: "a7dd97a6-507e-4441-b9d2-381bee4d6dd2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:40.423885 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:40.423743 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-workload-socket" pod "a7dd97a6-507e-4441-b9d2-381bee4d6dd2" (UID: "a7dd97a6-507e-4441-b9d2-381bee4d6dd2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:40.423885 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:40.423804 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-workload-certs" pod "a7dd97a6-507e-4441-b9d2-381bee4d6dd2" (UID: "a7dd97a6-507e-4441-b9d2-381bee4d6dd2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:40.424105 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:40.424087 5918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-credential-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:40.424171 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:40.424111 5918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-workload-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:40.424171 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:40.424115 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-istiod-ca-cert" pod "a7dd97a6-507e-4441-b9d2-381bee4d6dd2" (UID: "a7dd97a6-507e-4441-b9d2-381bee4d6dd2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:29:40.424171 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:40.424125 5918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-istio-data\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:40.424171 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:40.424142 5918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-workload-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:40.425662 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:40.425633 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-istio-podinfo" pod "a7dd97a6-507e-4441-b9d2-381bee4d6dd2" (UID: "a7dd97a6-507e-4441-b9d2-381bee4d6dd2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:29:40.425944 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:40.425930 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-istio-envoy" pod "a7dd97a6-507e-4441-b9d2-381bee4d6dd2" (UID: "a7dd97a6-507e-4441-b9d2-381bee4d6dd2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:40.426163 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:40.426145 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-kube-api-access-75vgf" pod "a7dd97a6-507e-4441-b9d2-381bee4d6dd2" (UID: "a7dd97a6-507e-4441-b9d2-381bee4d6dd2"). InnerVolumeSpecName "kube-api-access-75vgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:40.426163 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:40.426152 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-istio-token" pod "a7dd97a6-507e-4441-b9d2-381bee4d6dd2" (UID: "a7dd97a6-507e-4441-b9d2-381bee4d6dd2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:40.524781 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:40.524745 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75vgf\" (UniqueName: \"kubernetes.io/projected/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-kube-api-access-75vgf\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:40.524781 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:40.524771 5918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-istio-envoy\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:40.524781 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:40.524780 5918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-istiod-ca-cert\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:40.524987 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:40.524789 5918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-istio-podinfo\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:40.524987 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:40.524800 5918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7dd97a6-507e-4441-b9d2-381bee4d6dd2-istio-token\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:40.736257 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:40.736200 5918 generic.go:356] "Generic (PLEG): container finished" podID="a7dd97a6-507e-4441-b9d2-381bee4d6dd2" containerID="b0a42735700db7d8cec94372b4225d403b4ca362494598a87ecaec393ef504d4" exitCode=0 Oct 30 22:29:40.736545 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:40.736525 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" event={"ID":"a7dd97a6-507e-4441-b9d2-381bee4d6dd2","Type":"ContainerDied","Data":"b0a42735700db7d8cec94372b4225d403b4ca362494598a87ecaec393ef504d4"} Oct 30 22:29:40.736743 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:40.736727 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" event={"ID":"a7dd97a6-507e-4441-b9d2-381bee4d6dd2","Type":"ContainerDied","Data":"26f4faa0b9d6592718615fb692565002d6612030ea99d2000289055f26e4c630"} Oct 30 22:29:40.736850 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:40.736839 5918 scope.go:122] "RemoveContainer" containerID="b0a42735700db7d8cec94372b4225d403b4ca362494598a87ecaec393ef504d4" Oct 30 22:29:40.737172 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:40.737154 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh" Oct 30 22:29:40.751871 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:40.751853 5918 scope.go:122] "RemoveContainer" containerID="b0a42735700db7d8cec94372b4225d403b4ca362494598a87ecaec393ef504d4" Oct 30 22:29:40.752200 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:29:40.752177 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0a42735700db7d8cec94372b4225d403b4ca362494598a87ecaec393ef504d4\": container with ID starting with b0a42735700db7d8cec94372b4225d403b4ca362494598a87ecaec393ef504d4 not found: ID does not exist" containerID="b0a42735700db7d8cec94372b4225d403b4ca362494598a87ecaec393ef504d4" Oct 30 22:29:40.752273 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:40.752210 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0a42735700db7d8cec94372b4225d403b4ca362494598a87ecaec393ef504d4"} err="failed to get container status \"b0a42735700db7d8cec94372b4225d403b4ca362494598a87ecaec393ef504d4\": rpc error: code = NotFound desc = could not find container \"b0a42735700db7d8cec94372b4225d403b4ca362494598a87ecaec393ef504d4\": container with ID starting with b0a42735700db7d8cec94372b4225d403b4ca362494598a87ecaec393ef504d4 not found: ID does not exist" Oct 30 22:29:40.911826 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:40.911788 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh"] Oct 30 22:29:40.913151 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:40.913123 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d8837a-openshift-default-66b7f49787-btdgh"] Oct 30 22:29:41.851525 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:41.851492 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7dd97a6-507e-4441-b9d2-381bee4d6dd2" path="/var/lib/kubelet/pods/a7dd97a6-507e-4441-b9d2-381bee4d6dd2/volumes" Oct 30 22:29:51.276254 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:29:51.276027 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:29:52.637839 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:52.637806 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp"] Oct 30 22:29:52.638305 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:52.638067 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp" podUID="544a91a1-bd59-4dd2-91a6-33a33bebccb9" containerName="maas-api" containerID="cri-o://9fa36a2a0e417b4f96bf5ea5bae9b4b1341b5c8e8489df028808aff6f658befa" gracePeriod=30 Oct 30 22:29:52.807605 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:52.807556 5918 generic.go:356] "Generic (PLEG): container finished" podID="544a91a1-bd59-4dd2-91a6-33a33bebccb9" containerID="9fa36a2a0e417b4f96bf5ea5bae9b4b1341b5c8e8489df028808aff6f658befa" exitCode=0 Oct 30 22:29:52.807779 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:52.807640 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp" event={"ID":"544a91a1-bd59-4dd2-91a6-33a33bebccb9","Type":"ContainerDied","Data":"9fa36a2a0e417b4f96bf5ea5bae9b4b1341b5c8e8489df028808aff6f658befa"} Oct 30 22:29:52.908922 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:52.908900 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp" Oct 30 22:29:53.061865 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:53.061825 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/544a91a1-bd59-4dd2-91a6-33a33bebccb9-kube-api-access-hnzqk\" (UniqueName: \"kubernetes.io/projected/544a91a1-bd59-4dd2-91a6-33a33bebccb9-kube-api-access-hnzqk\") pod \"544a91a1-bd59-4dd2-91a6-33a33bebccb9\" (UID: \"544a91a1-bd59-4dd2-91a6-33a33bebccb9\") " Oct 30 22:29:53.062103 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:53.061930 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/544a91a1-bd59-4dd2-91a6-33a33bebccb9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/544a91a1-bd59-4dd2-91a6-33a33bebccb9-openshift-service-ca\") pod \"544a91a1-bd59-4dd2-91a6-33a33bebccb9\" (UID: \"544a91a1-bd59-4dd2-91a6-33a33bebccb9\") " Oct 30 22:29:53.062103 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:53.061980 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/544a91a1-bd59-4dd2-91a6-33a33bebccb9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/544a91a1-bd59-4dd2-91a6-33a33bebccb9-maas-api-tls\") pod \"544a91a1-bd59-4dd2-91a6-33a33bebccb9\" (UID: \"544a91a1-bd59-4dd2-91a6-33a33bebccb9\") " Oct 30 22:29:53.062103 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:53.062033 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/544a91a1-bd59-4dd2-91a6-33a33bebccb9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/544a91a1-bd59-4dd2-91a6-33a33bebccb9-metrics-certs\") pod \"544a91a1-bd59-4dd2-91a6-33a33bebccb9\" (UID: \"544a91a1-bd59-4dd2-91a6-33a33bebccb9\") " Oct 30 22:29:53.062593 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:53.062544 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/544a91a1-bd59-4dd2-91a6-33a33bebccb9-openshift-service-ca" pod "544a91a1-bd59-4dd2-91a6-33a33bebccb9" (UID: "544a91a1-bd59-4dd2-91a6-33a33bebccb9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:29:53.065089 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:53.065059 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/544a91a1-bd59-4dd2-91a6-33a33bebccb9-kube-api-access-hnzqk" pod "544a91a1-bd59-4dd2-91a6-33a33bebccb9" (UID: "544a91a1-bd59-4dd2-91a6-33a33bebccb9"). InnerVolumeSpecName "kube-api-access-hnzqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:53.068614 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:53.068572 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/544a91a1-bd59-4dd2-91a6-33a33bebccb9-maas-api-tls" pod "544a91a1-bd59-4dd2-91a6-33a33bebccb9" (UID: "544a91a1-bd59-4dd2-91a6-33a33bebccb9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:29:53.070672 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:53.070646 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/544a91a1-bd59-4dd2-91a6-33a33bebccb9-metrics-certs" pod "544a91a1-bd59-4dd2-91a6-33a33bebccb9" (UID: "544a91a1-bd59-4dd2-91a6-33a33bebccb9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:29:53.164519 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:53.164434 5918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/544a91a1-bd59-4dd2-91a6-33a33bebccb9-maas-api-tls\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:53.164519 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:53.164474 5918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/544a91a1-bd59-4dd2-91a6-33a33bebccb9-metrics-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:53.164519 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:53.164488 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnzqk\" (UniqueName: \"kubernetes.io/projected/544a91a1-bd59-4dd2-91a6-33a33bebccb9-kube-api-access-hnzqk\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:53.164519 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:53.164497 5918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/544a91a1-bd59-4dd2-91a6-33a33bebccb9-openshift-service-ca\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:53.814957 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:53.814932 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp" Oct 30 22:29:53.815385 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:53.814970 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp" event={"ID":"544a91a1-bd59-4dd2-91a6-33a33bebccb9","Type":"ContainerDied","Data":"417bd8f3d51266f15fbbeafa06af5a513689b90fa118581f3ae7075a0d697e19"} Oct 30 22:29:53.815385 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:53.815034 5918 scope.go:122] "RemoveContainer" containerID="9fa36a2a0e417b4f96bf5ea5bae9b4b1341b5c8e8489df028808aff6f658befa" Oct 30 22:29:54.003727 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:54.003694 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp"] Oct 30 22:29:54.054691 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:54.054659 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d67bf9-7db974fbc-kpthp"] Oct 30 22:29:55.851495 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:55.851466 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="544a91a1-bd59-4dd2-91a6-33a33bebccb9" path="/var/lib/kubelet/pods/544a91a1-bd59-4dd2-91a6-33a33bebccb9/volumes" Oct 30 22:29:56.329499 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:29:56.329465 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7"] Oct 30 22:30:02.399943 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:02.399904 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg"] Oct 30 22:30:07.401026 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:07.400983 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg" podUID="bc656e82-591f-4997-9576-9297fd4dde35" containerName="payload-pre-processing" containerID="cri-o://998e6a31afecae3cd39b7f2613c46e68f16862f72489fe7b3298b7b9ef9babd1" gracePeriod=25 Oct 30 22:30:11.350378 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:11.350311 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" podUID="1a1ae368-0e8c-408d-bfcd-8f4a8194464f" containerName="main" containerID="cri-o://2cbd5b84590571170aa2cce358162c27b1d27732e139afd2f596b7ce8a387bc3" gracePeriod=45 Oct 30 22:30:11.624102 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:11.624074 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" Oct 30 22:30:11.680048 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:11.680013 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-tmp-dir\") pod \"1a1ae368-0e8c-408d-bfcd-8f4a8194464f\" (UID: \"1a1ae368-0e8c-408d-bfcd-8f4a8194464f\") " Oct 30 22:30:11.680260 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:11.680079 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-tls-certs\" (UniqueName: \"kubernetes.io/secret/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-tls-certs\") pod \"1a1ae368-0e8c-408d-bfcd-8f4a8194464f\" (UID: \"1a1ae368-0e8c-408d-bfcd-8f4a8194464f\") " Oct 30 22:30:11.680260 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:11.680099 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-model-cache\") pod \"1a1ae368-0e8c-408d-bfcd-8f4a8194464f\" (UID: \"1a1ae368-0e8c-408d-bfcd-8f4a8194464f\") " Oct 30 22:30:11.680260 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:11.680128 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-dshm\") pod \"1a1ae368-0e8c-408d-bfcd-8f4a8194464f\" (UID: \"1a1ae368-0e8c-408d-bfcd-8f4a8194464f\") " Oct 30 22:30:11.680260 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:11.680172 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-kube-api-access-x5q4f\" (UniqueName: \"kubernetes.io/projected/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-kube-api-access-x5q4f\") pod \"1a1ae368-0e8c-408d-bfcd-8f4a8194464f\" (UID: \"1a1ae368-0e8c-408d-bfcd-8f4a8194464f\") " Oct 30 22:30:11.680493 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:11.680276 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-home\" (UniqueName: \"kubernetes.io/empty-dir/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-home\") pod \"1a1ae368-0e8c-408d-bfcd-8f4a8194464f\" (UID: \"1a1ae368-0e8c-408d-bfcd-8f4a8194464f\") " Oct 30 22:30:11.680493 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:11.680337 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-tmp-dir" pod "1a1ae368-0e8c-408d-bfcd-8f4a8194464f" (UID: "1a1ae368-0e8c-408d-bfcd-8f4a8194464f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:11.680493 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:11.680412 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-model-cache" pod "1a1ae368-0e8c-408d-bfcd-8f4a8194464f" (UID: "1a1ae368-0e8c-408d-bfcd-8f4a8194464f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:11.680624 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:11.680559 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-home" pod "1a1ae368-0e8c-408d-bfcd-8f4a8194464f" (UID: "1a1ae368-0e8c-408d-bfcd-8f4a8194464f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:11.680791 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:11.680769 5918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-tmp-dir\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:11.680835 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:11.680798 5918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-model-cache\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:11.680835 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:11.680815 5918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-home\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:11.682640 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:11.682619 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-dshm" pod "1a1ae368-0e8c-408d-bfcd-8f4a8194464f" (UID: "1a1ae368-0e8c-408d-bfcd-8f4a8194464f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:11.682777 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:11.682761 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-kube-api-access-x5q4f" pod "1a1ae368-0e8c-408d-bfcd-8f4a8194464f" (UID: "1a1ae368-0e8c-408d-bfcd-8f4a8194464f"). InnerVolumeSpecName "kube-api-access-x5q4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:11.682840 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:11.682821 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-tls-certs" pod "1a1ae368-0e8c-408d-bfcd-8f4a8194464f" (UID: "1a1ae368-0e8c-408d-bfcd-8f4a8194464f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:30:11.781882 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:11.781848 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5q4f\" (UniqueName: \"kubernetes.io/projected/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-kube-api-access-x5q4f\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:11.781882 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:11.781881 5918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-tls-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:11.781882 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:11.781890 5918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a1ae368-0e8c-408d-bfcd-8f4a8194464f-dshm\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:11.920829 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:11.920798 5918 generic.go:356] "Generic (PLEG): container finished" podID="1a1ae368-0e8c-408d-bfcd-8f4a8194464f" containerID="2cbd5b84590571170aa2cce358162c27b1d27732e139afd2f596b7ce8a387bc3" exitCode=0 Oct 30 22:30:11.921010 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:11.920838 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" Oct 30 22:30:11.921010 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:11.920890 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" event={"ID":"1a1ae368-0e8c-408d-bfcd-8f4a8194464f","Type":"ContainerDied","Data":"2cbd5b84590571170aa2cce358162c27b1d27732e139afd2f596b7ce8a387bc3"} Oct 30 22:30:11.921010 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:11.920935 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7" event={"ID":"1a1ae368-0e8c-408d-bfcd-8f4a8194464f","Type":"ContainerDied","Data":"cf6888bafbe6361d1ff63975e91ebd8b9db291cd3f14f22b1f44d9c4aedfee5d"} Oct 30 22:30:11.921010 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:11.920957 5918 scope.go:122] "RemoveContainer" containerID="2cbd5b84590571170aa2cce358162c27b1d27732e139afd2f596b7ce8a387bc3" Oct 30 22:30:11.931078 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:11.931062 5918 scope.go:122] "RemoveContainer" containerID="2cbd5b84590571170aa2cce358162c27b1d27732e139afd2f596b7ce8a387bc3" Oct 30 22:30:11.931403 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:30:11.931382 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cbd5b84590571170aa2cce358162c27b1d27732e139afd2f596b7ce8a387bc3\": container with ID starting with 2cbd5b84590571170aa2cce358162c27b1d27732e139afd2f596b7ce8a387bc3 not found: ID does not exist" containerID="2cbd5b84590571170aa2cce358162c27b1d27732e139afd2f596b7ce8a387bc3" Oct 30 22:30:11.931481 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:11.931411 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cbd5b84590571170aa2cce358162c27b1d27732e139afd2f596b7ce8a387bc3"} err="failed to get container status \"2cbd5b84590571170aa2cce358162c27b1d27732e139afd2f596b7ce8a387bc3\": rpc error: code = NotFound desc = could not find container \"2cbd5b84590571170aa2cce358162c27b1d27732e139afd2f596b7ce8a387bc3\": container with ID starting with 2cbd5b84590571170aa2cce358162c27b1d27732e139afd2f596b7ce8a387bc3 not found: ID does not exist" Oct 30 22:30:12.090071 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:12.089995 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7"] Oct 30 22:30:12.099425 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:12.099395 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-3783c2/premium-w5-3783c2-kserve-c6b4ffc5f-x6nd7"] Oct 30 22:30:13.851269 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:13.851237 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a1ae368-0e8c-408d-bfcd-8f4a8194464f" path="/var/lib/kubelet/pods/1a1ae368-0e8c-408d-bfcd-8f4a8194464f/volumes" Oct 30 22:30:14.671350 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:14.671313 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg" podUID="bc656e82-591f-4997-9576-9297fd4dde35" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:30:15.431900 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:15.431866 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d59c58-5b46f96fb9-9pd5g"] Oct 30 22:30:15.547033 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:15.547002 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d59c58-7678d58759-jsctb"] Oct 30 22:30:20.432384 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:20.432328 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-d59c58-5b46f96fb9-9pd5g" podUID="319acc25-060d-4921-bcaa-a62f41412146" containerName="payload-processing" containerID="cri-o://3671d6cf8a96977d07a220bc505b0212a684820637995cac5ec703f4238a4e3d" gracePeriod=25 Oct 30 22:30:20.548107 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:20.548069 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d59c58-7678d58759-jsctb" podUID="9b4c802f-e487-4d43-a8aa-ef87ddc27fcf" containerName="payload-pre-processing" containerID="cri-o://a1ce79fb24761b7e704af0651abdf99501a5f93c5f81dfde8e68a3e75f327228" gracePeriod=25 Oct 30 22:30:20.696619 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:20.695635 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d59c58-5b46f96fb9-9pd5g" Oct 30 22:30:20.777012 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:20.776981 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/319acc25-060d-4921-bcaa-a62f41412146-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/319acc25-060d-4921-bcaa-a62f41412146-plugins-config-volume\") pod \"319acc25-060d-4921-bcaa-a62f41412146\" (UID: \"319acc25-060d-4921-bcaa-a62f41412146\") " Oct 30 22:30:20.777194 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:20.777048 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/319acc25-060d-4921-bcaa-a62f41412146-kube-api-access-c9c7r\" (UniqueName: \"kubernetes.io/projected/319acc25-060d-4921-bcaa-a62f41412146-kube-api-access-c9c7r\") pod \"319acc25-060d-4921-bcaa-a62f41412146\" (UID: \"319acc25-060d-4921-bcaa-a62f41412146\") " Oct 30 22:30:20.777491 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:20.777459 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/319acc25-060d-4921-bcaa-a62f41412146-plugins-config-volume" pod "319acc25-060d-4921-bcaa-a62f41412146" (UID: "319acc25-060d-4921-bcaa-a62f41412146"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:20.779390 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:20.779357 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/319acc25-060d-4921-bcaa-a62f41412146-kube-api-access-c9c7r" pod "319acc25-060d-4921-bcaa-a62f41412146" (UID: "319acc25-060d-4921-bcaa-a62f41412146"). InnerVolumeSpecName "kube-api-access-c9c7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:20.797993 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:20.797972 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d59c58-7678d58759-jsctb" Oct 30 22:30:20.878323 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:20.878292 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b4c802f-e487-4d43-a8aa-ef87ddc27fcf-kube-api-access-rvf6t\" (UniqueName: \"kubernetes.io/projected/9b4c802f-e487-4d43-a8aa-ef87ddc27fcf-kube-api-access-rvf6t\") pod \"9b4c802f-e487-4d43-a8aa-ef87ddc27fcf\" (UID: \"9b4c802f-e487-4d43-a8aa-ef87ddc27fcf\") " Oct 30 22:30:20.878500 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:20.878372 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b4c802f-e487-4d43-a8aa-ef87ddc27fcf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b4c802f-e487-4d43-a8aa-ef87ddc27fcf-plugins-config-volume\") pod \"9b4c802f-e487-4d43-a8aa-ef87ddc27fcf\" (UID: \"9b4c802f-e487-4d43-a8aa-ef87ddc27fcf\") " Oct 30 22:30:20.878596 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:20.878578 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9c7r\" (UniqueName: \"kubernetes.io/projected/319acc25-060d-4921-bcaa-a62f41412146-kube-api-access-c9c7r\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:20.878634 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:20.878597 5918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/319acc25-060d-4921-bcaa-a62f41412146-plugins-config-volume\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:20.878746 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:20.878726 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b4c802f-e487-4d43-a8aa-ef87ddc27fcf-plugins-config-volume" pod "9b4c802f-e487-4d43-a8aa-ef87ddc27fcf" (UID: "9b4c802f-e487-4d43-a8aa-ef87ddc27fcf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:20.880603 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:20.880583 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b4c802f-e487-4d43-a8aa-ef87ddc27fcf-kube-api-access-rvf6t" pod "9b4c802f-e487-4d43-a8aa-ef87ddc27fcf" (UID: "9b4c802f-e487-4d43-a8aa-ef87ddc27fcf"). InnerVolumeSpecName "kube-api-access-rvf6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:20.980277 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:20.980182 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvf6t\" (UniqueName: \"kubernetes.io/projected/9b4c802f-e487-4d43-a8aa-ef87ddc27fcf-kube-api-access-rvf6t\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:20.980277 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:20.980225 5918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b4c802f-e487-4d43-a8aa-ef87ddc27fcf-plugins-config-volume\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:20.995878 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:20.995848 5918 generic.go:356] "Generic (PLEG): container finished" podID="9b4c802f-e487-4d43-a8aa-ef87ddc27fcf" containerID="a1ce79fb24761b7e704af0651abdf99501a5f93c5f81dfde8e68a3e75f327228" exitCode=0 Oct 30 22:30:20.996017 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:20.995892 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d59c58-7678d58759-jsctb" Oct 30 22:30:20.996017 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:20.995935 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d59c58-7678d58759-jsctb" event={"ID":"9b4c802f-e487-4d43-a8aa-ef87ddc27fcf","Type":"ContainerDied","Data":"a1ce79fb24761b7e704af0651abdf99501a5f93c5f81dfde8e68a3e75f327228"} Oct 30 22:30:20.996017 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:20.995987 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d59c58-7678d58759-jsctb" event={"ID":"9b4c802f-e487-4d43-a8aa-ef87ddc27fcf","Type":"ContainerDied","Data":"791a697682c22bb58e8e7884a4320cb1e77d80247f19da427b8e76bae90b7cdd"} Oct 30 22:30:20.996170 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:20.996016 5918 scope.go:122] "RemoveContainer" containerID="a1ce79fb24761b7e704af0651abdf99501a5f93c5f81dfde8e68a3e75f327228" Oct 30 22:30:20.997367 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:20.997343 5918 generic.go:356] "Generic (PLEG): container finished" podID="319acc25-060d-4921-bcaa-a62f41412146" containerID="3671d6cf8a96977d07a220bc505b0212a684820637995cac5ec703f4238a4e3d" exitCode=0 Oct 30 22:30:20.997445 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:20.997387 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d59c58-5b46f96fb9-9pd5g" Oct 30 22:30:20.997445 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:20.997395 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d59c58-5b46f96fb9-9pd5g" event={"ID":"319acc25-060d-4921-bcaa-a62f41412146","Type":"ContainerDied","Data":"3671d6cf8a96977d07a220bc505b0212a684820637995cac5ec703f4238a4e3d"} Oct 30 22:30:20.997445 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:20.997435 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d59c58-5b46f96fb9-9pd5g" event={"ID":"319acc25-060d-4921-bcaa-a62f41412146","Type":"ContainerDied","Data":"df4c3ee2b899c3afaac16762daa580f79cefdb5328c0a277dd8e38c0e2232c1d"} Oct 30 22:30:21.006723 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:21.006706 5918 scope.go:122] "RemoveContainer" containerID="a1ce79fb24761b7e704af0651abdf99501a5f93c5f81dfde8e68a3e75f327228" Oct 30 22:30:21.007051 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:30:21.007021 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1ce79fb24761b7e704af0651abdf99501a5f93c5f81dfde8e68a3e75f327228\": container with ID starting with a1ce79fb24761b7e704af0651abdf99501a5f93c5f81dfde8e68a3e75f327228 not found: ID does not exist" containerID="a1ce79fb24761b7e704af0651abdf99501a5f93c5f81dfde8e68a3e75f327228" Oct 30 22:30:21.007174 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:21.007061 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1ce79fb24761b7e704af0651abdf99501a5f93c5f81dfde8e68a3e75f327228"} err="failed to get container status \"a1ce79fb24761b7e704af0651abdf99501a5f93c5f81dfde8e68a3e75f327228\": rpc error: code = NotFound desc = could not find container \"a1ce79fb24761b7e704af0651abdf99501a5f93c5f81dfde8e68a3e75f327228\": container with ID starting with a1ce79fb24761b7e704af0651abdf99501a5f93c5f81dfde8e68a3e75f327228 not found: ID does not exist" Oct 30 22:30:21.007174 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:21.007084 5918 scope.go:122] "RemoveContainer" containerID="3671d6cf8a96977d07a220bc505b0212a684820637995cac5ec703f4238a4e3d" Oct 30 22:30:21.016039 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:21.016015 5918 scope.go:122] "RemoveContainer" containerID="3671d6cf8a96977d07a220bc505b0212a684820637995cac5ec703f4238a4e3d" Oct 30 22:30:21.016348 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:30:21.016329 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3671d6cf8a96977d07a220bc505b0212a684820637995cac5ec703f4238a4e3d\": container with ID starting with 3671d6cf8a96977d07a220bc505b0212a684820637995cac5ec703f4238a4e3d not found: ID does not exist" containerID="3671d6cf8a96977d07a220bc505b0212a684820637995cac5ec703f4238a4e3d" Oct 30 22:30:21.016405 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:21.016355 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3671d6cf8a96977d07a220bc505b0212a684820637995cac5ec703f4238a4e3d"} err="failed to get container status \"3671d6cf8a96977d07a220bc505b0212a684820637995cac5ec703f4238a4e3d\": rpc error: code = NotFound desc = could not find container \"3671d6cf8a96977d07a220bc505b0212a684820637995cac5ec703f4238a4e3d\": container with ID starting with 3671d6cf8a96977d07a220bc505b0212a684820637995cac5ec703f4238a4e3d not found: ID does not exist" Oct 30 22:30:21.188921 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:21.188888 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d59c58-7678d58759-jsctb"] Oct 30 22:30:21.193472 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:21.193447 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d59c58-7678d58759-jsctb"] Oct 30 22:30:21.274699 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:30:21.274615 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:30:21.310046 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:21.310014 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d59c58-5b46f96fb9-9pd5g"] Oct 30 22:30:21.316020 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:21.315995 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d59c58-5b46f96fb9-9pd5g"] Oct 30 22:30:21.850095 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:21.850063 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="319acc25-060d-4921-bcaa-a62f41412146" path="/var/lib/kubelet/pods/319acc25-060d-4921-bcaa-a62f41412146/volumes" Oct 30 22:30:21.850507 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:21.850489 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b4c802f-e487-4d43-a8aa-ef87ddc27fcf" path="/var/lib/kubelet/pods/9b4c802f-e487-4d43-a8aa-ef87ddc27fcf/volumes" Oct 30 22:30:24.671343 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:24.671308 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg" podUID="bc656e82-591f-4997-9576-9297fd4dde35" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:30:27.651894 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:27.651865 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg" Oct 30 22:30:27.758402 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:27.758368 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc656e82-591f-4997-9576-9297fd4dde35-kube-api-access-prgdk\" (UniqueName: \"kubernetes.io/projected/bc656e82-591f-4997-9576-9297fd4dde35-kube-api-access-prgdk\") pod \"bc656e82-591f-4997-9576-9297fd4dde35\" (UID: \"bc656e82-591f-4997-9576-9297fd4dde35\") " Oct 30 22:30:27.758575 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:27.758491 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc656e82-591f-4997-9576-9297fd4dde35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc656e82-591f-4997-9576-9297fd4dde35-plugins-config-volume\") pod \"bc656e82-591f-4997-9576-9297fd4dde35\" (UID: \"bc656e82-591f-4997-9576-9297fd4dde35\") " Oct 30 22:30:27.758950 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:27.758923 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc656e82-591f-4997-9576-9297fd4dde35-plugins-config-volume" pod "bc656e82-591f-4997-9576-9297fd4dde35" (UID: "bc656e82-591f-4997-9576-9297fd4dde35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:27.760795 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:27.760764 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc656e82-591f-4997-9576-9297fd4dde35-kube-api-access-prgdk" pod "bc656e82-591f-4997-9576-9297fd4dde35" (UID: "bc656e82-591f-4997-9576-9297fd4dde35"). InnerVolumeSpecName "kube-api-access-prgdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:27.859978 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:27.859951 5918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc656e82-591f-4997-9576-9297fd4dde35-plugins-config-volume\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:27.860108 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:27.859992 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prgdk\" (UniqueName: \"kubernetes.io/projected/bc656e82-591f-4997-9576-9297fd4dde35-kube-api-access-prgdk\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:28.038911 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:28.038825 5918 generic.go:356] "Generic (PLEG): container finished" podID="bc656e82-591f-4997-9576-9297fd4dde35" containerID="998e6a31afecae3cd39b7f2613c46e68f16862f72489fe7b3298b7b9ef9babd1" exitCode=0 Oct 30 22:30:28.038911 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:28.038873 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg" Oct 30 22:30:28.039088 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:28.038913 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg" event={"ID":"bc656e82-591f-4997-9576-9297fd4dde35","Type":"ContainerDied","Data":"998e6a31afecae3cd39b7f2613c46e68f16862f72489fe7b3298b7b9ef9babd1"} Oct 30 22:30:28.039088 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:28.038975 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg" event={"ID":"bc656e82-591f-4997-9576-9297fd4dde35","Type":"ContainerDied","Data":"2c98168cc5a42d3c5bae63825c30efa6e7a8d9f016539b04884cee00e38267b5"} Oct 30 22:30:28.039088 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:28.038997 5918 scope.go:122] "RemoveContainer" containerID="998e6a31afecae3cd39b7f2613c46e68f16862f72489fe7b3298b7b9ef9babd1" Oct 30 22:30:28.048332 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:28.048312 5918 scope.go:122] "RemoveContainer" containerID="998e6a31afecae3cd39b7f2613c46e68f16862f72489fe7b3298b7b9ef9babd1" Oct 30 22:30:28.048604 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:30:28.048586 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"998e6a31afecae3cd39b7f2613c46e68f16862f72489fe7b3298b7b9ef9babd1\": container with ID starting with 998e6a31afecae3cd39b7f2613c46e68f16862f72489fe7b3298b7b9ef9babd1 not found: ID does not exist" containerID="998e6a31afecae3cd39b7f2613c46e68f16862f72489fe7b3298b7b9ef9babd1" Oct 30 22:30:28.048642 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:28.048612 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"998e6a31afecae3cd39b7f2613c46e68f16862f72489fe7b3298b7b9ef9babd1"} err="failed to get container status \"998e6a31afecae3cd39b7f2613c46e68f16862f72489fe7b3298b7b9ef9babd1\": rpc error: code = NotFound desc = could not find container \"998e6a31afecae3cd39b7f2613c46e68f16862f72489fe7b3298b7b9ef9babd1\": container with ID starting with 998e6a31afecae3cd39b7f2613c46e68f16862f72489fe7b3298b7b9ef9babd1 not found: ID does not exist" Oct 30 22:30:28.220664 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:28.220626 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg"] Oct 30 22:30:28.271037 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:28.271006 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-3783c2-55645569bc-vvfgg"] Oct 30 22:30:29.850572 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:29.850537 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc656e82-591f-4997-9576-9297fd4dde35" path="/var/lib/kubelet/pods/bc656e82-591f-4997-9576-9297fd4dde35/volumes" Oct 30 22:30:33.235529 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:33.235495 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q"] Oct 30 22:30:33.235942 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:33.235726 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q" podUID="de7b01aa-6f28-4f10-b7ce-b615537fb99f" containerName="maas-api" containerID="cri-o://dca405e4d6eaa159fcd10cebae2823d3a64d045b2012aff63a79a5c38b85f83e" gracePeriod=30 Oct 30 22:30:33.509936 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:33.509908 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q" Oct 30 22:30:33.611335 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:33.611297 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de7b01aa-6f28-4f10-b7ce-b615537fb99f-kube-api-access-zjrxj\" (UniqueName: \"kubernetes.io/projected/de7b01aa-6f28-4f10-b7ce-b615537fb99f-kube-api-access-zjrxj\") pod \"de7b01aa-6f28-4f10-b7ce-b615537fb99f\" (UID: \"de7b01aa-6f28-4f10-b7ce-b615537fb99f\") " Oct 30 22:30:33.611505 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:33.611359 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de7b01aa-6f28-4f10-b7ce-b615537fb99f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de7b01aa-6f28-4f10-b7ce-b615537fb99f-openshift-service-ca\") pod \"de7b01aa-6f28-4f10-b7ce-b615537fb99f\" (UID: \"de7b01aa-6f28-4f10-b7ce-b615537fb99f\") " Oct 30 22:30:33.611505 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:33.611393 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de7b01aa-6f28-4f10-b7ce-b615537fb99f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/de7b01aa-6f28-4f10-b7ce-b615537fb99f-metrics-certs\") pod \"de7b01aa-6f28-4f10-b7ce-b615537fb99f\" (UID: \"de7b01aa-6f28-4f10-b7ce-b615537fb99f\") " Oct 30 22:30:33.611505 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:33.611435 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de7b01aa-6f28-4f10-b7ce-b615537fb99f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de7b01aa-6f28-4f10-b7ce-b615537fb99f-maas-api-tls\") pod \"de7b01aa-6f28-4f10-b7ce-b615537fb99f\" (UID: \"de7b01aa-6f28-4f10-b7ce-b615537fb99f\") " Oct 30 22:30:33.611828 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:33.611800 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de7b01aa-6f28-4f10-b7ce-b615537fb99f-openshift-service-ca" pod "de7b01aa-6f28-4f10-b7ce-b615537fb99f" (UID: "de7b01aa-6f28-4f10-b7ce-b615537fb99f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:33.612014 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:33.611984 5918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de7b01aa-6f28-4f10-b7ce-b615537fb99f-openshift-service-ca\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:33.613762 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:33.613737 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de7b01aa-6f28-4f10-b7ce-b615537fb99f-maas-api-tls" pod "de7b01aa-6f28-4f10-b7ce-b615537fb99f" (UID: "de7b01aa-6f28-4f10-b7ce-b615537fb99f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:30:33.613921 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:33.613871 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de7b01aa-6f28-4f10-b7ce-b615537fb99f-kube-api-access-zjrxj" pod "de7b01aa-6f28-4f10-b7ce-b615537fb99f" (UID: "de7b01aa-6f28-4f10-b7ce-b615537fb99f"). InnerVolumeSpecName "kube-api-access-zjrxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:33.614030 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:33.613772 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de7b01aa-6f28-4f10-b7ce-b615537fb99f-metrics-certs" pod "de7b01aa-6f28-4f10-b7ce-b615537fb99f" (UID: "de7b01aa-6f28-4f10-b7ce-b615537fb99f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:30:33.713428 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:33.713396 5918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de7b01aa-6f28-4f10-b7ce-b615537fb99f-metrics-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:33.713428 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:33.713422 5918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de7b01aa-6f28-4f10-b7ce-b615537fb99f-maas-api-tls\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:33.713428 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:33.713434 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjrxj\" (UniqueName: \"kubernetes.io/projected/de7b01aa-6f28-4f10-b7ce-b615537fb99f-kube-api-access-zjrxj\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:34.073501 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:34.073464 5918 generic.go:356] "Generic (PLEG): container finished" podID="de7b01aa-6f28-4f10-b7ce-b615537fb99f" containerID="dca405e4d6eaa159fcd10cebae2823d3a64d045b2012aff63a79a5c38b85f83e" exitCode=0 Oct 30 22:30:34.073681 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:34.073520 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q" Oct 30 22:30:34.073681 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:34.073526 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q" event={"ID":"de7b01aa-6f28-4f10-b7ce-b615537fb99f","Type":"ContainerDied","Data":"dca405e4d6eaa159fcd10cebae2823d3a64d045b2012aff63a79a5c38b85f83e"} Oct 30 22:30:34.073681 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:34.073576 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q" event={"ID":"de7b01aa-6f28-4f10-b7ce-b615537fb99f","Type":"ContainerDied","Data":"04a1abd867370cac0f885ee4a296d9511a70f8e8c375fd6d4e1f6113be9d5d95"} Oct 30 22:30:34.073681 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:34.073608 5918 scope.go:122] "RemoveContainer" containerID="dca405e4d6eaa159fcd10cebae2823d3a64d045b2012aff63a79a5c38b85f83e" Oct 30 22:30:34.083164 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:34.083143 5918 scope.go:122] "RemoveContainer" containerID="dca405e4d6eaa159fcd10cebae2823d3a64d045b2012aff63a79a5c38b85f83e" Oct 30 22:30:34.083480 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:30:34.083459 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dca405e4d6eaa159fcd10cebae2823d3a64d045b2012aff63a79a5c38b85f83e\": container with ID starting with dca405e4d6eaa159fcd10cebae2823d3a64d045b2012aff63a79a5c38b85f83e not found: ID does not exist" containerID="dca405e4d6eaa159fcd10cebae2823d3a64d045b2012aff63a79a5c38b85f83e" Oct 30 22:30:34.083557 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:34.083489 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dca405e4d6eaa159fcd10cebae2823d3a64d045b2012aff63a79a5c38b85f83e"} err="failed to get container status \"dca405e4d6eaa159fcd10cebae2823d3a64d045b2012aff63a79a5c38b85f83e\": rpc error: code = NotFound desc = could not find container \"dca405e4d6eaa159fcd10cebae2823d3a64d045b2012aff63a79a5c38b85f83e\": container with ID starting with dca405e4d6eaa159fcd10cebae2823d3a64d045b2012aff63a79a5c38b85f83e not found: ID does not exist" Oct 30 22:30:34.254477 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:34.254445 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q"] Oct 30 22:30:34.300419 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:34.300380 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d59c58-6bd895f8b8-2vl6q"] Oct 30 22:30:34.466452 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:34.466408 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q"] Oct 30 22:30:34.466684 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:34.466656 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q" podUID="035da59c-1f1a-4384-9c42-43bc5d184bea" containerName="maas-api" containerID="cri-o://4c1d5282cee64cf459ee1bca580fe2f1890e55d5edef498bb743ddeef686382b" gracePeriod=30 Oct 30 22:30:34.734905 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:34.734877 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q" Oct 30 22:30:34.827729 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:34.827701 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/035da59c-1f1a-4384-9c42-43bc5d184bea-kube-api-access-ms22t\" (UniqueName: \"kubernetes.io/projected/035da59c-1f1a-4384-9c42-43bc5d184bea-kube-api-access-ms22t\") pod \"035da59c-1f1a-4384-9c42-43bc5d184bea\" (UID: \"035da59c-1f1a-4384-9c42-43bc5d184bea\") " Oct 30 22:30:34.827913 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:34.827759 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/035da59c-1f1a-4384-9c42-43bc5d184bea-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/035da59c-1f1a-4384-9c42-43bc5d184bea-maas-api-tls\") pod \"035da59c-1f1a-4384-9c42-43bc5d184bea\" (UID: \"035da59c-1f1a-4384-9c42-43bc5d184bea\") " Oct 30 22:30:34.827913 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:34.827785 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/035da59c-1f1a-4384-9c42-43bc5d184bea-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/035da59c-1f1a-4384-9c42-43bc5d184bea-openshift-service-ca\") pod \"035da59c-1f1a-4384-9c42-43bc5d184bea\" (UID: \"035da59c-1f1a-4384-9c42-43bc5d184bea\") " Oct 30 22:30:34.827913 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:34.827887 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/035da59c-1f1a-4384-9c42-43bc5d184bea-metrics-certs\" (UniqueName: \"kubernetes.io/secret/035da59c-1f1a-4384-9c42-43bc5d184bea-metrics-certs\") pod \"035da59c-1f1a-4384-9c42-43bc5d184bea\" (UID: \"035da59c-1f1a-4384-9c42-43bc5d184bea\") " Oct 30 22:30:34.828136 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:34.828114 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/035da59c-1f1a-4384-9c42-43bc5d184bea-openshift-service-ca" pod "035da59c-1f1a-4384-9c42-43bc5d184bea" (UID: "035da59c-1f1a-4384-9c42-43bc5d184bea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:34.828368 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:34.828352 5918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/035da59c-1f1a-4384-9c42-43bc5d184bea-openshift-service-ca\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:34.829992 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:34.829939 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/035da59c-1f1a-4384-9c42-43bc5d184bea-kube-api-access-ms22t" pod "035da59c-1f1a-4384-9c42-43bc5d184bea" (UID: "035da59c-1f1a-4384-9c42-43bc5d184bea"). InnerVolumeSpecName "kube-api-access-ms22t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:34.830543 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:34.830521 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/035da59c-1f1a-4384-9c42-43bc5d184bea-metrics-certs" pod "035da59c-1f1a-4384-9c42-43bc5d184bea" (UID: "035da59c-1f1a-4384-9c42-43bc5d184bea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:30:34.830647 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:34.830543 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/035da59c-1f1a-4384-9c42-43bc5d184bea-maas-api-tls" pod "035da59c-1f1a-4384-9c42-43bc5d184bea" (UID: "035da59c-1f1a-4384-9c42-43bc5d184bea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:30:34.929778 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:34.929741 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms22t\" (UniqueName: \"kubernetes.io/projected/035da59c-1f1a-4384-9c42-43bc5d184bea-kube-api-access-ms22t\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:34.929778 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:34.929773 5918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/035da59c-1f1a-4384-9c42-43bc5d184bea-maas-api-tls\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:34.929778 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:34.929783 5918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/035da59c-1f1a-4384-9c42-43bc5d184bea-metrics-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:35.081323 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:35.081206 5918 generic.go:356] "Generic (PLEG): container finished" podID="035da59c-1f1a-4384-9c42-43bc5d184bea" containerID="4c1d5282cee64cf459ee1bca580fe2f1890e55d5edef498bb743ddeef686382b" exitCode=0 Oct 30 22:30:35.081323 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:35.081265 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q" event={"ID":"035da59c-1f1a-4384-9c42-43bc5d184bea","Type":"ContainerDied","Data":"4c1d5282cee64cf459ee1bca580fe2f1890e55d5edef498bb743ddeef686382b"} Oct 30 22:30:35.081323 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:35.081319 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q" event={"ID":"035da59c-1f1a-4384-9c42-43bc5d184bea","Type":"ContainerDied","Data":"6bfb9a9c2d463d191b0e5cf2f3b8d206849dfe3a040f9a02eff7eddae25a7e99"} Oct 30 22:30:35.081587 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:35.081341 5918 scope.go:122] "RemoveContainer" containerID="4c1d5282cee64cf459ee1bca580fe2f1890e55d5edef498bb743ddeef686382b" Oct 30 22:30:35.081587 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:35.081281 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q" Oct 30 22:30:35.091904 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:35.091886 5918 scope.go:122] "RemoveContainer" containerID="4c1d5282cee64cf459ee1bca580fe2f1890e55d5edef498bb743ddeef686382b" Oct 30 22:30:35.092167 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:30:35.092151 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c1d5282cee64cf459ee1bca580fe2f1890e55d5edef498bb743ddeef686382b\": container with ID starting with 4c1d5282cee64cf459ee1bca580fe2f1890e55d5edef498bb743ddeef686382b not found: ID does not exist" containerID="4c1d5282cee64cf459ee1bca580fe2f1890e55d5edef498bb743ddeef686382b" Oct 30 22:30:35.092226 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:35.092177 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c1d5282cee64cf459ee1bca580fe2f1890e55d5edef498bb743ddeef686382b"} err="failed to get container status \"4c1d5282cee64cf459ee1bca580fe2f1890e55d5edef498bb743ddeef686382b\": rpc error: code = NotFound desc = could not find container \"4c1d5282cee64cf459ee1bca580fe2f1890e55d5edef498bb743ddeef686382b\": container with ID starting with 4c1d5282cee64cf459ee1bca580fe2f1890e55d5edef498bb743ddeef686382b not found: ID does not exist" Oct 30 22:30:35.255501 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:35.255469 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q"] Oct 30 22:30:35.259225 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:35.259187 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-3783c2-bdf6bd6b-6xm9q"] Oct 30 22:30:35.851395 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:35.851358 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="035da59c-1f1a-4384-9c42-43bc5d184bea" path="/var/lib/kubelet/pods/035da59c-1f1a-4384-9c42-43bc5d184bea/volumes" Oct 30 22:30:35.852080 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:35.852059 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de7b01aa-6f28-4f10-b7ce-b615537fb99f" path="/var/lib/kubelet/pods/de7b01aa-6f28-4f10-b7ce-b615537fb99f/volumes" Oct 30 22:30:37.865760 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:37.865725 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728"] Oct 30 22:30:37.866256 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:37.865907 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" podUID="f9b3bd36-51f3-4b46-bed6-94703557bfdf" containerName="istio-proxy" containerID="cri-o://cb8aa5375f6f2c8f37b4bb60f755723baefbb297d9d5c7d55537a276b57ae445" gracePeriod=30 Oct 30 22:30:42.652674 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:42.652634 5918 patch_prober.go:29] interesting pod/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:30:42.653054 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:42.652686 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" podUID="f9b3bd36-51f3-4b46-bed6-94703557bfdf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:30:43.116893 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:43.116870 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" Oct 30 22:30:43.126484 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:43.126457 5918 generic.go:356] "Generic (PLEG): container finished" podID="f9b3bd36-51f3-4b46-bed6-94703557bfdf" containerID="cb8aa5375f6f2c8f37b4bb60f755723baefbb297d9d5c7d55537a276b57ae445" exitCode=0 Oct 30 22:30:43.126622 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:43.126514 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" Oct 30 22:30:43.126622 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:43.126539 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" event={"ID":"f9b3bd36-51f3-4b46-bed6-94703557bfdf","Type":"ContainerDied","Data":"cb8aa5375f6f2c8f37b4bb60f755723baefbb297d9d5c7d55537a276b57ae445"} Oct 30 22:30:43.126622 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:43.126598 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728" event={"ID":"f9b3bd36-51f3-4b46-bed6-94703557bfdf","Type":"ContainerDied","Data":"7c70994f716be35bf28cb7b0a3c927255a41351340dfc0a5c19ac6c5f9272f82"} Oct 30 22:30:43.126622 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:43.126620 5918 scope.go:122] "RemoveContainer" containerID="cb8aa5375f6f2c8f37b4bb60f755723baefbb297d9d5c7d55537a276b57ae445" Oct 30 22:30:43.135606 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:43.135588 5918 scope.go:122] "RemoveContainer" containerID="cb8aa5375f6f2c8f37b4bb60f755723baefbb297d9d5c7d55537a276b57ae445" Oct 30 22:30:43.135885 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:30:43.135864 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb8aa5375f6f2c8f37b4bb60f755723baefbb297d9d5c7d55537a276b57ae445\": container with ID starting with cb8aa5375f6f2c8f37b4bb60f755723baefbb297d9d5c7d55537a276b57ae445 not found: ID does not exist" containerID="cb8aa5375f6f2c8f37b4bb60f755723baefbb297d9d5c7d55537a276b57ae445" Oct 30 22:30:43.135956 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:43.135899 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb8aa5375f6f2c8f37b4bb60f755723baefbb297d9d5c7d55537a276b57ae445"} err="failed to get container status \"cb8aa5375f6f2c8f37b4bb60f755723baefbb297d9d5c7d55537a276b57ae445\": rpc error: code = NotFound desc = could not find container \"cb8aa5375f6f2c8f37b4bb60f755723baefbb297d9d5c7d55537a276b57ae445\": container with ID starting with cb8aa5375f6f2c8f37b4bb60f755723baefbb297d9d5c7d55537a276b57ae445 not found: ID does not exist" Oct 30 22:30:43.220147 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:43.220108 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9b3bd36-51f3-4b46-bed6-94703557bfdf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b3bd36-51f3-4b46-bed6-94703557bfdf-istiod-ca-cert\") pod \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\" (UID: \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\") " Oct 30 22:30:43.220368 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:43.220166 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9b3bd36-51f3-4b46-bed6-94703557bfdf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b3bd36-51f3-4b46-bed6-94703557bfdf-istio-podinfo\") pod \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\" (UID: \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\") " Oct 30 22:30:43.220368 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:43.220267 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9b3bd36-51f3-4b46-bed6-94703557bfdf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b3bd36-51f3-4b46-bed6-94703557bfdf-workload-socket\") pod \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\" (UID: \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\") " Oct 30 22:30:43.220368 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:43.220326 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9b3bd36-51f3-4b46-bed6-94703557bfdf-kube-api-access-hbprg\" (UniqueName: \"kubernetes.io/projected/f9b3bd36-51f3-4b46-bed6-94703557bfdf-kube-api-access-hbprg\") pod \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\" (UID: \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\") " Oct 30 22:30:43.220368 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:43.220360 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9b3bd36-51f3-4b46-bed6-94703557bfdf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b3bd36-51f3-4b46-bed6-94703557bfdf-credential-socket\") pod \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\" (UID: \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\") " Oct 30 22:30:43.220591 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:43.220391 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9b3bd36-51f3-4b46-bed6-94703557bfdf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b3bd36-51f3-4b46-bed6-94703557bfdf-workload-certs\") pod \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\" (UID: \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\") " Oct 30 22:30:43.220591 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:43.220434 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9b3bd36-51f3-4b46-bed6-94703557bfdf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b3bd36-51f3-4b46-bed6-94703557bfdf-istio-data\") pod \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\" (UID: \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\") " Oct 30 22:30:43.220591 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:43.220482 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9b3bd36-51f3-4b46-bed6-94703557bfdf-istio-token\" (UniqueName: \"kubernetes.io/projected/f9b3bd36-51f3-4b46-bed6-94703557bfdf-istio-token\") pod \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\" (UID: \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\") " Oct 30 22:30:43.220591 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:43.220560 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b3bd36-51f3-4b46-bed6-94703557bfdf-workload-socket" pod "f9b3bd36-51f3-4b46-bed6-94703557bfdf" (UID: "f9b3bd36-51f3-4b46-bed6-94703557bfdf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:43.220591 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:43.220581 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9b3bd36-51f3-4b46-bed6-94703557bfdf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b3bd36-51f3-4b46-bed6-94703557bfdf-istio-envoy\") pod \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\" (UID: \"f9b3bd36-51f3-4b46-bed6-94703557bfdf\") " Oct 30 22:30:43.220854 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:43.220589 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9b3bd36-51f3-4b46-bed6-94703557bfdf-istiod-ca-cert" pod "f9b3bd36-51f3-4b46-bed6-94703557bfdf" (UID: "f9b3bd36-51f3-4b46-bed6-94703557bfdf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:43.220854 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:43.220750 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b3bd36-51f3-4b46-bed6-94703557bfdf-workload-certs" pod "f9b3bd36-51f3-4b46-bed6-94703557bfdf" (UID: "f9b3bd36-51f3-4b46-bed6-94703557bfdf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:43.220854 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:43.220758 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b3bd36-51f3-4b46-bed6-94703557bfdf-credential-socket" pod "f9b3bd36-51f3-4b46-bed6-94703557bfdf" (UID: "f9b3bd36-51f3-4b46-bed6-94703557bfdf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:43.220854 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:43.220779 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b3bd36-51f3-4b46-bed6-94703557bfdf-istio-data" pod "f9b3bd36-51f3-4b46-bed6-94703557bfdf" (UID: "f9b3bd36-51f3-4b46-bed6-94703557bfdf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:43.221081 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:43.220958 5918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9b3bd36-51f3-4b46-bed6-94703557bfdf-istio-data\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:43.221081 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:43.220982 5918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9b3bd36-51f3-4b46-bed6-94703557bfdf-istiod-ca-cert\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:43.221081 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:43.221002 5918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b3bd36-51f3-4b46-bed6-94703557bfdf-workload-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:43.221081 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:43.221019 5918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9b3bd36-51f3-4b46-bed6-94703557bfdf-credential-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:43.221081 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:43.221031 5918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9b3bd36-51f3-4b46-bed6-94703557bfdf-workload-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:43.222950 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:43.222924 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9b3bd36-51f3-4b46-bed6-94703557bfdf-istio-token" pod "f9b3bd36-51f3-4b46-bed6-94703557bfdf" (UID: "f9b3bd36-51f3-4b46-bed6-94703557bfdf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:43.223058 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:43.222919 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b3bd36-51f3-4b46-bed6-94703557bfdf-istio-envoy" pod "f9b3bd36-51f3-4b46-bed6-94703557bfdf" (UID: "f9b3bd36-51f3-4b46-bed6-94703557bfdf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:43.223149 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:43.223133 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9b3bd36-51f3-4b46-bed6-94703557bfdf-kube-api-access-hbprg" pod "f9b3bd36-51f3-4b46-bed6-94703557bfdf" (UID: "f9b3bd36-51f3-4b46-bed6-94703557bfdf"). InnerVolumeSpecName "kube-api-access-hbprg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:43.223256 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:43.223145 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9b3bd36-51f3-4b46-bed6-94703557bfdf-istio-podinfo" pod "f9b3bd36-51f3-4b46-bed6-94703557bfdf" (UID: "f9b3bd36-51f3-4b46-bed6-94703557bfdf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:30:43.321662 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:43.321579 5918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9b3bd36-51f3-4b46-bed6-94703557bfdf-istio-envoy\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:43.321662 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:43.321606 5918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9b3bd36-51f3-4b46-bed6-94703557bfdf-istio-podinfo\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:43.321662 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:43.321616 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbprg\" (UniqueName: \"kubernetes.io/projected/f9b3bd36-51f3-4b46-bed6-94703557bfdf-kube-api-access-hbprg\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:43.321662 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:43.321626 5918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9b3bd36-51f3-4b46-bed6-94703557bfdf-istio-token\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:43.550466 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:43.550433 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728"] Oct 30 22:30:43.556994 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:43.556954 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-2182a5-openshift-default-89fd49f79-vd728"] Oct 30 22:30:43.850332 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:43.850297 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9b3bd36-51f3-4b46-bed6-94703557bfdf" path="/var/lib/kubelet/pods/f9b3bd36-51f3-4b46-bed6-94703557bfdf/volumes" Oct 30 22:30:44.812939 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:44.812905 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb"] Oct 30 22:30:44.813142 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:44.813120 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" podUID="e5345433-337f-4434-a218-a9880269ddc5" containerName="istio-proxy" containerID="cri-o://63f79333c56ff4649518117ae1e6cae4caf1680548007953679d46a552b3eed0" gracePeriod=30 Oct 30 22:30:50.073411 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.073381 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:30:50.166526 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.166440 5918 generic.go:356] "Generic (PLEG): container finished" podID="e5345433-337f-4434-a218-a9880269ddc5" containerID="63f79333c56ff4649518117ae1e6cae4caf1680548007953679d46a552b3eed0" exitCode=0 Oct 30 22:30:50.166526 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.166490 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" Oct 30 22:30:50.166526 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.166504 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" event={"ID":"e5345433-337f-4434-a218-a9880269ddc5","Type":"ContainerDied","Data":"63f79333c56ff4649518117ae1e6cae4caf1680548007953679d46a552b3eed0"} Oct 30 22:30:50.166769 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.166547 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb" event={"ID":"e5345433-337f-4434-a218-a9880269ddc5","Type":"ContainerDied","Data":"325cefe10d26831cb5e5349211d8dba81866fef41629f884d3d00a1f6cb2968c"} Oct 30 22:30:50.166769 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.166567 5918 scope.go:122] "RemoveContainer" containerID="63f79333c56ff4649518117ae1e6cae4caf1680548007953679d46a552b3eed0" Oct 30 22:30:50.175615 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.175594 5918 scope.go:122] "RemoveContainer" containerID="55569ea765efb899cd1a6c5657da7820a75548cd6069b499d0ce7f9047e11102" Oct 30 22:30:50.186955 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.186910 5918 scope.go:122] "RemoveContainer" containerID="63f79333c56ff4649518117ae1e6cae4caf1680548007953679d46a552b3eed0" Oct 30 22:30:50.187469 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:30:50.187438 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63f79333c56ff4649518117ae1e6cae4caf1680548007953679d46a552b3eed0\": container with ID starting with 63f79333c56ff4649518117ae1e6cae4caf1680548007953679d46a552b3eed0 not found: ID does not exist" containerID="63f79333c56ff4649518117ae1e6cae4caf1680548007953679d46a552b3eed0" Oct 30 22:30:50.187577 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.187477 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63f79333c56ff4649518117ae1e6cae4caf1680548007953679d46a552b3eed0"} err="failed to get container status \"63f79333c56ff4649518117ae1e6cae4caf1680548007953679d46a552b3eed0\": rpc error: code = NotFound desc = could not find container \"63f79333c56ff4649518117ae1e6cae4caf1680548007953679d46a552b3eed0\": container with ID starting with 63f79333c56ff4649518117ae1e6cae4caf1680548007953679d46a552b3eed0 not found: ID does not exist" Oct 30 22:30:50.187577 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.187496 5918 scope.go:122] "RemoveContainer" containerID="55569ea765efb899cd1a6c5657da7820a75548cd6069b499d0ce7f9047e11102" Oct 30 22:30:50.187773 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:30:50.187751 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55569ea765efb899cd1a6c5657da7820a75548cd6069b499d0ce7f9047e11102\": container with ID starting with 55569ea765efb899cd1a6c5657da7820a75548cd6069b499d0ce7f9047e11102 not found: ID does not exist" containerID="55569ea765efb899cd1a6c5657da7820a75548cd6069b499d0ce7f9047e11102" Oct 30 22:30:50.187833 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.187784 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55569ea765efb899cd1a6c5657da7820a75548cd6069b499d0ce7f9047e11102"} err="failed to get container status \"55569ea765efb899cd1a6c5657da7820a75548cd6069b499d0ce7f9047e11102\": rpc error: code = NotFound desc = could not find container \"55569ea765efb899cd1a6c5657da7820a75548cd6069b499d0ce7f9047e11102\": container with ID starting with 55569ea765efb899cd1a6c5657da7820a75548cd6069b499d0ce7f9047e11102 not found: ID does not exist" Oct 30 22:30:50.189090 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.189072 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5345433-337f-4434-a218-a9880269ddc5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5345433-337f-4434-a218-a9880269ddc5-istio-podinfo\") pod \"e5345433-337f-4434-a218-a9880269ddc5\" (UID: \"e5345433-337f-4434-a218-a9880269ddc5\") " Oct 30 22:30:50.189196 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.189113 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5345433-337f-4434-a218-a9880269ddc5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5345433-337f-4434-a218-a9880269ddc5-istiod-ca-cert\") pod \"e5345433-337f-4434-a218-a9880269ddc5\" (UID: \"e5345433-337f-4434-a218-a9880269ddc5\") " Oct 30 22:30:50.189196 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.189155 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5345433-337f-4434-a218-a9880269ddc5-istio-token\" (UniqueName: \"kubernetes.io/projected/e5345433-337f-4434-a218-a9880269ddc5-istio-token\") pod \"e5345433-337f-4434-a218-a9880269ddc5\" (UID: \"e5345433-337f-4434-a218-a9880269ddc5\") " Oct 30 22:30:50.189325 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.189192 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5345433-337f-4434-a218-a9880269ddc5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5345433-337f-4434-a218-a9880269ddc5-istio-data\") pod \"e5345433-337f-4434-a218-a9880269ddc5\" (UID: \"e5345433-337f-4434-a218-a9880269ddc5\") " Oct 30 22:30:50.189378 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.189328 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5345433-337f-4434-a218-a9880269ddc5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5345433-337f-4434-a218-a9880269ddc5-workload-certs\") pod \"e5345433-337f-4434-a218-a9880269ddc5\" (UID: \"e5345433-337f-4434-a218-a9880269ddc5\") " Oct 30 22:30:50.189432 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.189399 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5345433-337f-4434-a218-a9880269ddc5-kube-api-access-tvqtn\" (UniqueName: \"kubernetes.io/projected/e5345433-337f-4434-a218-a9880269ddc5-kube-api-access-tvqtn\") pod \"e5345433-337f-4434-a218-a9880269ddc5\" (UID: \"e5345433-337f-4434-a218-a9880269ddc5\") " Oct 30 22:30:50.189485 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.189437 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5345433-337f-4434-a218-a9880269ddc5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5345433-337f-4434-a218-a9880269ddc5-credential-socket\") pod \"e5345433-337f-4434-a218-a9880269ddc5\" (UID: \"e5345433-337f-4434-a218-a9880269ddc5\") " Oct 30 22:30:50.189536 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.189510 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5345433-337f-4434-a218-a9880269ddc5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5345433-337f-4434-a218-a9880269ddc5-istio-envoy\") pod \"e5345433-337f-4434-a218-a9880269ddc5\" (UID: \"e5345433-337f-4434-a218-a9880269ddc5\") " Oct 30 22:30:50.189584 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.189550 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5345433-337f-4434-a218-a9880269ddc5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5345433-337f-4434-a218-a9880269ddc5-workload-socket\") pod \"e5345433-337f-4434-a218-a9880269ddc5\" (UID: \"e5345433-337f-4434-a218-a9880269ddc5\") " Oct 30 22:30:50.189673 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.189630 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5345433-337f-4434-a218-a9880269ddc5-workload-certs" pod "e5345433-337f-4434-a218-a9880269ddc5" (UID: "e5345433-337f-4434-a218-a9880269ddc5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:50.189768 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.189742 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5345433-337f-4434-a218-a9880269ddc5-credential-socket" pod "e5345433-337f-4434-a218-a9880269ddc5" (UID: "e5345433-337f-4434-a218-a9880269ddc5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:50.190126 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.189966 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5345433-337f-4434-a218-a9880269ddc5-istio-data" pod "e5345433-337f-4434-a218-a9880269ddc5" (UID: "e5345433-337f-4434-a218-a9880269ddc5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:50.190126 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.189981 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5345433-337f-4434-a218-a9880269ddc5-workload-socket" pod "e5345433-337f-4434-a218-a9880269ddc5" (UID: "e5345433-337f-4434-a218-a9880269ddc5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:50.190126 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.190072 5918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5345433-337f-4434-a218-a9880269ddc5-workload-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:50.190126 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.190093 5918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5345433-337f-4434-a218-a9880269ddc5-credential-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:50.190425 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.190309 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5345433-337f-4434-a218-a9880269ddc5-istiod-ca-cert" pod "e5345433-337f-4434-a218-a9880269ddc5" (UID: "e5345433-337f-4434-a218-a9880269ddc5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:50.191618 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.191595 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5345433-337f-4434-a218-a9880269ddc5-istio-token" pod "e5345433-337f-4434-a218-a9880269ddc5" (UID: "e5345433-337f-4434-a218-a9880269ddc5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:50.191728 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.191707 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5345433-337f-4434-a218-a9880269ddc5-istio-envoy" pod "e5345433-337f-4434-a218-a9880269ddc5" (UID: "e5345433-337f-4434-a218-a9880269ddc5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:50.192022 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.192001 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5345433-337f-4434-a218-a9880269ddc5-istio-podinfo" pod "e5345433-337f-4434-a218-a9880269ddc5" (UID: "e5345433-337f-4434-a218-a9880269ddc5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:30:50.192022 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.192008 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5345433-337f-4434-a218-a9880269ddc5-kube-api-access-tvqtn" pod "e5345433-337f-4434-a218-a9880269ddc5" (UID: "e5345433-337f-4434-a218-a9880269ddc5"). InnerVolumeSpecName "kube-api-access-tvqtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:50.291291 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.291257 5918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5345433-337f-4434-a218-a9880269ddc5-istio-envoy\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:50.291291 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.291291 5918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5345433-337f-4434-a218-a9880269ddc5-workload-socket\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:50.291515 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.291304 5918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5345433-337f-4434-a218-a9880269ddc5-istio-podinfo\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:50.291515 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.291316 5918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5345433-337f-4434-a218-a9880269ddc5-istiod-ca-cert\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:50.291515 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.291328 5918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5345433-337f-4434-a218-a9880269ddc5-istio-token\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:50.291515 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.291344 5918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5345433-337f-4434-a218-a9880269ddc5-istio-data\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:50.291515 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.291356 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvqtn\" (UniqueName: \"kubernetes.io/projected/e5345433-337f-4434-a218-a9880269ddc5-kube-api-access-tvqtn\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:50.301782 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.301749 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d"] Oct 30 22:30:50.597066 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.597032 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb"] Oct 30 22:30:50.602658 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:50.602629 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d59c58-openshift-default-79b4dc54b7-dwdxb"] Oct 30 22:30:51.274312 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:30:51.274277 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:30:51.851979 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:30:51.851940 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5345433-337f-4434-a218-a9880269ddc5" path="/var/lib/kubelet/pods/e5345433-337f-4434-a218-a9880269ddc5/volumes" Oct 30 22:31:05.322368 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:05.322305 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" podUID="e63fad41-0e42-41ee-ae20-919f15fffe55" containerName="main" containerID="cri-o://3db2b01524a206c993016abd96340146235b614665b091d2842a0e7a398612cd" gracePeriod=45 Oct 30 22:31:05.567788 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:05.567765 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" Oct 30 22:31:05.653973 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:05.653897 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e63fad41-0e42-41ee-ae20-919f15fffe55-home\" (UniqueName: \"kubernetes.io/empty-dir/e63fad41-0e42-41ee-ae20-919f15fffe55-home\") pod \"e63fad41-0e42-41ee-ae20-919f15fffe55\" (UID: \"e63fad41-0e42-41ee-ae20-919f15fffe55\") " Oct 30 22:31:05.653973 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:05.653948 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e63fad41-0e42-41ee-ae20-919f15fffe55-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e63fad41-0e42-41ee-ae20-919f15fffe55-tmp-dir\") pod \"e63fad41-0e42-41ee-ae20-919f15fffe55\" (UID: \"e63fad41-0e42-41ee-ae20-919f15fffe55\") " Oct 30 22:31:05.654170 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:05.653989 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e63fad41-0e42-41ee-ae20-919f15fffe55-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e63fad41-0e42-41ee-ae20-919f15fffe55-model-cache\") pod \"e63fad41-0e42-41ee-ae20-919f15fffe55\" (UID: \"e63fad41-0e42-41ee-ae20-919f15fffe55\") " Oct 30 22:31:05.654170 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:05.654030 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e63fad41-0e42-41ee-ae20-919f15fffe55-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e63fad41-0e42-41ee-ae20-919f15fffe55-dshm\") pod \"e63fad41-0e42-41ee-ae20-919f15fffe55\" (UID: \"e63fad41-0e42-41ee-ae20-919f15fffe55\") " Oct 30 22:31:05.654170 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:05.654062 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e63fad41-0e42-41ee-ae20-919f15fffe55-tls-certs\" (UniqueName: \"kubernetes.io/secret/e63fad41-0e42-41ee-ae20-919f15fffe55-tls-certs\") pod \"e63fad41-0e42-41ee-ae20-919f15fffe55\" (UID: \"e63fad41-0e42-41ee-ae20-919f15fffe55\") " Oct 30 22:31:05.654170 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:05.654094 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e63fad41-0e42-41ee-ae20-919f15fffe55-kube-api-access-mjk5d\" (UniqueName: \"kubernetes.io/projected/e63fad41-0e42-41ee-ae20-919f15fffe55-kube-api-access-mjk5d\") pod \"e63fad41-0e42-41ee-ae20-919f15fffe55\" (UID: \"e63fad41-0e42-41ee-ae20-919f15fffe55\") " Oct 30 22:31:05.654421 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:05.654193 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e63fad41-0e42-41ee-ae20-919f15fffe55-home" pod "e63fad41-0e42-41ee-ae20-919f15fffe55" (UID: "e63fad41-0e42-41ee-ae20-919f15fffe55"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:05.654421 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:05.654274 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e63fad41-0e42-41ee-ae20-919f15fffe55-tmp-dir" pod "e63fad41-0e42-41ee-ae20-919f15fffe55" (UID: "e63fad41-0e42-41ee-ae20-919f15fffe55"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:05.654421 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:05.654320 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e63fad41-0e42-41ee-ae20-919f15fffe55-model-cache" pod "e63fad41-0e42-41ee-ae20-919f15fffe55" (UID: "e63fad41-0e42-41ee-ae20-919f15fffe55"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:05.654550 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:05.654486 5918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e63fad41-0e42-41ee-ae20-919f15fffe55-tmp-dir\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:05.654550 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:05.654504 5918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e63fad41-0e42-41ee-ae20-919f15fffe55-model-cache\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:05.654550 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:05.654516 5918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e63fad41-0e42-41ee-ae20-919f15fffe55-home\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:05.656432 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:05.656406 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e63fad41-0e42-41ee-ae20-919f15fffe55-dshm" pod "e63fad41-0e42-41ee-ae20-919f15fffe55" (UID: "e63fad41-0e42-41ee-ae20-919f15fffe55"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:05.656559 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:05.656454 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e63fad41-0e42-41ee-ae20-919f15fffe55-tls-certs" pod "e63fad41-0e42-41ee-ae20-919f15fffe55" (UID: "e63fad41-0e42-41ee-ae20-919f15fffe55"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:05.656559 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:05.656490 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e63fad41-0e42-41ee-ae20-919f15fffe55-kube-api-access-mjk5d" pod "e63fad41-0e42-41ee-ae20-919f15fffe55" (UID: "e63fad41-0e42-41ee-ae20-919f15fffe55"). InnerVolumeSpecName "kube-api-access-mjk5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:05.755651 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:05.755614 5918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e63fad41-0e42-41ee-ae20-919f15fffe55-dshm\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:05.755651 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:05.755644 5918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e63fad41-0e42-41ee-ae20-919f15fffe55-tls-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:05.755651 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:05.755657 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjk5d\" (UniqueName: \"kubernetes.io/projected/e63fad41-0e42-41ee-ae20-919f15fffe55-kube-api-access-mjk5d\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:06.254712 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:06.254676 5918 generic.go:356] "Generic (PLEG): container finished" podID="e63fad41-0e42-41ee-ae20-919f15fffe55" containerID="3db2b01524a206c993016abd96340146235b614665b091d2842a0e7a398612cd" exitCode=0 Oct 30 22:31:06.254902 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:06.254727 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" Oct 30 22:31:06.254902 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:06.254729 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" event={"ID":"e63fad41-0e42-41ee-ae20-919f15fffe55","Type":"ContainerDied","Data":"3db2b01524a206c993016abd96340146235b614665b091d2842a0e7a398612cd"} Oct 30 22:31:06.254902 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:06.254845 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d" event={"ID":"e63fad41-0e42-41ee-ae20-919f15fffe55","Type":"ContainerDied","Data":"b97d4f1a3493cab2bc97dfde29b65e12290e83bedb2d674090064b8f3da4fc43"} Oct 30 22:31:06.254902 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:06.254868 5918 scope.go:122] "RemoveContainer" containerID="3db2b01524a206c993016abd96340146235b614665b091d2842a0e7a398612cd" Oct 30 22:31:06.264011 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:06.263992 5918 scope.go:122] "RemoveContainer" containerID="3db2b01524a206c993016abd96340146235b614665b091d2842a0e7a398612cd" Oct 30 22:31:06.264288 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:31:06.264264 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3db2b01524a206c993016abd96340146235b614665b091d2842a0e7a398612cd\": container with ID starting with 3db2b01524a206c993016abd96340146235b614665b091d2842a0e7a398612cd not found: ID does not exist" containerID="3db2b01524a206c993016abd96340146235b614665b091d2842a0e7a398612cd" Oct 30 22:31:06.264366 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:06.264301 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3db2b01524a206c993016abd96340146235b614665b091d2842a0e7a398612cd"} err="failed to get container status \"3db2b01524a206c993016abd96340146235b614665b091d2842a0e7a398612cd\": rpc error: code = NotFound desc = could not find container \"3db2b01524a206c993016abd96340146235b614665b091d2842a0e7a398612cd\": container with ID starting with 3db2b01524a206c993016abd96340146235b614665b091d2842a0e7a398612cd not found: ID does not exist" Oct 30 22:31:06.423156 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:06.423117 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d"] Oct 30 22:31:06.473960 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:06.473924 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-8ca52a/simulator-w3-8ca52a-kserve-7b8d5774d-jrw6d"] Oct 30 22:31:07.850880 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:07.850850 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e63fad41-0e42-41ee-ae20-919f15fffe55" path="/var/lib/kubelet/pods/e63fad41-0e42-41ee-ae20-919f15fffe55/volumes" Oct 30 22:31:12.067571 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:12.067528 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx"] Oct 30 22:31:12.068107 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:12.067758 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx" podUID="599ae550-c43b-4094-be3d-d9a256a25a63" containerName="maas-api" containerID="cri-o://33372e9fcc5a4e7e13fc055fb2a5ee3771280ee3d2620fbec4e4e7c491e767c8" gracePeriod=30 Oct 30 22:31:12.288650 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:12.288618 5918 generic.go:356] "Generic (PLEG): container finished" podID="599ae550-c43b-4094-be3d-d9a256a25a63" containerID="33372e9fcc5a4e7e13fc055fb2a5ee3771280ee3d2620fbec4e4e7c491e767c8" exitCode=0 Oct 30 22:31:12.288840 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:12.288691 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx" event={"ID":"599ae550-c43b-4094-be3d-d9a256a25a63","Type":"ContainerDied","Data":"33372e9fcc5a4e7e13fc055fb2a5ee3771280ee3d2620fbec4e4e7c491e767c8"} Oct 30 22:31:12.334034 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:12.334008 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx" Oct 30 22:31:12.422227 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:12.422185 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/599ae550-c43b-4094-be3d-d9a256a25a63-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/599ae550-c43b-4094-be3d-d9a256a25a63-maas-api-tls\") pod \"599ae550-c43b-4094-be3d-d9a256a25a63\" (UID: \"599ae550-c43b-4094-be3d-d9a256a25a63\") " Oct 30 22:31:12.422400 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:12.422258 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/599ae550-c43b-4094-be3d-d9a256a25a63-metrics-certs\" (UniqueName: \"kubernetes.io/secret/599ae550-c43b-4094-be3d-d9a256a25a63-metrics-certs\") pod \"599ae550-c43b-4094-be3d-d9a256a25a63\" (UID: \"599ae550-c43b-4094-be3d-d9a256a25a63\") " Oct 30 22:31:12.422400 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:12.422322 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/599ae550-c43b-4094-be3d-d9a256a25a63-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/599ae550-c43b-4094-be3d-d9a256a25a63-openshift-service-ca\") pod \"599ae550-c43b-4094-be3d-d9a256a25a63\" (UID: \"599ae550-c43b-4094-be3d-d9a256a25a63\") " Oct 30 22:31:12.422528 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:12.422419 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/599ae550-c43b-4094-be3d-d9a256a25a63-kube-api-access-c6lcs\" (UniqueName: \"kubernetes.io/projected/599ae550-c43b-4094-be3d-d9a256a25a63-kube-api-access-c6lcs\") pod \"599ae550-c43b-4094-be3d-d9a256a25a63\" (UID: \"599ae550-c43b-4094-be3d-d9a256a25a63\") " Oct 30 22:31:12.422644 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:12.422615 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/599ae550-c43b-4094-be3d-d9a256a25a63-openshift-service-ca" pod "599ae550-c43b-4094-be3d-d9a256a25a63" (UID: "599ae550-c43b-4094-be3d-d9a256a25a63"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:12.422777 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:12.422757 5918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/599ae550-c43b-4094-be3d-d9a256a25a63-openshift-service-ca\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:12.424864 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:12.424820 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/599ae550-c43b-4094-be3d-d9a256a25a63-maas-api-tls" pod "599ae550-c43b-4094-be3d-d9a256a25a63" (UID: "599ae550-c43b-4094-be3d-d9a256a25a63"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:12.424864 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:12.424847 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/599ae550-c43b-4094-be3d-d9a256a25a63-metrics-certs" pod "599ae550-c43b-4094-be3d-d9a256a25a63" (UID: "599ae550-c43b-4094-be3d-d9a256a25a63"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:12.424864 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:12.424855 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/599ae550-c43b-4094-be3d-d9a256a25a63-kube-api-access-c6lcs" pod "599ae550-c43b-4094-be3d-d9a256a25a63" (UID: "599ae550-c43b-4094-be3d-d9a256a25a63"). InnerVolumeSpecName "kube-api-access-c6lcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:12.523844 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:12.523811 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6lcs\" (UniqueName: \"kubernetes.io/projected/599ae550-c43b-4094-be3d-d9a256a25a63-kube-api-access-c6lcs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:12.523844 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:12.523842 5918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/599ae550-c43b-4094-be3d-d9a256a25a63-maas-api-tls\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:12.523844 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:12.523853 5918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/599ae550-c43b-4094-be3d-d9a256a25a63-metrics-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:31:13.295669 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:13.295633 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx" event={"ID":"599ae550-c43b-4094-be3d-d9a256a25a63","Type":"ContainerDied","Data":"604e413602f974af43b4dd361c77bd1ce816a0a16f722d77185ef78b4835b295"} Oct 30 22:31:13.296134 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:13.295690 5918 scope.go:122] "RemoveContainer" containerID="33372e9fcc5a4e7e13fc055fb2a5ee3771280ee3d2620fbec4e4e7c491e767c8" Oct 30 22:31:13.296134 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:13.295649 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx" Oct 30 22:31:13.464086 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:13.464053 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx"] Oct 30 22:31:13.514579 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:13.514545 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8ca52a-655c8f7444-8kwbx"] Oct 30 22:31:13.850622 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:31:13.850574 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="599ae550-c43b-4094-be3d-d9a256a25a63" path="/var/lib/kubelet/pods/599ae550-c43b-4094-be3d-d9a256a25a63/volumes" Oct 30 22:31:21.274478 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:31:21.274397 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:31:51.274801 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:31:51.274742 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:32:21.273708 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:32:21.273665 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:32:51.274406 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:32:51.274326 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:33:21.285616 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:33:21.285574 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:33:42.093042 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:33:42.093009 5918 scope.go:122] "RemoveContainer" containerID="25bcbdee8822855109eeeb2cfb7ccc96d78849f535cabc5ab6710be39b35c7a9" Oct 30 22:33:51.273484 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:33:51.273443 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:34:21.275484 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:34:21.275399 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:34:51.274334 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:34:51.274294 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:34:55.679418 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:34:55.679371 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dkzwh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:34:55.679860 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:34:55.679517 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dkzwh" podUID="4488ee36-392f-42c2-8dad-962472895f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:35:21.274929 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:35:21.274893 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:35:51.274106 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:35:51.274023 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:36:21.274556 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:36:21.274515 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:36:51.286642 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:36:51.286592 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:37:12.611240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:37:12.610408 5918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-zwvws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:37:12.611240 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:37:12.610498 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zwvws" podUID="ac8d2c79-d9b9-4650-8ffa-39022153bff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:37:21.274674 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:37:21.274581 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:37:51.274323 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:37:51.274285 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:38:21.274868 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:38:21.274826 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:38:51.274319 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:38:51.274208 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:39:21.274847 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:39:21.274811 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:39:50.639722 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.639682 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs"] Oct 30 22:39:50.640341 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640283 5918 cpu_manager.go:414] "containerMap: removing container" podUID="f6f37484-37b2-4fab-8b6b-cdc1c171ff93" containerName="istio-proxy" Oct 30 22:39:50.640341 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640305 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6f37484-37b2-4fab-8b6b-cdc1c171ff93" containerName="istio-proxy" Oct 30 22:39:50.640341 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640323 5918 cpu_manager.go:414] "containerMap: removing container" podUID="bc656e82-591f-4997-9576-9297fd4dde35" containerName="payload-pre-processing" Oct 30 22:39:50.640341 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640331 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc656e82-591f-4997-9576-9297fd4dde35" containerName="payload-pre-processing" Oct 30 22:39:50.640579 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640344 5918 cpu_manager.go:414] "containerMap: removing container" podUID="319acc25-060d-4921-bcaa-a62f41412146" containerName="payload-processing" Oct 30 22:39:50.640579 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640352 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="319acc25-060d-4921-bcaa-a62f41412146" containerName="payload-processing" Oct 30 22:39:50.640579 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640367 5918 cpu_manager.go:414] "containerMap: removing container" podUID="f9b3bd36-51f3-4b46-bed6-94703557bfdf" containerName="istio-proxy" Oct 30 22:39:50.640579 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640376 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9b3bd36-51f3-4b46-bed6-94703557bfdf" containerName="istio-proxy" Oct 30 22:39:50.640579 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640385 5918 cpu_manager.go:414] "containerMap: removing container" podUID="ab957b47-90d6-497a-8334-1d5d4541dc7a" containerName="main" Oct 30 22:39:50.640579 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640393 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab957b47-90d6-497a-8334-1d5d4541dc7a" containerName="main" Oct 30 22:39:50.640579 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640406 5918 cpu_manager.go:414] "containerMap: removing container" podUID="e63fad41-0e42-41ee-ae20-919f15fffe55" containerName="main" Oct 30 22:39:50.640579 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640412 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e63fad41-0e42-41ee-ae20-919f15fffe55" containerName="main" Oct 30 22:39:50.640579 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640427 5918 cpu_manager.go:414] "containerMap: removing container" podUID="9b4c802f-e487-4d43-a8aa-ef87ddc27fcf" containerName="payload-pre-processing" Oct 30 22:39:50.640579 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640434 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b4c802f-e487-4d43-a8aa-ef87ddc27fcf" containerName="payload-pre-processing" Oct 30 22:39:50.640579 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640445 5918 cpu_manager.go:414] "containerMap: removing container" podUID="de7b01aa-6f28-4f10-b7ce-b615537fb99f" containerName="maas-api" Oct 30 22:39:50.640579 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640452 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de7b01aa-6f28-4f10-b7ce-b615537fb99f" containerName="maas-api" Oct 30 22:39:50.640579 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640469 5918 cpu_manager.go:414] "containerMap: removing container" podUID="035da59c-1f1a-4384-9c42-43bc5d184bea" containerName="maas-api" Oct 30 22:39:50.640579 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640476 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="035da59c-1f1a-4384-9c42-43bc5d184bea" containerName="maas-api" Oct 30 22:39:50.640579 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640485 5918 cpu_manager.go:414] "containerMap: removing container" podUID="544a91a1-bd59-4dd2-91a6-33a33bebccb9" containerName="maas-api" Oct 30 22:39:50.640579 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640493 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="544a91a1-bd59-4dd2-91a6-33a33bebccb9" containerName="maas-api" Oct 30 22:39:50.640579 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640502 5918 cpu_manager.go:414] "containerMap: removing container" podUID="100467f8-8ee2-4cda-a9a7-f52b11af838c" containerName="istio-proxy" Oct 30 22:39:50.640579 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640509 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="100467f8-8ee2-4cda-a9a7-f52b11af838c" containerName="istio-proxy" Oct 30 22:39:50.640579 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640524 5918 cpu_manager.go:414] "containerMap: removing container" podUID="dc20dd97-eb75-42c5-8e50-6b0b5cb54b45" containerName="main" Oct 30 22:39:50.640579 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640533 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc20dd97-eb75-42c5-8e50-6b0b5cb54b45" containerName="main" Oct 30 22:39:50.640579 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640545 5918 cpu_manager.go:414] "containerMap: removing container" podUID="97f4e41b-50c7-441c-8ff8-6c815143b79b" containerName="istio-proxy" Oct 30 22:39:50.640579 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640552 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97f4e41b-50c7-441c-8ff8-6c815143b79b" containerName="istio-proxy" Oct 30 22:39:50.640579 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640565 5918 cpu_manager.go:414] "containerMap: removing container" podUID="e5345433-337f-4434-a218-a9880269ddc5" containerName="istio-proxy" Oct 30 22:39:50.640579 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640572 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5345433-337f-4434-a218-a9880269ddc5" containerName="istio-proxy" Oct 30 22:39:50.641766 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640590 5918 cpu_manager.go:414] "containerMap: removing container" podUID="599ae550-c43b-4094-be3d-d9a256a25a63" containerName="maas-api" Oct 30 22:39:50.641766 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640631 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="599ae550-c43b-4094-be3d-d9a256a25a63" containerName="maas-api" Oct 30 22:39:50.641766 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640643 5918 cpu_manager.go:414] "containerMap: removing container" podUID="e5345433-337f-4434-a218-a9880269ddc5" containerName="istio-proxy" Oct 30 22:39:50.641766 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640650 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5345433-337f-4434-a218-a9880269ddc5" containerName="istio-proxy" Oct 30 22:39:50.641766 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640665 5918 cpu_manager.go:414] "containerMap: removing container" podUID="a7dd97a6-507e-4441-b9d2-381bee4d6dd2" containerName="istio-proxy" Oct 30 22:39:50.641766 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640673 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7dd97a6-507e-4441-b9d2-381bee4d6dd2" containerName="istio-proxy" Oct 30 22:39:50.641766 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640698 5918 cpu_manager.go:414] "containerMap: removing container" podUID="e0018134-1144-4076-95ca-7383b46a2985" containerName="istio-proxy" Oct 30 22:39:50.641766 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640705 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0018134-1144-4076-95ca-7383b46a2985" containerName="istio-proxy" Oct 30 22:39:50.641766 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640738 5918 cpu_manager.go:414] "containerMap: removing container" podUID="1a1ae368-0e8c-408d-bfcd-8f4a8194464f" containerName="main" Oct 30 22:39:50.641766 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640745 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a1ae368-0e8c-408d-bfcd-8f4a8194464f" containerName="main" Oct 30 22:39:50.641766 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640857 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="599ae550-c43b-4094-be3d-d9a256a25a63" containerName="maas-api" Oct 30 22:39:50.641766 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640869 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5345433-337f-4434-a218-a9880269ddc5" containerName="istio-proxy" Oct 30 22:39:50.641766 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640879 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7dd97a6-507e-4441-b9d2-381bee4d6dd2" containerName="istio-proxy" Oct 30 22:39:50.641766 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640889 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6f37484-37b2-4fab-8b6b-cdc1c171ff93" containerName="istio-proxy" Oct 30 22:39:50.641766 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640899 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0018134-1144-4076-95ca-7383b46a2985" containerName="istio-proxy" Oct 30 22:39:50.641766 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640909 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a1ae368-0e8c-408d-bfcd-8f4a8194464f" containerName="main" Oct 30 22:39:50.641766 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640922 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6f37484-37b2-4fab-8b6b-cdc1c171ff93" containerName="istio-proxy" Oct 30 22:39:50.641766 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640932 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc656e82-591f-4997-9576-9297fd4dde35" containerName="payload-pre-processing" Oct 30 22:39:50.641766 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640941 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="319acc25-060d-4921-bcaa-a62f41412146" containerName="payload-processing" Oct 30 22:39:50.641766 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640951 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9b3bd36-51f3-4b46-bed6-94703557bfdf" containerName="istio-proxy" Oct 30 22:39:50.641766 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640960 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="e63fad41-0e42-41ee-ae20-919f15fffe55" containerName="main" Oct 30 22:39:50.641766 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640967 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab957b47-90d6-497a-8334-1d5d4541dc7a" containerName="main" Oct 30 22:39:50.641766 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640977 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b4c802f-e487-4d43-a8aa-ef87ddc27fcf" containerName="payload-pre-processing" Oct 30 22:39:50.641766 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640987 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="de7b01aa-6f28-4f10-b7ce-b615537fb99f" containerName="maas-api" Oct 30 22:39:50.641766 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.640995 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="035da59c-1f1a-4384-9c42-43bc5d184bea" containerName="maas-api" Oct 30 22:39:50.641766 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.641004 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="544a91a1-bd59-4dd2-91a6-33a33bebccb9" containerName="maas-api" Oct 30 22:39:50.641766 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.641014 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="100467f8-8ee2-4cda-a9a7-f52b11af838c" containerName="istio-proxy" Oct 30 22:39:50.641766 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.641024 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc20dd97-eb75-42c5-8e50-6b0b5cb54b45" containerName="main" Oct 30 22:39:50.641766 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.641032 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="97f4e41b-50c7-441c-8ff8-6c815143b79b" containerName="istio-proxy" Oct 30 22:39:50.644541 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.644520 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs" Oct 30 22:39:50.748940 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.748902 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-ca3b17\"/\"kube-root-ca.crt\"" Oct 30 22:39:50.748940 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.748902 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-ca3b17\"/\"openshift-service-ca.crt\"" Oct 30 22:39:50.749137 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.748957 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-ca3b17\"/\"simulator-single-main-ca3b17-kserve-self-signed-certs\"" Oct 30 22:39:50.749137 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.748968 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-ca3b17\"/\"default-dockercfg-rxsp4\"" Oct 30 22:39:50.749779 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.749757 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f72d065-2264-40ce-9682-e02ab4a7a942-dshm\") pod \"simulator-single-main-ca3b17-kserve-59c959669f-9fgbs\" (UID: \"2f72d065-2264-40ce-9682-e02ab4a7a942\") " pod="e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs" Oct 30 22:39:50.749858 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.749809 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chd4r\" (UniqueName: \"kubernetes.io/projected/2f72d065-2264-40ce-9682-e02ab4a7a942-kube-api-access-chd4r\") pod \"simulator-single-main-ca3b17-kserve-59c959669f-9fgbs\" (UID: \"2f72d065-2264-40ce-9682-e02ab4a7a942\") " pod="e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs" Oct 30 22:39:50.749858 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.749831 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f72d065-2264-40ce-9682-e02ab4a7a942-model-cache\") pod \"simulator-single-main-ca3b17-kserve-59c959669f-9fgbs\" (UID: \"2f72d065-2264-40ce-9682-e02ab4a7a942\") " pod="e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs" Oct 30 22:39:50.749957 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.749941 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f72d065-2264-40ce-9682-e02ab4a7a942-tmp-dir\") pod \"simulator-single-main-ca3b17-kserve-59c959669f-9fgbs\" (UID: \"2f72d065-2264-40ce-9682-e02ab4a7a942\") " pod="e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs" Oct 30 22:39:50.749997 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.749989 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f72d065-2264-40ce-9682-e02ab4a7a942-tls-certs\") pod \"simulator-single-main-ca3b17-kserve-59c959669f-9fgbs\" (UID: \"2f72d065-2264-40ce-9682-e02ab4a7a942\") " pod="e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs" Oct 30 22:39:50.750104 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.750084 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f72d065-2264-40ce-9682-e02ab4a7a942-home\") pod \"simulator-single-main-ca3b17-kserve-59c959669f-9fgbs\" (UID: \"2f72d065-2264-40ce-9682-e02ab4a7a942\") " pod="e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs" Oct 30 22:39:50.753519 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.753495 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs"] Oct 30 22:39:50.851404 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.851364 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f72d065-2264-40ce-9682-e02ab4a7a942-tls-certs\") pod \"simulator-single-main-ca3b17-kserve-59c959669f-9fgbs\" (UID: \"2f72d065-2264-40ce-9682-e02ab4a7a942\") " pod="e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs" Oct 30 22:39:50.851606 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.851518 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f72d065-2264-40ce-9682-e02ab4a7a942-home\") pod \"simulator-single-main-ca3b17-kserve-59c959669f-9fgbs\" (UID: \"2f72d065-2264-40ce-9682-e02ab4a7a942\") " pod="e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs" Oct 30 22:39:50.851606 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.851578 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f72d065-2264-40ce-9682-e02ab4a7a942-dshm\") pod \"simulator-single-main-ca3b17-kserve-59c959669f-9fgbs\" (UID: \"2f72d065-2264-40ce-9682-e02ab4a7a942\") " pod="e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs" Oct 30 22:39:50.851738 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.851658 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chd4r\" (UniqueName: \"kubernetes.io/projected/2f72d065-2264-40ce-9682-e02ab4a7a942-kube-api-access-chd4r\") pod \"simulator-single-main-ca3b17-kserve-59c959669f-9fgbs\" (UID: \"2f72d065-2264-40ce-9682-e02ab4a7a942\") " pod="e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs" Oct 30 22:39:50.851738 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.851695 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f72d065-2264-40ce-9682-e02ab4a7a942-model-cache\") pod \"simulator-single-main-ca3b17-kserve-59c959669f-9fgbs\" (UID: \"2f72d065-2264-40ce-9682-e02ab4a7a942\") " pod="e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs" Oct 30 22:39:50.851738 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.851722 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f72d065-2264-40ce-9682-e02ab4a7a942-tmp-dir\") pod \"simulator-single-main-ca3b17-kserve-59c959669f-9fgbs\" (UID: \"2f72d065-2264-40ce-9682-e02ab4a7a942\") " pod="e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs" Oct 30 22:39:50.852039 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.852012 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f72d065-2264-40ce-9682-e02ab4a7a942-home\") pod \"simulator-single-main-ca3b17-kserve-59c959669f-9fgbs\" (UID: \"2f72d065-2264-40ce-9682-e02ab4a7a942\") " pod="e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs" Oct 30 22:39:50.852159 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.852056 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f72d065-2264-40ce-9682-e02ab4a7a942-model-cache\") pod \"simulator-single-main-ca3b17-kserve-59c959669f-9fgbs\" (UID: \"2f72d065-2264-40ce-9682-e02ab4a7a942\") " pod="e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs" Oct 30 22:39:50.852159 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.852133 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f72d065-2264-40ce-9682-e02ab4a7a942-tmp-dir\") pod \"simulator-single-main-ca3b17-kserve-59c959669f-9fgbs\" (UID: \"2f72d065-2264-40ce-9682-e02ab4a7a942\") " pod="e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs" Oct 30 22:39:50.853938 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.853919 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f72d065-2264-40ce-9682-e02ab4a7a942-dshm\") pod \"simulator-single-main-ca3b17-kserve-59c959669f-9fgbs\" (UID: \"2f72d065-2264-40ce-9682-e02ab4a7a942\") " pod="e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs" Oct 30 22:39:50.854119 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.854100 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f72d065-2264-40ce-9682-e02ab4a7a942-tls-certs\") pod \"simulator-single-main-ca3b17-kserve-59c959669f-9fgbs\" (UID: \"2f72d065-2264-40ce-9682-e02ab4a7a942\") " pod="e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs" Oct 30 22:39:50.915544 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.915459 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chd4r\" (UniqueName: \"kubernetes.io/projected/2f72d065-2264-40ce-9682-e02ab4a7a942-kube-api-access-chd4r\") pod \"simulator-single-main-ca3b17-kserve-59c959669f-9fgbs\" (UID: \"2f72d065-2264-40ce-9682-e02ab4a7a942\") " pod="e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs" Oct 30 22:39:50.955755 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:50.955711 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs" Oct 30 22:39:51.274585 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:39:51.274546 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:39:51.351565 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:51.351416 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs"] Oct 30 22:39:51.353864 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:39:51.353828 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f72d065_2264_40ce_9682_e02ab4a7a942.slice/crio-8310f804d0e742bb485e832d294d25dda8be82d5b47ab8b628f43a6379f202d1 WatchSource:0}: Error finding container 8310f804d0e742bb485e832d294d25dda8be82d5b47ab8b628f43a6379f202d1: Status 404 returned error can't find the container with id 8310f804d0e742bb485e832d294d25dda8be82d5b47ab8b628f43a6379f202d1 Oct 30 22:39:52.085439 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:52.085403 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs" event={"ID":"2f72d065-2264-40ce-9682-e02ab4a7a942","Type":"ContainerStarted","Data":"d672882690109528afc7a20f655a030b7e43f92da2e8229412bab825a6fca77c"} Oct 30 22:39:52.085439 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:52.085448 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs" event={"ID":"2f72d065-2264-40ce-9682-e02ab4a7a942","Type":"ContainerStarted","Data":"8310f804d0e742bb485e832d294d25dda8be82d5b47ab8b628f43a6379f202d1"} Oct 30 22:39:52.210484 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:52.210430 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs" podStartSLOduration=2.2104169909999998 podStartE2EDuration="2.210416991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:39:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:39:52.209060223 +0000 UTC m=+2952.479758308" watchObservedRunningTime="2026-10-30 22:39:52.210416991 +0000 UTC m=+2952.481115074" Oct 30 22:39:52.956317 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:39:52.956281 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs" Oct 30 22:40:00.956273 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:40:00.956244 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs" Oct 30 22:40:00.968892 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:40:00.968866 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs" Oct 30 22:40:01.142033 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:40:01.141991 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs" Oct 30 22:40:21.274831 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:40:21.274736 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:40:51.275096 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:40:51.275048 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:40:52.005049 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:40:52.005011 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs"] Oct 30 22:41:07.026143 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:41:07.026093 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs" podUID="2f72d065-2264-40ce-9682-e02ab4a7a942" containerName="main" containerID="cri-o://d672882690109528afc7a20f655a030b7e43f92da2e8229412bab825a6fca77c" gracePeriod=45 Oct 30 22:41:07.131102 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:41:07.131064 5918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs" podUID="2f72d065-2264-40ce-9682-e02ab4a7a942" containerName="main" probeResult="failure" output="Get \"https://10.134.0.115:8000/ready\": dial tcp 10.134.0.115:8000: connect: connection refused" Oct 30 22:41:07.283713 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:41:07.283653 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs" Oct 30 22:41:07.431671 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:41:07.431634 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f72d065-2264-40ce-9682-e02ab4a7a942-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f72d065-2264-40ce-9682-e02ab4a7a942-model-cache\") pod \"2f72d065-2264-40ce-9682-e02ab4a7a942\" (UID: \"2f72d065-2264-40ce-9682-e02ab4a7a942\") " Oct 30 22:41:07.431867 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:41:07.431712 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f72d065-2264-40ce-9682-e02ab4a7a942-home\" (UniqueName: \"kubernetes.io/empty-dir/2f72d065-2264-40ce-9682-e02ab4a7a942-home\") pod \"2f72d065-2264-40ce-9682-e02ab4a7a942\" (UID: \"2f72d065-2264-40ce-9682-e02ab4a7a942\") " Oct 30 22:41:07.431867 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:41:07.431747 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f72d065-2264-40ce-9682-e02ab4a7a942-kube-api-access-chd4r\" (UniqueName: \"kubernetes.io/projected/2f72d065-2264-40ce-9682-e02ab4a7a942-kube-api-access-chd4r\") pod \"2f72d065-2264-40ce-9682-e02ab4a7a942\" (UID: \"2f72d065-2264-40ce-9682-e02ab4a7a942\") " Oct 30 22:41:07.431867 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:41:07.431775 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f72d065-2264-40ce-9682-e02ab4a7a942-tls-certs\" (UniqueName: \"kubernetes.io/secret/2f72d065-2264-40ce-9682-e02ab4a7a942-tls-certs\") pod \"2f72d065-2264-40ce-9682-e02ab4a7a942\" (UID: \"2f72d065-2264-40ce-9682-e02ab4a7a942\") " Oct 30 22:41:07.431867 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:41:07.431842 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f72d065-2264-40ce-9682-e02ab4a7a942-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f72d065-2264-40ce-9682-e02ab4a7a942-dshm\") pod \"2f72d065-2264-40ce-9682-e02ab4a7a942\" (UID: \"2f72d065-2264-40ce-9682-e02ab4a7a942\") " Oct 30 22:41:07.432086 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:41:07.431940 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f72d065-2264-40ce-9682-e02ab4a7a942-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f72d065-2264-40ce-9682-e02ab4a7a942-tmp-dir\") pod \"2f72d065-2264-40ce-9682-e02ab4a7a942\" (UID: \"2f72d065-2264-40ce-9682-e02ab4a7a942\") " Oct 30 22:41:07.432086 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:41:07.431989 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f72d065-2264-40ce-9682-e02ab4a7a942-home" pod "2f72d065-2264-40ce-9682-e02ab4a7a942" (UID: "2f72d065-2264-40ce-9682-e02ab4a7a942"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:07.432086 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:41:07.432016 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f72d065-2264-40ce-9682-e02ab4a7a942-model-cache" pod "2f72d065-2264-40ce-9682-e02ab4a7a942" (UID: "2f72d065-2264-40ce-9682-e02ab4a7a942"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:07.432302 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:41:07.432285 5918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f72d065-2264-40ce-9682-e02ab4a7a942-model-cache\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:07.432362 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:41:07.432305 5918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f72d065-2264-40ce-9682-e02ab4a7a942-home\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:07.432362 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:41:07.432307 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f72d065-2264-40ce-9682-e02ab4a7a942-tmp-dir" pod "2f72d065-2264-40ce-9682-e02ab4a7a942" (UID: "2f72d065-2264-40ce-9682-e02ab4a7a942"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:07.434712 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:41:07.434684 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f72d065-2264-40ce-9682-e02ab4a7a942-kube-api-access-chd4r" pod "2f72d065-2264-40ce-9682-e02ab4a7a942" (UID: "2f72d065-2264-40ce-9682-e02ab4a7a942"). InnerVolumeSpecName "kube-api-access-chd4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:07.434850 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:41:07.434713 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f72d065-2264-40ce-9682-e02ab4a7a942-dshm" pod "2f72d065-2264-40ce-9682-e02ab4a7a942" (UID: "2f72d065-2264-40ce-9682-e02ab4a7a942"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:07.434850 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:41:07.434715 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f72d065-2264-40ce-9682-e02ab4a7a942-tls-certs" pod "2f72d065-2264-40ce-9682-e02ab4a7a942" (UID: "2f72d065-2264-40ce-9682-e02ab4a7a942"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:41:07.464521 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:41:07.464493 5918 generic.go:356] "Generic (PLEG): container finished" podID="2f72d065-2264-40ce-9682-e02ab4a7a942" containerID="d672882690109528afc7a20f655a030b7e43f92da2e8229412bab825a6fca77c" exitCode=0 Oct 30 22:41:07.464660 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:41:07.464541 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs" Oct 30 22:41:07.464660 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:41:07.464581 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs" event={"ID":"2f72d065-2264-40ce-9682-e02ab4a7a942","Type":"ContainerDied","Data":"d672882690109528afc7a20f655a030b7e43f92da2e8229412bab825a6fca77c"} Oct 30 22:41:07.464660 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:41:07.464627 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs" event={"ID":"2f72d065-2264-40ce-9682-e02ab4a7a942","Type":"ContainerDied","Data":"8310f804d0e742bb485e832d294d25dda8be82d5b47ab8b628f43a6379f202d1"} Oct 30 22:41:07.464660 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:41:07.464649 5918 scope.go:122] "RemoveContainer" containerID="d672882690109528afc7a20f655a030b7e43f92da2e8229412bab825a6fca77c" Oct 30 22:41:07.474137 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:41:07.474121 5918 scope.go:122] "RemoveContainer" containerID="d672882690109528afc7a20f655a030b7e43f92da2e8229412bab825a6fca77c" Oct 30 22:41:07.474466 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:41:07.474445 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d672882690109528afc7a20f655a030b7e43f92da2e8229412bab825a6fca77c\": container with ID starting with d672882690109528afc7a20f655a030b7e43f92da2e8229412bab825a6fca77c not found: ID does not exist" containerID="d672882690109528afc7a20f655a030b7e43f92da2e8229412bab825a6fca77c" Oct 30 22:41:07.474553 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:41:07.474478 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d672882690109528afc7a20f655a030b7e43f92da2e8229412bab825a6fca77c"} err="failed to get container status \"d672882690109528afc7a20f655a030b7e43f92da2e8229412bab825a6fca77c\": rpc error: code = NotFound desc = could not find container \"d672882690109528afc7a20f655a030b7e43f92da2e8229412bab825a6fca77c\": container with ID starting with d672882690109528afc7a20f655a030b7e43f92da2e8229412bab825a6fca77c not found: ID does not exist" Oct 30 22:41:07.533655 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:41:07.533628 5918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f72d065-2264-40ce-9682-e02ab4a7a942-dshm\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:07.533655 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:41:07.533653 5918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f72d065-2264-40ce-9682-e02ab4a7a942-tmp-dir\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:07.533655 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:41:07.533663 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chd4r\" (UniqueName: \"kubernetes.io/projected/2f72d065-2264-40ce-9682-e02ab4a7a942-kube-api-access-chd4r\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:07.533891 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:41:07.533672 5918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f72d065-2264-40ce-9682-e02ab4a7a942-tls-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:07.632822 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:41:07.632788 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs"] Oct 30 22:41:07.639082 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:41:07.639057 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-ca3b17/simulator-single-main-ca3b17-kserve-59c959669f-9fgbs"] Oct 30 22:41:07.850514 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:41:07.850438 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f72d065-2264-40ce-9682-e02ab4a7a942" path="/var/lib/kubelet/pods/2f72d065-2264-40ce-9682-e02ab4a7a942/volumes" Oct 30 22:41:21.274861 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:41:21.274825 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:41:51.274663 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:41:51.274570 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:42:21.274790 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:42:21.274749 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:42:51.275853 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:42:51.275810 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:43:21.275549 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:43:21.275457 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:43:51.273632 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:43:51.273592 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:44:21.276054 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:44:21.276015 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:44:33.572027 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:33.571992 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-ef21ee/simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs"] Oct 30 22:44:33.572518 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:33.572414 5918 cpu_manager.go:414] "containerMap: removing container" podUID="2f72d065-2264-40ce-9682-e02ab4a7a942" containerName="main" Oct 30 22:44:33.572518 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:33.572425 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f72d065-2264-40ce-9682-e02ab4a7a942" containerName="main" Oct 30 22:44:33.572518 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:33.572441 5918 cpu_manager.go:414] "containerMap: removing container" podUID="f6f37484-37b2-4fab-8b6b-cdc1c171ff93" containerName="istio-proxy" Oct 30 22:44:33.572518 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:33.572446 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6f37484-37b2-4fab-8b6b-cdc1c171ff93" containerName="istio-proxy" Oct 30 22:44:33.572518 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:33.572507 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5345433-337f-4434-a218-a9880269ddc5" containerName="istio-proxy" Oct 30 22:44:33.572518 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:33.572518 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f72d065-2264-40ce-9682-e02ab4a7a942" containerName="main" Oct 30 22:44:33.575718 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:33.575701 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ef21ee/simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs" Oct 30 22:44:33.630137 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:33.630098 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-ef21ee\"/\"openshift-service-ca.crt\"" Oct 30 22:44:33.679894 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:33.679861 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-ef21ee\"/\"default-dockercfg-ctfpf\"" Oct 30 22:44:33.680084 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:33.679911 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-ef21ee\"/\"kube-root-ca.crt\"" Oct 30 22:44:33.680084 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:33.679934 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-ef21ee\"/\"simulator-single-main-ef21ee-kserve-self-signed-certs\"" Oct 30 22:44:33.682077 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:33.682056 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-ef21ee/simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs"] Oct 30 22:44:33.741731 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:33.741696 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm4fj\" (UniqueName: \"kubernetes.io/projected/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-kube-api-access-dm4fj\") pod \"simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs\" (UID: \"1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8\") " pod="e2e-models-e2e-worker-single-main-ef21ee/simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs" Oct 30 22:44:33.741908 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:33.741750 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-tmp-dir\") pod \"simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs\" (UID: \"1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8\") " pod="e2e-models-e2e-worker-single-main-ef21ee/simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs" Oct 30 22:44:33.741908 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:33.741864 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-dshm\") pod \"simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs\" (UID: \"1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8\") " pod="e2e-models-e2e-worker-single-main-ef21ee/simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs" Oct 30 22:44:33.741908 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:33.741891 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-model-cache\") pod \"simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs\" (UID: \"1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8\") " pod="e2e-models-e2e-worker-single-main-ef21ee/simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs" Oct 30 22:44:33.742039 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:33.741915 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-tls-certs\") pod \"simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs\" (UID: \"1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8\") " pod="e2e-models-e2e-worker-single-main-ef21ee/simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs" Oct 30 22:44:33.742039 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:33.741973 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-home\") pod \"simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs\" (UID: \"1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8\") " pod="e2e-models-e2e-worker-single-main-ef21ee/simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs" Oct 30 22:44:33.843316 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:33.843210 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-tmp-dir\") pod \"simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs\" (UID: \"1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8\") " pod="e2e-models-e2e-worker-single-main-ef21ee/simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs" Oct 30 22:44:33.843501 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:33.843315 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-dshm\") pod \"simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs\" (UID: \"1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8\") " pod="e2e-models-e2e-worker-single-main-ef21ee/simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs" Oct 30 22:44:33.843501 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:33.843343 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-model-cache\") pod \"simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs\" (UID: \"1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8\") " pod="e2e-models-e2e-worker-single-main-ef21ee/simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs" Oct 30 22:44:33.843501 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:33.843377 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-tls-certs\") pod \"simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs\" (UID: \"1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8\") " pod="e2e-models-e2e-worker-single-main-ef21ee/simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs" Oct 30 22:44:33.843501 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:33.843412 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-home\") pod \"simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs\" (UID: \"1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8\") " pod="e2e-models-e2e-worker-single-main-ef21ee/simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs" Oct 30 22:44:33.843501 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:33.843493 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm4fj\" (UniqueName: \"kubernetes.io/projected/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-kube-api-access-dm4fj\") pod \"simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs\" (UID: \"1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8\") " pod="e2e-models-e2e-worker-single-main-ef21ee/simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs" Oct 30 22:44:33.843853 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:33.843716 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-tmp-dir\") pod \"simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs\" (UID: \"1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8\") " pod="e2e-models-e2e-worker-single-main-ef21ee/simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs" Oct 30 22:44:33.843853 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:33.843799 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-model-cache\") pod \"simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs\" (UID: \"1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8\") " pod="e2e-models-e2e-worker-single-main-ef21ee/simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs" Oct 30 22:44:33.843988 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:33.843968 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-home\") pod \"simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs\" (UID: \"1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8\") " pod="e2e-models-e2e-worker-single-main-ef21ee/simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs" Oct 30 22:44:33.846484 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:33.846449 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-dshm\") pod \"simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs\" (UID: \"1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8\") " pod="e2e-models-e2e-worker-single-main-ef21ee/simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs" Oct 30 22:44:33.846621 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:33.846602 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-tls-certs\") pod \"simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs\" (UID: \"1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8\") " pod="e2e-models-e2e-worker-single-main-ef21ee/simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs" Oct 30 22:44:33.906772 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:33.906737 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm4fj\" (UniqueName: \"kubernetes.io/projected/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-kube-api-access-dm4fj\") pod \"simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs\" (UID: \"1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8\") " pod="e2e-models-e2e-worker-single-main-ef21ee/simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs" Oct 30 22:44:34.187199 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:34.187165 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ef21ee/simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs" Oct 30 22:44:34.567298 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:34.567276 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-ef21ee/simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs"] Oct 30 22:44:34.569030 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:44:34.569008 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a6d4b86_dadb_4ed4_a3a7_6f39e4c5e0b8.slice/crio-cb2c946e812509c7cf36f22980f4609d29b4afbb9db99690316f4f3bae41c48a WatchSource:0}: Error finding container cb2c946e812509c7cf36f22980f4609d29b4afbb9db99690316f4f3bae41c48a: Status 404 returned error can't find the container with id cb2c946e812509c7cf36f22980f4609d29b4afbb9db99690316f4f3bae41c48a Oct 30 22:44:35.541149 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:35.541113 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ef21ee/simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs" event={"ID":"1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8","Type":"ContainerStarted","Data":"b09745e620249e900ca11828405f0c42d3bc56d5537075db301f2928b9187fb1"} Oct 30 22:44:35.541149 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:35.541153 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ef21ee/simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs" event={"ID":"1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8","Type":"ContainerStarted","Data":"cb2c946e812509c7cf36f22980f4609d29b4afbb9db99690316f4f3bae41c48a"} Oct 30 22:44:35.662286 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:35.662200 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-ef21ee/simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs" podStartSLOduration=2.662179186 podStartE2EDuration="2.662179186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:44:35.661873385 +0000 UTC m=+3235.932571470" watchObservedRunningTime="2026-10-30 22:44:35.662179186 +0000 UTC m=+3235.932877275" Oct 30 22:44:36.188197 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:36.188166 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-ef21ee/simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs" Oct 30 22:44:44.188062 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:44.188030 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-ef21ee/simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs" Oct 30 22:44:44.200812 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:44.200784 5918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-ef21ee/simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs" Oct 30 22:44:44.594448 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:44:44.594355 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-ef21ee/simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs" Oct 30 22:44:51.274497 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:44:51.274409 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:45:21.274711 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:45:21.274665 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:45:34.746643 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:45:34.746608 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-ef21ee/simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs"] Oct 30 22:45:49.767189 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:45:49.767119 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-ef21ee/simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs" podUID="1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8" containerName="main" containerID="cri-o://b09745e620249e900ca11828405f0c42d3bc56d5537075db301f2928b9187fb1" gracePeriod=45 Oct 30 22:45:49.906105 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:45:49.906072 5918 generic.go:356] "Generic (PLEG): container finished" podID="1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8" containerID="b09745e620249e900ca11828405f0c42d3bc56d5537075db301f2928b9187fb1" exitCode=0 Oct 30 22:45:49.906286 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:45:49.906135 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ef21ee/simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs" event={"ID":"1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8","Type":"ContainerDied","Data":"b09745e620249e900ca11828405f0c42d3bc56d5537075db301f2928b9187fb1"} Oct 30 22:45:50.018997 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:45:50.018927 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ef21ee/simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs" Oct 30 22:45:50.101790 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:45:50.101761 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-kube-api-access-dm4fj\" (UniqueName: \"kubernetes.io/projected/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-kube-api-access-dm4fj\") pod \"1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8\" (UID: \"1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8\") " Oct 30 22:45:50.101965 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:45:50.101812 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-tmp-dir\") pod \"1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8\" (UID: \"1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8\") " Oct 30 22:45:50.101965 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:45:50.101843 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-model-cache\") pod \"1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8\" (UID: \"1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8\") " Oct 30 22:45:50.101965 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:45:50.101926 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-home\" (UniqueName: \"kubernetes.io/empty-dir/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-home\") pod \"1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8\" (UID: \"1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8\") " Oct 30 22:45:50.102146 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:45:50.102001 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-dshm\") pod \"1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8\" (UID: \"1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8\") " Oct 30 22:45:50.102146 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:45:50.102031 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-tls-certs\" (UniqueName: \"kubernetes.io/secret/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-tls-certs\") pod \"1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8\" (UID: \"1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8\") " Oct 30 22:45:50.102146 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:45:50.102097 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-tmp-dir" pod "1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8" (UID: "1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:50.102331 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:45:50.102180 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-model-cache" pod "1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8" (UID: "1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:50.102331 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:45:50.102252 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-home" pod "1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8" (UID: "1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:50.102526 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:45:50.102479 5918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-home\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:45:50.102526 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:45:50.102504 5918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-tmp-dir\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:45:50.102526 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:45:50.102519 5918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-model-cache\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:45:50.104317 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:45:50.104292 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-dshm" pod "1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8" (UID: "1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:50.104408 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:45:50.104355 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-kube-api-access-dm4fj" pod "1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8" (UID: "1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8"). InnerVolumeSpecName "kube-api-access-dm4fj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:50.104408 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:45:50.104396 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-tls-certs" pod "1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8" (UID: "1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:45:50.203612 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:45:50.203578 5918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-dshm\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:45:50.203612 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:45:50.203612 5918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-tls-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:45:50.203810 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:45:50.203628 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm4fj\" (UniqueName: \"kubernetes.io/projected/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8-kube-api-access-dm4fj\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:45:50.912885 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:45:50.912853 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ef21ee/simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs" Oct 30 22:45:50.913348 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:45:50.912894 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ef21ee/simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs" event={"ID":"1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8","Type":"ContainerDied","Data":"cb2c946e812509c7cf36f22980f4609d29b4afbb9db99690316f4f3bae41c48a"} Oct 30 22:45:50.913348 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:45:50.912957 5918 scope.go:122] "RemoveContainer" containerID="b09745e620249e900ca11828405f0c42d3bc56d5537075db301f2928b9187fb1" Oct 30 22:45:51.082850 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:45:51.082815 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-ef21ee/simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs"] Oct 30 22:45:51.133108 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:45:51.133078 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-ef21ee/simulator-single-main-ef21ee-kserve-8545bd6bd6-94vhs"] Oct 30 22:45:51.274247 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:45:51.274180 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:45:51.850923 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:45:51.850888 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8" path="/var/lib/kubelet/pods/1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8/volumes" Oct 30 22:46:21.276916 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:46:21.276828 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:46:51.279746 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:46:51.279695 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:47:21.285538 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:47:21.285501 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:47:51.274783 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:47:51.274703 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:48:21.273794 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:48:21.273756 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:48:33.704020 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:33.703979 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-857794cb89-d7nl2"] Oct 30 22:48:33.704588 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:33.704457 5918 cpu_manager.go:414] "containerMap: removing container" podUID="1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8" containerName="main" Oct 30 22:48:33.704588 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:33.704469 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8" containerName="main" Oct 30 22:48:33.704588 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:33.704543 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a6d4b86-dadb-4ed4-a3a7-6f39e4c5e0b8" containerName="main" Oct 30 22:48:33.707885 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:33.707867 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-857794cb89-d7nl2" Oct 30 22:48:33.761405 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:33.761375 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzrks\" (UniqueName: \"kubernetes.io/projected/38825407-9c99-46f6-92d0-2fe72f297bd3-kube-api-access-pzrks\") pod \"maas-api-857794cb89-d7nl2\" (UID: \"38825407-9c99-46f6-92d0-2fe72f297bd3\") " pod="odh-ai-gateway-infra/maas-api-857794cb89-d7nl2" Oct 30 22:48:33.761568 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:33.761423 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38825407-9c99-46f6-92d0-2fe72f297bd3-metrics-certs\") pod \"maas-api-857794cb89-d7nl2\" (UID: \"38825407-9c99-46f6-92d0-2fe72f297bd3\") " pod="odh-ai-gateway-infra/maas-api-857794cb89-d7nl2" Oct 30 22:48:33.761633 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:33.761613 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38825407-9c99-46f6-92d0-2fe72f297bd3-maas-api-tls\") pod \"maas-api-857794cb89-d7nl2\" (UID: \"38825407-9c99-46f6-92d0-2fe72f297bd3\") " pod="odh-ai-gateway-infra/maas-api-857794cb89-d7nl2" Oct 30 22:48:33.761688 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:33.761642 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38825407-9c99-46f6-92d0-2fe72f297bd3-openshift-service-ca\") pod \"maas-api-857794cb89-d7nl2\" (UID: \"38825407-9c99-46f6-92d0-2fe72f297bd3\") " pod="odh-ai-gateway-infra/maas-api-857794cb89-d7nl2" Oct 30 22:48:33.815880 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:33.815515 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-857794cb89-d7nl2"] Oct 30 22:48:33.862752 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:33.862721 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38825407-9c99-46f6-92d0-2fe72f297bd3-maas-api-tls\") pod \"maas-api-857794cb89-d7nl2\" (UID: \"38825407-9c99-46f6-92d0-2fe72f297bd3\") " pod="odh-ai-gateway-infra/maas-api-857794cb89-d7nl2" Oct 30 22:48:33.862752 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:33.862754 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38825407-9c99-46f6-92d0-2fe72f297bd3-openshift-service-ca\") pod \"maas-api-857794cb89-d7nl2\" (UID: \"38825407-9c99-46f6-92d0-2fe72f297bd3\") " pod="odh-ai-gateway-infra/maas-api-857794cb89-d7nl2" Oct 30 22:48:33.862990 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:33.862888 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzrks\" (UniqueName: \"kubernetes.io/projected/38825407-9c99-46f6-92d0-2fe72f297bd3-kube-api-access-pzrks\") pod \"maas-api-857794cb89-d7nl2\" (UID: \"38825407-9c99-46f6-92d0-2fe72f297bd3\") " pod="odh-ai-gateway-infra/maas-api-857794cb89-d7nl2" Oct 30 22:48:33.863054 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:33.863025 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38825407-9c99-46f6-92d0-2fe72f297bd3-metrics-certs\") pod \"maas-api-857794cb89-d7nl2\" (UID: \"38825407-9c99-46f6-92d0-2fe72f297bd3\") " pod="odh-ai-gateway-infra/maas-api-857794cb89-d7nl2" Oct 30 22:48:33.863534 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:33.863512 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38825407-9c99-46f6-92d0-2fe72f297bd3-openshift-service-ca\") pod \"maas-api-857794cb89-d7nl2\" (UID: \"38825407-9c99-46f6-92d0-2fe72f297bd3\") " pod="odh-ai-gateway-infra/maas-api-857794cb89-d7nl2" Oct 30 22:48:33.866064 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:33.866043 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38825407-9c99-46f6-92d0-2fe72f297bd3-maas-api-tls\") pod \"maas-api-857794cb89-d7nl2\" (UID: \"38825407-9c99-46f6-92d0-2fe72f297bd3\") " pod="odh-ai-gateway-infra/maas-api-857794cb89-d7nl2" Oct 30 22:48:33.866191 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:33.866169 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38825407-9c99-46f6-92d0-2fe72f297bd3-metrics-certs\") pod \"maas-api-857794cb89-d7nl2\" (UID: \"38825407-9c99-46f6-92d0-2fe72f297bd3\") " pod="odh-ai-gateway-infra/maas-api-857794cb89-d7nl2" Oct 30 22:48:33.926936 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:33.926909 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzrks\" (UniqueName: \"kubernetes.io/projected/38825407-9c99-46f6-92d0-2fe72f297bd3-kube-api-access-pzrks\") pod \"maas-api-857794cb89-d7nl2\" (UID: \"38825407-9c99-46f6-92d0-2fe72f297bd3\") " pod="odh-ai-gateway-infra/maas-api-857794cb89-d7nl2" Oct 30 22:48:34.021367 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:34.021271 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-857794cb89-d7nl2" Oct 30 22:48:34.401657 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:34.401631 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-857794cb89-d7nl2"] Oct 30 22:48:34.403860 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:48:34.403832 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38825407_9c99_46f6_92d0_2fe72f297bd3.slice/crio-99a53896e5a4fe72438d5c0b8e57610780bc7eaa8158a88abffcd0451f1777d0 WatchSource:0}: Error finding container 99a53896e5a4fe72438d5c0b8e57610780bc7eaa8158a88abffcd0451f1777d0: Status 404 returned error can't find the container with id 99a53896e5a4fe72438d5c0b8e57610780bc7eaa8158a88abffcd0451f1777d0 Oct 30 22:48:34.405191 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:34.405172 5918 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:48:34.853444 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:34.853410 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-857794cb89-d7nl2" event={"ID":"38825407-9c99-46f6-92d0-2fe72f297bd3","Type":"ContainerStarted","Data":"99a53896e5a4fe72438d5c0b8e57610780bc7eaa8158a88abffcd0451f1777d0"} Oct 30 22:48:35.859652 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:35.859619 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-857794cb89-d7nl2" event={"ID":"38825407-9c99-46f6-92d0-2fe72f297bd3","Type":"ContainerStarted","Data":"260b1045acd0092e9ee5f58e4709f5990673d09123035973cacd1a595bcf04c1"} Oct 30 22:48:35.860058 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:35.859742 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-857794cb89-d7nl2" Oct 30 22:48:35.982433 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:35.982381 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-857794cb89-d7nl2" podStartSLOduration=2.171693757 podStartE2EDuration="2.982367062s" totalImagesPullingTime="810.673305ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:48:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:48:35.981250356 +0000 UTC m=+3476.251948438" watchObservedRunningTime="2026-10-30 22:48:35.982367062 +0000 UTC m=+3476.253065146" Oct 30 22:48:41.869975 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:41.869946 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-857794cb89-d7nl2" Oct 30 22:48:42.054085 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:42.054054 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-57fcf7d945-4fsw9"] Oct 30 22:48:42.054275 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:42.054255 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-57fcf7d945-4fsw9" podUID="96d23985-308b-44bd-928f-0563b9d19ff9" containerName="maas-api" containerID="cri-o://888776de194d5e4b481f34f750f0396538cd734770bfb627cb06ab9e062c9489" gracePeriod=30 Oct 30 22:48:43.296244 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:43.296189 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57fcf7d945-4fsw9" Oct 30 22:48:43.360541 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:43.360500 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96d23985-308b-44bd-928f-0563b9d19ff9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/96d23985-308b-44bd-928f-0563b9d19ff9-metrics-certs\") pod \"96d23985-308b-44bd-928f-0563b9d19ff9\" (UID: \"96d23985-308b-44bd-928f-0563b9d19ff9\") " Oct 30 22:48:43.360719 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:43.360660 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96d23985-308b-44bd-928f-0563b9d19ff9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96d23985-308b-44bd-928f-0563b9d19ff9-maas-api-tls\") pod \"96d23985-308b-44bd-928f-0563b9d19ff9\" (UID: \"96d23985-308b-44bd-928f-0563b9d19ff9\") " Oct 30 22:48:43.360773 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:43.360751 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96d23985-308b-44bd-928f-0563b9d19ff9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d23985-308b-44bd-928f-0563b9d19ff9-openshift-service-ca\") pod \"96d23985-308b-44bd-928f-0563b9d19ff9\" (UID: \"96d23985-308b-44bd-928f-0563b9d19ff9\") " Oct 30 22:48:43.360835 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:43.360797 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96d23985-308b-44bd-928f-0563b9d19ff9-kube-api-access-t95bf\" (UniqueName: \"kubernetes.io/projected/96d23985-308b-44bd-928f-0563b9d19ff9-kube-api-access-t95bf\") pod \"96d23985-308b-44bd-928f-0563b9d19ff9\" (UID: \"96d23985-308b-44bd-928f-0563b9d19ff9\") " Oct 30 22:48:43.361135 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:43.361106 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96d23985-308b-44bd-928f-0563b9d19ff9-openshift-service-ca" pod "96d23985-308b-44bd-928f-0563b9d19ff9" (UID: "96d23985-308b-44bd-928f-0563b9d19ff9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:48:43.362834 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:43.362803 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96d23985-308b-44bd-928f-0563b9d19ff9-metrics-certs" pod "96d23985-308b-44bd-928f-0563b9d19ff9" (UID: "96d23985-308b-44bd-928f-0563b9d19ff9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:48:43.362953 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:43.362911 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96d23985-308b-44bd-928f-0563b9d19ff9-maas-api-tls" pod "96d23985-308b-44bd-928f-0563b9d19ff9" (UID: "96d23985-308b-44bd-928f-0563b9d19ff9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:48:43.363350 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:43.363322 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96d23985-308b-44bd-928f-0563b9d19ff9-kube-api-access-t95bf" pod "96d23985-308b-44bd-928f-0563b9d19ff9" (UID: "96d23985-308b-44bd-928f-0563b9d19ff9"). InnerVolumeSpecName "kube-api-access-t95bf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:48:43.462115 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:43.462077 5918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96d23985-308b-44bd-928f-0563b9d19ff9-maas-api-tls\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:48:43.462115 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:43.462110 5918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d23985-308b-44bd-928f-0563b9d19ff9-openshift-service-ca\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:48:43.462115 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:43.462120 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t95bf\" (UniqueName: \"kubernetes.io/projected/96d23985-308b-44bd-928f-0563b9d19ff9-kube-api-access-t95bf\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:48:43.462405 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:43.462130 5918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96d23985-308b-44bd-928f-0563b9d19ff9-metrics-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:48:43.904899 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:43.904865 5918 generic.go:356] "Generic (PLEG): container finished" podID="96d23985-308b-44bd-928f-0563b9d19ff9" containerID="888776de194d5e4b481f34f750f0396538cd734770bfb627cb06ab9e062c9489" exitCode=0 Oct 30 22:48:43.905080 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:43.904912 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57fcf7d945-4fsw9" event={"ID":"96d23985-308b-44bd-928f-0563b9d19ff9","Type":"ContainerDied","Data":"888776de194d5e4b481f34f750f0396538cd734770bfb627cb06ab9e062c9489"} Oct 30 22:48:43.905080 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:43.904921 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57fcf7d945-4fsw9" Oct 30 22:48:43.905080 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:43.904948 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57fcf7d945-4fsw9" event={"ID":"96d23985-308b-44bd-928f-0563b9d19ff9","Type":"ContainerDied","Data":"ac650c0d135b44bc3b94dd42f59b624744bacf79262d40992a9702b0f297e9d3"} Oct 30 22:48:43.905080 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:43.904969 5918 scope.go:122] "RemoveContainer" containerID="888776de194d5e4b481f34f750f0396538cd734770bfb627cb06ab9e062c9489" Oct 30 22:48:43.913933 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:43.913913 5918 scope.go:122] "RemoveContainer" containerID="888776de194d5e4b481f34f750f0396538cd734770bfb627cb06ab9e062c9489" Oct 30 22:48:43.914253 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:48:43.914203 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"888776de194d5e4b481f34f750f0396538cd734770bfb627cb06ab9e062c9489\": container with ID starting with 888776de194d5e4b481f34f750f0396538cd734770bfb627cb06ab9e062c9489 not found: ID does not exist" containerID="888776de194d5e4b481f34f750f0396538cd734770bfb627cb06ab9e062c9489" Oct 30 22:48:43.914340 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:43.914268 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"888776de194d5e4b481f34f750f0396538cd734770bfb627cb06ab9e062c9489"} err="failed to get container status \"888776de194d5e4b481f34f750f0396538cd734770bfb627cb06ab9e062c9489\": rpc error: code = NotFound desc = could not find container \"888776de194d5e4b481f34f750f0396538cd734770bfb627cb06ab9e062c9489\": container with ID starting with 888776de194d5e4b481f34f750f0396538cd734770bfb627cb06ab9e062c9489 not found: ID does not exist" Oct 30 22:48:44.074007 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:44.073976 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-57fcf7d945-4fsw9"] Oct 30 22:48:44.079741 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:44.079717 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-57fcf7d945-4fsw9"] Oct 30 22:48:45.850585 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:48:45.850548 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96d23985-308b-44bd-928f-0563b9d19ff9" path="/var/lib/kubelet/pods/96d23985-308b-44bd-928f-0563b9d19ff9/volumes" Oct 30 22:48:51.274272 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:48:51.274237 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:49:21.274497 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:49:21.274419 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:49:51.274606 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:49:51.274571 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:49:52.472445 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:49:52.472410 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b"] Oct 30 22:49:52.473040 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:49:52.472857 5918 cpu_manager.go:414] "containerMap: removing container" podUID="96d23985-308b-44bd-928f-0563b9d19ff9" containerName="maas-api" Oct 30 22:49:52.473040 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:49:52.472869 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96d23985-308b-44bd-928f-0563b9d19ff9" containerName="maas-api" Oct 30 22:49:52.473040 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:49:52.472976 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="96d23985-308b-44bd-928f-0563b9d19ff9" containerName="maas-api" Oct 30 22:49:52.476137 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:49:52.476122 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b" Oct 30 22:49:52.530501 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:49:52.530478 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-40b9a8-serving-cert\"" Oct 30 22:49:52.537789 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:49:52.537759 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df612bf7-b12b-426b-aaaf-7ab74cf3452d-metrics-certs\") pod \"maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b\" (UID: \"df612bf7-b12b-426b-aaaf-7ab74cf3452d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b" Oct 30 22:49:52.537934 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:49:52.537815 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jc8z\" (UniqueName: \"kubernetes.io/projected/df612bf7-b12b-426b-aaaf-7ab74cf3452d-kube-api-access-4jc8z\") pod \"maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b\" (UID: \"df612bf7-b12b-426b-aaaf-7ab74cf3452d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b" Oct 30 22:49:52.537999 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:49:52.537929 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df612bf7-b12b-426b-aaaf-7ab74cf3452d-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b\" (UID: \"df612bf7-b12b-426b-aaaf-7ab74cf3452d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b" Oct 30 22:49:52.538038 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:49:52.538010 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df612bf7-b12b-426b-aaaf-7ab74cf3452d-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b\" (UID: \"df612bf7-b12b-426b-aaaf-7ab74cf3452d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b" Oct 30 22:49:52.582890 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:49:52.582853 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b"] Oct 30 22:49:52.639536 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:49:52.639502 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df612bf7-b12b-426b-aaaf-7ab74cf3452d-metrics-certs\") pod \"maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b\" (UID: \"df612bf7-b12b-426b-aaaf-7ab74cf3452d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b" Oct 30 22:49:52.639740 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:49:52.639555 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jc8z\" (UniqueName: \"kubernetes.io/projected/df612bf7-b12b-426b-aaaf-7ab74cf3452d-kube-api-access-4jc8z\") pod \"maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b\" (UID: \"df612bf7-b12b-426b-aaaf-7ab74cf3452d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b" Oct 30 22:49:52.639740 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:49:52.639585 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df612bf7-b12b-426b-aaaf-7ab74cf3452d-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b\" (UID: \"df612bf7-b12b-426b-aaaf-7ab74cf3452d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b" Oct 30 22:49:52.639740 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:49:52.639615 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df612bf7-b12b-426b-aaaf-7ab74cf3452d-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b\" (UID: \"df612bf7-b12b-426b-aaaf-7ab74cf3452d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b" Oct 30 22:49:52.639912 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:49:52.639761 5918 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-authcfg-http-40b9a8-serving-cert: secret "maas-api-e2e-authcfg-http-40b9a8-serving-cert" not found Oct 30 22:49:52.639912 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:49:52.639825 5918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df612bf7-b12b-426b-aaaf-7ab74cf3452d-maas-api-tls podName:df612bf7-b12b-426b-aaaf-7ab74cf3452d nodeName:}" failed. No retries permitted until 2026-10-30 22:49:53.139803158 +0000 UTC m=+3553.410501222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/df612bf7-b12b-426b-aaaf-7ab74cf3452d-maas-api-tls") pod "maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b" (UID: "df612bf7-b12b-426b-aaaf-7ab74cf3452d") : secret "maas-api-e2e-authcfg-http-40b9a8-serving-cert" not found Oct 30 22:49:52.640302 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:49:52.640279 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df612bf7-b12b-426b-aaaf-7ab74cf3452d-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b\" (UID: \"df612bf7-b12b-426b-aaaf-7ab74cf3452d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b" Oct 30 22:49:52.642074 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:49:52.642049 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df612bf7-b12b-426b-aaaf-7ab74cf3452d-metrics-certs\") pod \"maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b\" (UID: \"df612bf7-b12b-426b-aaaf-7ab74cf3452d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b" Oct 30 22:49:52.705029 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:49:52.705002 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jc8z\" (UniqueName: \"kubernetes.io/projected/df612bf7-b12b-426b-aaaf-7ab74cf3452d-kube-api-access-4jc8z\") pod \"maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b\" (UID: \"df612bf7-b12b-426b-aaaf-7ab74cf3452d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b" Oct 30 22:49:53.144598 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:49:53.144556 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df612bf7-b12b-426b-aaaf-7ab74cf3452d-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b\" (UID: \"df612bf7-b12b-426b-aaaf-7ab74cf3452d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b" Oct 30 22:49:53.147145 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:49:53.147118 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df612bf7-b12b-426b-aaaf-7ab74cf3452d-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b\" (UID: \"df612bf7-b12b-426b-aaaf-7ab74cf3452d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b" Oct 30 22:49:53.389332 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:49:53.389304 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b" Oct 30 22:49:53.766598 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:49:53.766571 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b"] Oct 30 22:49:53.768677 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:49:53.768646 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf612bf7_b12b_426b_aaaf_7ab74cf3452d.slice/crio-0d462dd9c0a3c3597f7f5d3fb7b83647b8c4000a6372fe7648c29787f3253b05 WatchSource:0}: Error finding container 0d462dd9c0a3c3597f7f5d3fb7b83647b8c4000a6372fe7648c29787f3253b05: Status 404 returned error can't find the container with id 0d462dd9c0a3c3597f7f5d3fb7b83647b8c4000a6372fe7648c29787f3253b05 Oct 30 22:49:54.252014 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:49:54.251975 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b" event={"ID":"df612bf7-b12b-426b-aaaf-7ab74cf3452d","Type":"ContainerStarted","Data":"0d462dd9c0a3c3597f7f5d3fb7b83647b8c4000a6372fe7648c29787f3253b05"} Oct 30 22:49:55.265560 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:49:55.265513 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b" event={"ID":"df612bf7-b12b-426b-aaaf-7ab74cf3452d","Type":"ContainerStarted","Data":"46c0157459c91ba74192ac934015245d1da5a469b4481bdde247b7e631676270"} Oct 30 22:49:55.265988 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:49:55.265649 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b" Oct 30 22:49:55.387850 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:49:55.387799 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b" podStartSLOduration=2.600162285 podStartE2EDuration="3.387782957s" totalImagesPullingTime="787.620672ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:55.387312019 +0000 UTC m=+3555.658010107" watchObservedRunningTime="2026-10-30 22:49:55.387782957 +0000 UTC m=+3555.658481041" Oct 30 22:50:01.277039 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:50:01.277007 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b" Oct 30 22:50:21.274893 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:50:21.274852 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:50:51.274422 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:50:51.274335 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:51:21.275282 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:51:21.275237 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:51:49.041163 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:51:49.041086 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b"] Oct 30 22:51:49.041727 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:51:49.041396 5918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b" podUID="df612bf7-b12b-426b-aaaf-7ab74cf3452d" containerName="maas-api" containerID="cri-o://46c0157459c91ba74192ac934015245d1da5a469b4481bdde247b7e631676270" gracePeriod=30 Oct 30 22:51:49.306991 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:51:49.306934 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b" Oct 30 22:51:49.401176 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:51:49.401140 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df612bf7-b12b-426b-aaaf-7ab74cf3452d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df612bf7-b12b-426b-aaaf-7ab74cf3452d-maas-api-tls\") pod \"df612bf7-b12b-426b-aaaf-7ab74cf3452d\" (UID: \"df612bf7-b12b-426b-aaaf-7ab74cf3452d\") " Oct 30 22:51:49.401396 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:51:49.401185 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df612bf7-b12b-426b-aaaf-7ab74cf3452d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/df612bf7-b12b-426b-aaaf-7ab74cf3452d-metrics-certs\") pod \"df612bf7-b12b-426b-aaaf-7ab74cf3452d\" (UID: \"df612bf7-b12b-426b-aaaf-7ab74cf3452d\") " Oct 30 22:51:49.401396 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:51:49.401271 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df612bf7-b12b-426b-aaaf-7ab74cf3452d-kube-api-access-4jc8z\" (UniqueName: \"kubernetes.io/projected/df612bf7-b12b-426b-aaaf-7ab74cf3452d-kube-api-access-4jc8z\") pod \"df612bf7-b12b-426b-aaaf-7ab74cf3452d\" (UID: \"df612bf7-b12b-426b-aaaf-7ab74cf3452d\") " Oct 30 22:51:49.401396 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:51:49.401376 5918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df612bf7-b12b-426b-aaaf-7ab74cf3452d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df612bf7-b12b-426b-aaaf-7ab74cf3452d-openshift-service-ca\") pod \"df612bf7-b12b-426b-aaaf-7ab74cf3452d\" (UID: \"df612bf7-b12b-426b-aaaf-7ab74cf3452d\") " Oct 30 22:51:49.401810 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:51:49.401775 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df612bf7-b12b-426b-aaaf-7ab74cf3452d-openshift-service-ca" pod "df612bf7-b12b-426b-aaaf-7ab74cf3452d" (UID: "df612bf7-b12b-426b-aaaf-7ab74cf3452d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:51:49.403585 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:51:49.403554 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df612bf7-b12b-426b-aaaf-7ab74cf3452d-maas-api-tls" pod "df612bf7-b12b-426b-aaaf-7ab74cf3452d" (UID: "df612bf7-b12b-426b-aaaf-7ab74cf3452d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:51:49.403696 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:51:49.403593 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df612bf7-b12b-426b-aaaf-7ab74cf3452d-kube-api-access-4jc8z" pod "df612bf7-b12b-426b-aaaf-7ab74cf3452d" (UID: "df612bf7-b12b-426b-aaaf-7ab74cf3452d"). InnerVolumeSpecName "kube-api-access-4jc8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:51:49.403696 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:51:49.403631 5918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df612bf7-b12b-426b-aaaf-7ab74cf3452d-metrics-certs" pod "df612bf7-b12b-426b-aaaf-7ab74cf3452d" (UID: "df612bf7-b12b-426b-aaaf-7ab74cf3452d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:51:49.502747 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:51:49.502713 5918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df612bf7-b12b-426b-aaaf-7ab74cf3452d-maas-api-tls\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:49.502747 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:51:49.502739 5918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df612bf7-b12b-426b-aaaf-7ab74cf3452d-metrics-certs\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:49.502747 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:51:49.502749 5918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jc8z\" (UniqueName: \"kubernetes.io/projected/df612bf7-b12b-426b-aaaf-7ab74cf3452d-kube-api-access-4jc8z\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:49.502978 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:51:49.502760 5918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df612bf7-b12b-426b-aaaf-7ab74cf3452d-openshift-service-ca\") on node \"ip-10-0-137-81.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:51:49.813862 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:51:49.813828 5918 generic.go:356] "Generic (PLEG): container finished" podID="df612bf7-b12b-426b-aaaf-7ab74cf3452d" containerID="46c0157459c91ba74192ac934015245d1da5a469b4481bdde247b7e631676270" exitCode=0 Oct 30 22:51:49.814056 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:51:49.813880 5918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b" Oct 30 22:51:49.814056 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:51:49.813897 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b" event={"ID":"df612bf7-b12b-426b-aaaf-7ab74cf3452d","Type":"ContainerDied","Data":"46c0157459c91ba74192ac934015245d1da5a469b4481bdde247b7e631676270"} Oct 30 22:51:49.814056 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:51:49.813939 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b" event={"ID":"df612bf7-b12b-426b-aaaf-7ab74cf3452d","Type":"ContainerDied","Data":"0d462dd9c0a3c3597f7f5d3fb7b83647b8c4000a6372fe7648c29787f3253b05"} Oct 30 22:51:49.814056 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:51:49.813960 5918 scope.go:122] "RemoveContainer" containerID="46c0157459c91ba74192ac934015245d1da5a469b4481bdde247b7e631676270" Oct 30 22:51:49.829791 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:51:49.829704 5918 scope.go:122] "RemoveContainer" containerID="46c0157459c91ba74192ac934015245d1da5a469b4481bdde247b7e631676270" Oct 30 22:51:49.830085 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:51:49.830051 5918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46c0157459c91ba74192ac934015245d1da5a469b4481bdde247b7e631676270\": container with ID starting with 46c0157459c91ba74192ac934015245d1da5a469b4481bdde247b7e631676270 not found: ID does not exist" containerID="46c0157459c91ba74192ac934015245d1da5a469b4481bdde247b7e631676270" Oct 30 22:51:49.830231 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:51:49.830096 5918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46c0157459c91ba74192ac934015245d1da5a469b4481bdde247b7e631676270"} err="failed to get container status \"46c0157459c91ba74192ac934015245d1da5a469b4481bdde247b7e631676270\": rpc error: code = NotFound desc = could not find container \"46c0157459c91ba74192ac934015245d1da5a469b4481bdde247b7e631676270\": container with ID starting with 46c0157459c91ba74192ac934015245d1da5a469b4481bdde247b7e631676270 not found: ID does not exist" Oct 30 22:51:49.983873 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:51:49.983841 5918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b"] Oct 30 22:51:49.990755 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:51:49.990727 5918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-40b9a8-7f6d4ccbdd-kg74b"] Oct 30 22:51:51.273973 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:51:51.273940 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:51:51.850113 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:51:51.850076 5918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df612bf7-b12b-426b-aaaf-7ab74cf3452d" path="/var/lib/kubelet/pods/df612bf7-b12b-426b-aaaf-7ab74cf3452d/volumes" Oct 30 22:52:21.273955 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:52:21.273877 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:52:51.273885 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:52:51.273847 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:53:20.409904 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:53:20.409871 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-rqjfj_7e8c5776-206b-4be6-8f5b-4b3c26c6da52/manager/0.log" Oct 30 22:53:20.700031 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:53:20.700004 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-b2lk7_aef35927-4a0c-445c-8579-61c28d79e02b/manager/0.log" Oct 30 22:53:21.017027 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:53:21.016953 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-6v8v2_e70b1e47-1244-4502-92f3-7a2288cc28ab/manager/0.log" Oct 30 22:53:21.274805 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:53:21.274714 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:53:23.570617 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:53:23.570590 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-857794cb89-d7nl2_38825407-9c99-46f6-92d0-2fe72f297bd3/maas-api/0.log" Oct 30 22:53:27.298334 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:53:27.298308 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-xm656_aea8b7e0-9ce6-46b9-8164-314dc39f9114/kuadrant-console-plugin/0.log" Oct 30 22:53:29.762427 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:53:29.762396 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b76569bf4-jtdtb_669ab9c8-6c2a-4689-9e4d-e59f77e78fbb/kube-auth-proxy/0.log" Oct 30 22:53:30.464163 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:53:30.464126 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-dkzwh_4488ee36-392f-42c2-8dad-962472895f91/istio-proxy/0.log" Oct 30 22:53:32.165447 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:53:32.165412 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-xmr6p_1fe658cf-6504-47e1-8640-cc6404e8a882/istio-proxy/0.log" Oct 30 22:53:32.489812 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:53:32.489782 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-zwvws_ac8d2c79-d9b9-4650-8ffa-39022153bff9/istio-proxy/0.log" Oct 30 22:53:46.732349 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:53:46.732313 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-rqjfj_7e8c5776-206b-4be6-8f5b-4b3c26c6da52/manager/0.log" Oct 30 22:53:47.018829 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:53:47.018737 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-b2lk7_aef35927-4a0c-445c-8579-61c28d79e02b/manager/0.log" Oct 30 22:53:47.365779 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:53:47.365701 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-6v8v2_e70b1e47-1244-4502-92f3-7a2288cc28ab/manager/0.log" Oct 30 22:53:49.317549 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:53:49.317516 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-857794cb89-d7nl2_38825407-9c99-46f6-92d0-2fe72f297bd3/maas-api/0.log" Oct 30 22:53:51.273914 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:53:51.273879 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:53:52.764513 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:53:52.764487 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-xm656_aea8b7e0-9ce6-46b9-8164-314dc39f9114/kuadrant-console-plugin/0.log" Oct 30 22:53:54.901424 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:53:54.901391 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b76569bf4-jtdtb_669ab9c8-6c2a-4689-9e4d-e59f77e78fbb/kube-auth-proxy/0.log" Oct 30 22:53:55.585592 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:53:55.585559 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-dkzwh_4488ee36-392f-42c2-8dad-962472895f91/istio-proxy/0.log" Oct 30 22:53:57.310017 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:53:57.309985 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-xmr6p_1fe658cf-6504-47e1-8640-cc6404e8a882/istio-proxy/0.log" Oct 30 22:53:57.627781 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:53:57.627698 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-zwvws_ac8d2c79-d9b9-4650-8ffa-39022153bff9/istio-proxy/0.log" Oct 30 22:54:21.273815 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:54:21.273776 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:54:51.274874 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:54:51.274834 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:55:18.394414 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:18.394369 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xbwzg_4630b412-da87-49f7-bbbb-ba3dc820ddd1/global-pull-secret-syncer/0.log" Oct 30 22:55:20.913661 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:20.913575 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hmx87_18a7ec6e-13fa-4893-a179-529f13fb2172/konnectivity-agent/0.log" Oct 30 22:55:21.274585 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:55:21.274552 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor" Oct 30 22:55:22.408637 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:22.408601 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-81.us-west-1.compute.internal_ff2bf62cbfb1edc876c4b4c7d0808622/haproxy/0.log" Oct 30 22:55:27.247157 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:27.247111 5918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8jvzx/perf-node-gather-daemonset-rv8w2"] Oct 30 22:55:27.247931 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:27.247907 5918 cpu_manager.go:414] "containerMap: removing container" podUID="df612bf7-b12b-426b-aaaf-7ab74cf3452d" containerName="maas-api" Oct 30 22:55:27.247980 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:27.247933 5918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df612bf7-b12b-426b-aaaf-7ab74cf3452d" containerName="maas-api" Oct 30 22:55:27.248037 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:27.248027 5918 memory_manager.go:365] "RemoveStaleState removing state" podUID="df612bf7-b12b-426b-aaaf-7ab74cf3452d" containerName="maas-api" Oct 30 22:55:27.251166 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:27.251149 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-rv8w2" Oct 30 22:55:27.309269 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:27.309237 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8jvzx/perf-node-gather-daemonset-rv8w2"] Oct 30 22:55:27.357281 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:27.357249 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8jvzx\"/\"openshift-service-ca.crt\"" Oct 30 22:55:27.357445 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:27.357296 5918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8jvzx\"/\"kube-root-ca.crt\"" Oct 30 22:55:27.358964 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:27.358943 5918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8jvzx\"/\"default-dockercfg-sdt8t\"" Oct 30 22:55:27.390551 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:27.390525 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a4686866-e722-4385-b717-7a0d1c416be4-proc\") pod \"perf-node-gather-daemonset-rv8w2\" (UID: \"a4686866-e722-4385-b717-7a0d1c416be4\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-rv8w2" Oct 30 22:55:27.390681 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:27.390563 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4686866-e722-4385-b717-7a0d1c416be4-lib-modules\") pod \"perf-node-gather-daemonset-rv8w2\" (UID: \"a4686866-e722-4385-b717-7a0d1c416be4\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-rv8w2" Oct 30 22:55:27.390681 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:27.390582 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4686866-e722-4385-b717-7a0d1c416be4-sys\") pod \"perf-node-gather-daemonset-rv8w2\" (UID: \"a4686866-e722-4385-b717-7a0d1c416be4\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-rv8w2" Oct 30 22:55:27.390776 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:27.390760 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a4686866-e722-4385-b717-7a0d1c416be4-podres\") pod \"perf-node-gather-daemonset-rv8w2\" (UID: \"a4686866-e722-4385-b717-7a0d1c416be4\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-rv8w2" Oct 30 22:55:27.390829 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:27.390819 5918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72l62\" (UniqueName: \"kubernetes.io/projected/a4686866-e722-4385-b717-7a0d1c416be4-kube-api-access-72l62\") pod \"perf-node-gather-daemonset-rv8w2\" (UID: \"a4686866-e722-4385-b717-7a0d1c416be4\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-rv8w2" Oct 30 22:55:27.491619 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:27.491588 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72l62\" (UniqueName: \"kubernetes.io/projected/a4686866-e722-4385-b717-7a0d1c416be4-kube-api-access-72l62\") pod \"perf-node-gather-daemonset-rv8w2\" (UID: \"a4686866-e722-4385-b717-7a0d1c416be4\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-rv8w2" Oct 30 22:55:27.491785 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:27.491643 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a4686866-e722-4385-b717-7a0d1c416be4-proc\") pod \"perf-node-gather-daemonset-rv8w2\" (UID: \"a4686866-e722-4385-b717-7a0d1c416be4\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-rv8w2" Oct 30 22:55:27.491785 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:27.491693 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4686866-e722-4385-b717-7a0d1c416be4-lib-modules\") pod \"perf-node-gather-daemonset-rv8w2\" (UID: \"a4686866-e722-4385-b717-7a0d1c416be4\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-rv8w2" Oct 30 22:55:27.491785 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:27.491731 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a4686866-e722-4385-b717-7a0d1c416be4-proc\") pod \"perf-node-gather-daemonset-rv8w2\" (UID: \"a4686866-e722-4385-b717-7a0d1c416be4\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-rv8w2" Oct 30 22:55:27.491785 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:27.491740 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4686866-e722-4385-b717-7a0d1c416be4-sys\") pod \"perf-node-gather-daemonset-rv8w2\" (UID: \"a4686866-e722-4385-b717-7a0d1c416be4\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-rv8w2" Oct 30 22:55:27.492010 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:27.491786 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4686866-e722-4385-b717-7a0d1c416be4-sys\") pod \"perf-node-gather-daemonset-rv8w2\" (UID: \"a4686866-e722-4385-b717-7a0d1c416be4\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-rv8w2" Oct 30 22:55:27.492010 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:27.491816 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4686866-e722-4385-b717-7a0d1c416be4-lib-modules\") pod \"perf-node-gather-daemonset-rv8w2\" (UID: \"a4686866-e722-4385-b717-7a0d1c416be4\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-rv8w2" Oct 30 22:55:27.492010 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:27.491906 5918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a4686866-e722-4385-b717-7a0d1c416be4-podres\") pod \"perf-node-gather-daemonset-rv8w2\" (UID: \"a4686866-e722-4385-b717-7a0d1c416be4\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-rv8w2" Oct 30 22:55:27.492112 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:27.492023 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a4686866-e722-4385-b717-7a0d1c416be4-podres\") pod \"perf-node-gather-daemonset-rv8w2\" (UID: \"a4686866-e722-4385-b717-7a0d1c416be4\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-rv8w2" Oct 30 22:55:27.555485 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:27.555417 5918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72l62\" (UniqueName: \"kubernetes.io/projected/a4686866-e722-4385-b717-7a0d1c416be4-kube-api-access-72l62\") pod \"perf-node-gather-daemonset-rv8w2\" (UID: \"a4686866-e722-4385-b717-7a0d1c416be4\") " pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-rv8w2" Oct 30 22:55:27.563355 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:27.563332 5918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-rv8w2" Oct 30 22:55:27.936669 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:27.936638 5918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8jvzx/perf-node-gather-daemonset-rv8w2"] Oct 30 22:55:27.938827 ip-10-0-137-81 kubenswrapper[5918]: W1030 22:55:27.938797 5918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda4686866_e722_4385_b717_7a0d1c416be4.slice/crio-98f124021f00e1c6f514fe7f8a70e133abba1dca1ba3f70ae6e658f626ba4f49 WatchSource:0}: Error finding container 98f124021f00e1c6f514fe7f8a70e133abba1dca1ba3f70ae6e658f626ba4f49: Status 404 returned error can't find the container with id 98f124021f00e1c6f514fe7f8a70e133abba1dca1ba3f70ae6e658f626ba4f49 Oct 30 22:55:27.952414 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:27.952384 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-rv8w2" event={"ID":"a4686866-e722-4385-b717-7a0d1c416be4","Type":"ContainerStarted","Data":"98f124021f00e1c6f514fe7f8a70e133abba1dca1ba3f70ae6e658f626ba4f49"} Oct 30 22:55:27.978725 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:27.978702 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-xm656_aea8b7e0-9ce6-46b9-8164-314dc39f9114/kuadrant-console-plugin/0.log" Oct 30 22:55:28.958393 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:28.958356 5918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-rv8w2" event={"ID":"a4686866-e722-4385-b717-7a0d1c416be4","Type":"ContainerStarted","Data":"238787708e74413f055895fa1063510d4e7932d7ab7c6a5e50834ae9636a0171"} Oct 30 22:55:28.958916 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:28.958467 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-rv8w2" Oct 30 22:55:29.078768 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:29.078714 5918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-rv8w2" podStartSLOduration=2.078700544 podStartE2EDuration="2.078700544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:55:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:55:29.077327837 +0000 UTC m=+3889.348025902" watchObservedRunningTime="2026-10-30 22:55:29.078700544 +0000 UTC m=+3889.349398627" Oct 30 22:55:31.993364 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:31.993331 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b077ad14-ff57-45ba-87c3-853a2e104003/alertmanager/0.log" Oct 30 22:55:32.417804 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:32.417723 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b077ad14-ff57-45ba-87c3-853a2e104003/config-reloader/0.log" Oct 30 22:55:32.840616 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:32.840581 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b077ad14-ff57-45ba-87c3-853a2e104003/kube-rbac-proxy-web/0.log" Oct 30 22:55:33.291539 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:33.291511 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b077ad14-ff57-45ba-87c3-853a2e104003/kube-rbac-proxy/0.log" Oct 30 22:55:33.718544 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:33.718515 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b077ad14-ff57-45ba-87c3-853a2e104003/kube-rbac-proxy-metric/0.log" Oct 30 22:55:34.151183 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:34.151096 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b077ad14-ff57-45ba-87c3-853a2e104003/prom-label-proxy/0.log" Oct 30 22:55:34.578057 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:34.578023 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b077ad14-ff57-45ba-87c3-853a2e104003/init-config-reloader/0.log" Oct 30 22:55:34.973539 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:34.973513 5918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8jvzx/perf-node-gather-daemonset-rv8w2" Oct 30 22:55:35.555461 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:35.555426 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-lbjd4_1e76444f-1a5b-460c-b42f-2197b7512574/kube-state-metrics/0.log" Oct 30 22:55:36.025111 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:36.025083 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-lbjd4_1e76444f-1a5b-460c-b42f-2197b7512574/kube-rbac-proxy-main/0.log" Oct 30 22:55:36.484900 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:36.484871 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-lbjd4_1e76444f-1a5b-460c-b42f-2197b7512574/kube-rbac-proxy-self/0.log" Oct 30 22:55:36.936331 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:36.936299 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-d6fb658f8-zvqw4_e321c1b1-e9e2-46fc-9a35-14a3d6bed233/metrics-server/0.log" Oct 30 22:55:37.372514 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:37.372486 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-dd5t2_290eb94c-6ae5-40e8-8ba3-dbe017243d9d/monitoring-plugin/0.log" Oct 30 22:55:39.135022 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:39.134988 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q5c65_4a7edc22-1eb1-4458-8f8c-6c46a2bd583c/node-exporter/0.log" Oct 30 22:55:39.565470 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:39.565436 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q5c65_4a7edc22-1eb1-4458-8f8c-6c46a2bd583c/kube-rbac-proxy/0.log" Oct 30 22:55:39.990629 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:39.990598 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q5c65_4a7edc22-1eb1-4458-8f8c-6c46a2bd583c/init-textfile/0.log" Oct 30 22:55:41.704490 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:41.704450 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-x796z_3c199354-31a1-45b4-8c60-a485fd65f1e2/kube-rbac-proxy-main/0.log" Oct 30 22:55:42.131459 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:42.131374 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-x796z_3c199354-31a1-45b4-8c60-a485fd65f1e2/kube-rbac-proxy-self/0.log" Oct 30 22:55:42.554728 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:42.554696 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-x796z_3c199354-31a1-45b4-8c60-a485fd65f1e2/openshift-state-metrics/0.log" Oct 30 22:55:42.992078 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:42.992048 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_35140ca6-78c4-49d2-8755-ba7f22efb045/prometheus/0.log" Oct 30 22:55:43.413383 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:43.413299 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_35140ca6-78c4-49d2-8755-ba7f22efb045/config-reloader/0.log" Oct 30 22:55:43.836627 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:43.836580 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_35140ca6-78c4-49d2-8755-ba7f22efb045/thanos-sidecar/0.log" Oct 30 22:55:44.259916 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:44.259888 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_35140ca6-78c4-49d2-8755-ba7f22efb045/kube-rbac-proxy-web/0.log" Oct 30 22:55:44.684577 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:44.684509 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_35140ca6-78c4-49d2-8755-ba7f22efb045/kube-rbac-proxy/0.log" Oct 30 22:55:45.107907 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:45.107817 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_35140ca6-78c4-49d2-8755-ba7f22efb045/kube-rbac-proxy-thanos/0.log" Oct 30 22:55:45.536957 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:45.536907 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_35140ca6-78c4-49d2-8755-ba7f22efb045/init-config-reloader/0.log" Oct 30 22:55:47.301614 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:47.301582 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-89884678b-sxtjs_8a4d3550-acb2-4cf1-b192-adf694522695/telemeter-client/0.log" Oct 30 22:55:47.727226 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:47.727182 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-89884678b-sxtjs_8a4d3550-acb2-4cf1-b192-adf694522695/reload/0.log" Oct 30 22:55:48.150876 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:48.150791 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-89884678b-sxtjs_8a4d3550-acb2-4cf1-b192-adf694522695/kube-rbac-proxy/0.log" Oct 30 22:55:48.591867 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:48.591836 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-796d66f799-qtc4t_f4925cf0-04b5-4ca4-b284-c0355e63b4ca/thanos-query/0.log" Oct 30 22:55:49.017767 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:49.017731 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-796d66f799-qtc4t_f4925cf0-04b5-4ca4-b284-c0355e63b4ca/kube-rbac-proxy-web/0.log" Oct 30 22:55:49.446435 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:49.446398 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-796d66f799-qtc4t_f4925cf0-04b5-4ca4-b284-c0355e63b4ca/kube-rbac-proxy/0.log" Oct 30 22:55:49.872068 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:49.871988 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-796d66f799-qtc4t_f4925cf0-04b5-4ca4-b284-c0355e63b4ca/prom-label-proxy/0.log" Oct 30 22:55:50.296470 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:50.296434 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-796d66f799-qtc4t_f4925cf0-04b5-4ca4-b284-c0355e63b4ca/kube-rbac-proxy-rules/0.log" Oct 30 22:55:50.726796 ip-10-0-137-81 kubenswrapper[5918]: I1030 22:55:50.726771 5918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-796d66f799-qtc4t_f4925cf0-04b5-4ca4-b284-c0355e63b4ca/kube-rbac-proxy-metrics/0.log" Oct 30 22:55:51.274881 ip-10-0-137-81 kubenswrapper[5918]: E1030 22:55:51.274834 5918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-v6wts" containerName="extractor"